Feature/multilingual

docs/source/index.rst

@@ -66,6 +66,7 @@ Code reference
    payloads
    _config
    _plugins
+   translator
 
 Plugin structure
 ^^^^^^^^^^^^^^^^

docs/source/translator.rst

@@ -0,0 +1,165 @@
+The `translator.py` module in the Garak framework is designed to handle text translation tasks using various translation services and models. 
+It provides several classes, each implementing different translation strategies and models, including both cloud-based services like DeepL and NIM, and local models like m2m100 from Hugging Face.
+
+garak.translator
+=============
+
+.. automodule:: garak.translator
+   :members:
+   :undoc-members:
+   :show-inheritance:   
+
+Multilingual support
+====================
+
+This feature adds multilingual probes and detector keywords and triggers.
+You can check the model vulnerability for multilingual languages.
+
+* limitation:
+  - This function only supports for `bcp47` code is "en".
+  - Huggingface detector only supports English. You need to bring the target language NLI model for the detector.
+  - Some detectors only support English, such as the `snowball` detector.
+  - If you fail to load probes or detectors, you need to choose a smaller translation model.
+
+pre-requirements
+----------------
+
+.. code-block:: bash
+
+    pip install nvidia-riva-client==2.16.0  pyenchant==3.2.2
+
+Support translation service
+---------------------------
+
+- Huggingface
+  - This code uses the following translation models:
+    - `Helsinki-NLP/opus-mt-en-{lang} <https://huggingface.co/docs/transformers/model_doc/marian>`_
+    - `facebook/m2m100_418M <https://huggingface.co/facebook/m2m100_418M>`_
+    - `facebook/m2m100_1.2B <https://huggingface.co/facebook/m2m100_1.2B>`_
+- `DeepL <https://www.deepl.com/docs-api>`_
+- `NIM <https://build.nvidia.com/nvidia/megatron-1b-nmt>`_
+
+API KEY
+-------
+
+You can use DeepL API or NIM API to translate probe and detector keywords and triggers.
+
+You need an API key for the preferred service.
+- `DeepL <https://www.deepl.com/en/pro-api>`_
+- `NIM <https://build.nvidia.com/nvidia/megatron-1b-nmt>`_
+
+Supported languages:
+- `DeepL <https://developers.deepl.com/docs/resources/supported-languages>`_
+- `NIM <https://build.nvidia.com/nvidia/megatron-1b-nmt/modelcard>`_
+
+Set up the API key with the following command:
+
+DeepL
+~~~~~
+
+.. code-block:: bash
+
+    export DEEPL_API_KEY=xxxx
+
+NIM
+~~~
+
+.. code-block:: bash
+
+    export NIM_API_KEY=xxxx
+
+config file
+-----------
+
+You can pass the translation service, source language, and target language by the argument.
+
+- translation_service: "nim" or "deepl", "local"
+- lang_spec: "ja", "ja,fr" etc. (you can set multiple language codes)
+
+* Note: The `Helsinki-NLP/opus-mt-en-{lang}` case uses different language formats. The language codes used to name models are inconsistent. Two-digit codes can usually be found here, while three-digit codes require a search such as “language code {code}". More details can be found `here <https://github.com/Helsinki-NLP/OPUS-MT-train/tree/master/models>`_.
+
+You can also configure this via a config file:
+
+.. code-block:: yaml
+
+    run:
+      translation_service: {you choose translation service "nim" or "deepl", "local"}
+      lang_spec: {you choose language code}
+
+Examples for multilingual
+-------------------------
+
+DeepL
+~~~~~
+
+To use the translation option for garak, run the following command:
+
+.. code-block:: bash
+
+    export DEEPL_API_KEY=xxxx
+    python3 -m garak --model_type nim --model_name meta/llama-3.1-8b-instruct --probes encoding --translation_service deepl --lang_spec ja
+
+If you save the config file as "garak/configs/simple_translate_config_deepl.yaml", use this command:
+
+.. code-block:: bash
+
+    export DEEPL_API_KEY=xxxx
+    python3 -m garak --model_type nim --model_name meta/llama-3.1-8b-instruct --probes encoding --config garak/configs/simple_translate_config_deepl.yaml
+
+Example config file:
+
+.. code-block:: yaml
+
+    run:
+      translation_service: "deepl"
+      lang_spec: "ja"
+
+NIM
+~~~
+
+For NIM, run the following command:
+
+.. code-block:: bash
+
+    export NIM_API_KEY=xxxx
+    python3 -m garak --model_type nim --model_name meta/llama-3.1-8b-instruct --probes encoding --translation_service nim --lang_spec ja
+
+If you save the config file as "garak/configs/simple_translate_config_nim.yaml", use this command:
+
+.. code-block:: bash
+
+    export NIM_API_KEY=xxxx
+    python3 -m garak --model_type nim --model_name meta/llama-3.1-8b-instruct --probes encoding --config garak/configs/simple_translate_config_nim.yaml
+
+Example config file:
+
+.. code-block:: yaml
+
+    run:
+      translation_service: "nim"
+      lang_spec: "ja"
+
+Local
+~~~~~
+
+For local translation, use the following command:
+
+.. code-block:: bash
+
+    python3 -m garak --model_type nim --model_name meta/llama-3.1-8b-instruct --probes encoding --translation_service local --lang_spec ja
+
+If you save the config file as "garak/configs/simple_translate_config_local.yaml", use this command:
+
+.. code-block:: bash
+
+    python3 -m garak --model_type nim --model_name meta/llama-3.1-8b-instruct --probes encoding --config garak/configs/simple_translate_config_local.yaml
+
+Example config file:
+
+.. code-block:: yaml
+
+    run:
+      translation_service: local
+      local_model_name: "facebook/m2m100_418M"
+      local_tokenizer_name: "facebook/m2m100_418M"
+      lang_spec: "ja"

garak/cli.py

@@ -247,6 +247,22 @@ def main(arguments=None) -> None:
         action="store_true",
         help="Launch garak in interactive.py mode",
     )
+    parser.add_argument('--lang_spec', type=str, help='Target language for translation')
+    parser.add_argument(
+       "--translation_service",
+       choices=["deepl", "nim", "local"],
+       help="Choose the translation service to use (overrides config file setting)",
+    )
+    parser.add_argument(
+       "--local_model_name",
+       type=str,
+       help="Model name",
+    )
+    parser.add_argument(
+       "--local_tokenizer_name",
+       type=str,
+       help="Tokenizer name",
+    )
 
     logging.debug("args - raw argument string received: %s", arguments)
 

garak/detectors/base.py

@@ -14,6 +14,7 @@
 from garak.configurable import Configurable
 from garak.generators.huggingface import HFCompatible
 import garak.attempt
+from garak.translator import SimpleTranslator, LocalTranslator, is_english
 
 
 class Detector(Configurable):
@@ -61,7 +62,38 @@ def __init__(self, config_root=_config):
             )
 
         logging.info(f"detector init: {self}")
-
+        translation_service = ""
+        if hasattr(config_root.run, 'translation_service'):
+            translation_service = config_root.run.translation_service
+        if translation_service == "local":
+            self.translator = LocalTranslator(config_root)
+        else:
+            self.translator = SimpleTranslator(config_root)
+        if hasattr(config_root.run, 'lang_spec'):
+            self.target_lang = config_root.run.lang_spec
+        if hasattr(self, 'substrings'):
+            self.substrings = self.translate_keywords(self.substrings)
+
+    def _translate(self, words: List[str]) -> List[str]:
+        if hasattr(self, 'target_lang') is False or self.bcp47 == "*":
+            return words
+        translated_keywords = []
+        for lang in self.target_lang.split(","):
+            if self.bcp47 == lang:
+                continue 
+            for word in words:
+                mean_word_judge = is_english(word)
+                if mean_word_judge:
+                    translated_keywords.append(self.translator._get_response(word, self.bcp47, lang))
+                else:
+                    translated_keywords.append(word)
+        words = list(words)
+        words.extend(translated_keywords)
+        return words 
+
+    def translate_keywords(self, keywords: List[str]) -> List[str]:
+        return self._translate(keywords)
+
     def detect(self, attempt: garak.attempt.Attempt) -> Iterable[float]:
         """Takes a list of Attempts; classifies them; returns a list of results
         in the range 0.0-1.0 where 0.0 represents no hit, 1.0 represents successful hit
@@ -169,6 +201,7 @@ class StringDetector(Detector):
     def __init__(self, substrings, config_root=_config):
         super().__init__(config_root=config_root)
         self.substrings = substrings
+        self.substrings = self.translate_keywords(self.substrings)
 
     def detect(
         self, attempt: garak.attempt.Attempt, case_sensitive=False
@@ -206,6 +239,7 @@ def detect(
         detector_results = []
         if "triggers" in attempt.notes:
             triggers = attempt.notes["triggers"]
+            triggers = self.translate_keywords(triggers)
             if isinstance(triggers, str):
                 triggers = [triggers]
             for output in attempt.all_outputs:

garak/detectors/leakreplay.py

@@ -18,6 +18,7 @@ def detect(self, attempt: Attempt, case_sensitive=False) -> List[float]:
         detector_results = []
         if "triggers" in attempt.notes:
             triggers = attempt.notes["triggers"]
+            triggers = self.translate_keywords(triggers)
             for output in attempt.all_outputs:
                 if output is None:
                     continue

garak/probes/atkgen.py

@@ -24,6 +24,7 @@
 import garak._plugins
 import garak.attempt
 from garak.probes.base import Probe
+from garak.translator import SimpleTranslator, LocalTranslator
 
 TEMPLATE_PLACEHOLDER = "[query]"
 
@@ -139,54 +140,57 @@ def probe(self, generator) -> List[garak.attempt.Attempt]:
                         challenge = re.sub(
                             self.red_team_postproc_rm_regex, "", challenge
                         ).strip()
+                    challenges = self._translate([challenge])
 
                 if not output_is_conversation:
                     t.update()
-                turn = ("probe", challenge)
+
+                for challenge in challenges:
+                    turn = ("probe", challenge)
 
-                if len(turns) == 0:
-                    this_attempt.prompt = challenge
-                    this_attempt._expand_prompt_to_histories(1)
-                else:
-                    this_attempt._add_turn("user", [challenge])
-
-                turns.append(turn)
-                logging.debug("atkgen: probe: %s", turn)
-                if output_is_conversation:
-                    probe_text = re.sub(r"[\r\n]+", "\n", turn[1]).strip()
-                    print(
-                        f"atkgen: 🔴 probe: {Fore.LIGHTYELLOW_EX}{probe_text}{Style.RESET_ALL}"
-                    )
-                else:
-                    t.set_description(
-                        f" turn {t.n:02}: waiting for [{generator.name[:10]:<10}]"
-                    )
-                # send the challenge and get the response
-                response = generator.generate(challenge)
-                if response is None or len(response) == 0:
-                    response = ""
-                else:
-                    response = response[0].strip() if response[0] is not None else ""
-
-                # log the response
-                turn = ("model", response)
-                this_attempt._add_turn("assistant", [response])
-                turns.append(turn)
-                logging.debug("atkgen: model: %s", turn)
-                if output_is_conversation:
-                    print(f"atkgen: 🦜 model: {Style.BRIGHT}{turn[1]}{Style.RESET_ALL}")
-                else:
-                    t.update()
-                # increment calls_made
-                calls_made += 1
-                # check if the resp is empty or if it matches the previous resp
-                if not len(response) and not self.constructive_tension:
-                    keep_going = False
-                if response == last_response:
-                    keep_going = False
-                # update last_response
-                last_response = response.replace("\n", " ").strip()
-                self.redteamer.max_new_tokens = 170  # after first iter, give a limit
+                    if len(turns) == 0:
+                        this_attempt.prompt = challenge
+                        this_attempt._expand_prompt_to_histories(1)
+                    else:
+                        this_attempt._add_turn("user", [challenge])
+
+                    turns.append(turn)
+                    logging.debug("atkgen: probe: %s", turn)
+                    if output_is_conversation:
+                        probe_text = re.sub(r"[\r\n]+", "\n", turn[1]).strip()
+                        print(
+                            f"atkgen: 🔴 probe: {Fore.LIGHTYELLOW_EX}{probe_text}{Style.RESET_ALL}"
+                        )
+                    else:
+                        t.set_description(
+                            f" turn {t.n:02}: waiting for [{generator.name[:10]:<10}]"
+                        )
+                    # send the challenge and get the response
+                    response = generator.generate(challenge)
+                    if response is None or len(response) == 0:
+                        response = ""
+                    else:
+                        response = response[0].strip() if response[0] is not None else ""
+
+                    # log the response
+                    turn = ("model", response)
+                    this_attempt._add_turn("assistant", [response])
+                    turns.append(turn)
+                    logging.debug("atkgen: model: %s", turn)
+                    if output_is_conversation:
+                        print(f"atkgen: 🦜 model: {Style.BRIGHT}{turn[1]}{Style.RESET_ALL}")
+                    else:
+                        t.update()
+                    # increment calls_made
+                    calls_made += 1
+                    # check if the resp is empty or if it matches the previous resp
+                    if not len(response) and not self.constructive_tension:
+                        keep_going = False
+                    if response == last_response:
+                        keep_going = False
+                    # update last_response
+                    last_response = response.replace("\n", " ").strip()
+                    self.redteamer.max_new_tokens = 170  # after first iter, give a limit
 
             if not output_is_conversation:
                 t.close()
@@ -234,3 +238,11 @@ def __init__(self, config_root=_config):
             msg = f"No query placeholder {TEMPLATE_PLACEHOLDER} in {self.__class__.__name__} prompt template {self.red_team_prompt_template}"
             logging.critical(msg)
             raise ValueError(msg)
+        translation_service = ""
+        if hasattr(config_root, 'run'):
+            if hasattr(config_root.run, 'translation_service'):
+                translation_service = config_root.run.translation_service
+        if translation_service == "local":
+            self.translator = LocalTranslator(config_root)
+        else:
+            self.translator = SimpleTranslator(config_root)