MAINT: Standardize garak.encoding defaults and fix atomic-attack name collisions

[varunj-msft]

Description

Curates the garak.encoding scenario's default run and fixes atomic-attack name collisions. The largest change is to the defaults: the previous default finished too quickly to be a meaningful scan, so this widens the default scheme set and dataset size to land the default run inside a ~10–20 minute window.

What changed

• Introduced a curated DEFAULT aggregate and made it the default technique (default_technique ALL → DEFAULT). Previously the default was ALL at a tiny dataset size, so the default run was effectively unscoped. DEFAULT now expands to 13 schemes spanning encoding families so the default scan is a meaningful spread:
  • base-N: Base64, Base2048, Base16, Base32, ASCII85, Hex
  • byte-encodings: QuotedPrintable, UUencode
  • substitution ciphers: ROT13, Atbash, LeetSpeak
  • symbolic alphabets: MorseCode, NATO
  • The 4 niche/lossy schemes stay ALL-only: Braille, Ecoji, Zalgo, AsciiSmuggler.
• Raised max_dataset_size 3 → 10 for both default datasets (garak_slur_terms_en, garak_web_html_js).
• Trimmed near-duplicate Base64 variants. The Base64 converter row set was reduced to the distinct encodings; the trim is called out in a code comment next to the variant list.
• Fixed atomic-attack name collisions. Each converter variant now yields a uniquely-named atomic attack (e.g. ascii85_a85 / ascii85_b85) instead of colliding on the scheme name, so results/display groups no longer overwrite each other.
• VERSION 1 → 2.

Scale (verified by dry-init). DEFAULT = 76 atomic attacks (incl. the baseline) × 20 objectives (size-10 × 2 datasets) = 1,520 target requests. ALL = 96 atomic attacks × 20 = 1,920 requests.

Wall-clock (live measurement). Measured on the shipped DEFAULT run against gpt-4o (Azure gpt-4o-unsafe, Entra ID); the objective scorer is the local DecodingScorer (no extra LLM calls):

Config	max_concurrency	Wall-clock
DEFAULT (default concurrency = 4)	4	10.83 min (649.6 s)
DEFAULT (concurrency 12, scale probe)	12	3.79 min (227.7 s)

(1,520 requests; only 9/1,520 returned empty 204s and 0 rate-limit 429s at concurrency 4, so the numbers reflect near-clean round-trip throughput.) A real pyrit_scan garak.encoding --target … run uses max_concurrency=4 by default (the CLI has no explicit default; the scenario applies 4 when --max-concurrency is omitted), so 10.8 min is the user-facing number — inside the 10–20 min target window. The size-10 / 13-scheme curation moved the default from "finishes too quickly" into the target window; if a longer run is desired, the remaining levers are a further max_dataset_size bump or moving more schemes into DEFAULT.

Tests and Documentation

• tests/unit/scenario/garak/test_encoding.py — 30 tests, all passing. Covers: 17 total scheme names; DEFAULT is the default technique and DEFAULT excludes the 4 ALL-only schemes; Base64 trimmed to the two distinct variants (default + url-safe); atomic-attack name uniqueness per converter variant (collision fix); size-10 default dataset config. (The 76 / 96 atomic-attack counts above are from a dry-init of the scenario, not a hard-coded test assertion.)
• Docs: doc/scanner/garak.py + doc/scanner/garak.ipynb updated for the new default. Synced with jupytext.
• Ran: ruff check, ruff format --check, ty, pytest tests/unit/scenario/garak/test_encoding.py.

Note: MorseCode / NATO are lossy for XSS-style payloads in the garak_web_html_js dataset — this is a pre-existing default kept as-is; useful for the slur dataset. Flag if these should be scoped out of DEFAULT.

[rlundeen2]

Looks great! Only nit comments

[rlundeen2]

Rich agrees with this comment but it is copilot generated:

Nit / convention: this reads self._memory_labels (and the method also uses self._objective_target / self._scenario_techniques), but the scenarios guideline is to read runtime inputs from the immutable context snapshot passed into _build_atomic_attacks_async. It's functionally fine here since state is fully resolved by the time this runs, but context.memory_labels is right there — threading context down into _get_converter_attacks/_get_prompt_attacks would match the documented "read from context, not self._*" convention. Non-blocking.