[BREAKING] MAINT: Standardize Jailbreak scenario around dataset × techniques × jailbreaks by varunj-msft · Pull Request #2192 · microsoft/PyRIT

varunj-msft · 2026-07-14T22:43:46Z

Description

Rewrites the Jailbreak garak/airt scenario so a run is an explicit three-axis cross-product — dataset × techniques × jailbreaks. A user selects a dataset (e.g. harmbench), techniques (the default is a simple "just send", with the normal ones available as opt-in), and a set of jailbreak templates. The default technique set is deliberately small because jailbreak × technique explodes quickly.

What changed

Techniques axis = registry attack techniques. The technique enum (JailbreakTechnique) is now built dynamically from the shared AttackTechniqueRegistry (like RapidResponse / Leakage / Cyber), so the "normal" techniques (role_play_*, many_shot, tap, crescendo_*, …) are available as opt-in. The default is a single scenario-local prompt_sending technique ("just send") — a deliberately small default because jailbreak × technique explodes. DEFAULT resolves to {prompt_sending}; ALL / SINGLE_TURN / MULTI_TURN aggregates come from the registry tags.
Jailbreaks axis = a separate template selector. Which jailbreak templates to run is its own set of run-parameters: num_templates (draw N random templates), jailbreak_names (explicit file list, e.g. aim.yaml dan_11.yaml), and num_attempts (repeat each technique×template×objective N times). A small curated default set (aim.yaml, dan_11.yaml) runs when neither selector is given. Chosen template names are persisted to scenario metadata so a --resume run replays the same set (random samples don't get redrawn).
Delivery = inline request converter. Each selected jailbreak template is applied as a TextJailbreakConverter on every selected technique's outgoing requests (threaded through the matrix builder's technique_converters → factory.create(extra_request_converters=...)). The objective is rendered inline into the template's {{prompt}} slot, so the target sees the jailbroken prompt exactly as authored. Delivering the jailbreak as a request converter (rather than prepended framing) keeps the scenario target-agnostic and lets it attach to any technique's outgoing requests. Any per-technique converters a caller layers on via --techniques <name>:converter.* are preserved (the jailbreak converter is added on top, not in place of them).
Default dataset → harmbench (max_dataset_size=4), replacing airt_harms.
BASELINE_ATTACK_POLICY = Disabled. An un-jailbroken prompt-send over the objectives is supported but off by default (the jailbreak templates dominate a run); callers opt in per run with include_baseline=True. This aligns with the canonical Disabled baseline convention used by sibling scenarios.
VERSION 1 → 2. Replaces the old match/dispatch enum (SIMPLE, resolved via _get_atomic_attack_from_technique_async) and its converter block with the registry technique axis + inline-converter delivery above.

Known framework limitation (pre-existing). The default technique prompt_sending works end-to-end. Opting into a simulated-conversation technique (role_play_*, crescendo_*) builds atomic attacks but raises at run time — ValueError: adversarial_chat is required … (attack_parameters.py) — because the shared matrix builder does not propagate a default adversarial target to those techniques when no adversarial axis is passed. This is framework-wide and sibling-consistent: the merged RapidResponse scenario's own default (role_play_movie_script + many_shot) raises the identical error. It is out of scope here (the clean fix belongs in the shared MatrixAtomicAttackBuilder, e.g. falling back to a resolved default adversarial target) and should be tracked as a separate framework follow-up.

Why converter delivery (and not a system/user "delivery-method" axis): an earlier iteration modeled delivery as a technique-seed framing axis (System / User). Review found it structurally broken: a jailbreak system seed at sequence 0 collides with the simulated-conversation seed range, so role_play_* / crescendo_* × jailbreak silently produced zero attacks (~9 of the registry techniques); many_shot ignored the template entirely (it builds its own prompt and excludes prepended_conversation); templates with text after the {{prompt}} placeholder were emptied; and it forced a native EDITABLE_HISTORY requirement on the target. Inline-converter delivery — how the V1 scenario already composed the jailbreak — has none of these problems and is what this PR ships.

Deliberately deferred (open question): a dedicated system-vs-user delivery-method axis is not included; user-role framing needs base-layer support to avoid dropping the objective. If wanted, it's a clean opt-in follow-up on top of this axis. Also confirm: (1) the jailbreaks axis currently ships as num_templates (random draw of N) + jailbreak_names (explicit list) rather than a single param literally named jailbreaks — can consolidate to one jailbreaks selector if preferred; and (2) curated-vs-random default template set (this PR ships a small curated default; can switch to a random draw).

Composition (shipped default). Jailbreak() default = prompt_sending × the 2 curated default templates (aim.yaml, dan_11.yaml) × harmbench (max_dataset_size=4) = 2 atomic attacks (no baseline by default) over 4 harmbench objectives = 8 objective-executions, single-turn. The default is intentionally kept small/fast; the full technique catalog and template catalog are opt-in via --techniques / --num-templates / --jailbreak-names.

Tests and Documentation

tests/unit/scenario/airt/test_jailbreak.py — 30 tests, all passing. Covers: template selection (num_templates / jailbreak_names mutual exclusion, unknown-name error, subdirectory names, persisted-name resume); jailbreak delivered as a TextJailbreakConverter via factory.create (+ objective seed groups carry no prepended framing); user-supplied technique converters preserved alongside the jailbreak converter; role_play_* / crescendo_* × jailbreak still build atomic attacks at init (guards against the zero-attacks failure the superseded system-seed design introduced — note these techniques raise at run time under the framework adversarial_chat limitation above; the test asserts build-time production + converter attachment only); baseline policy = Disabled (default omits baseline, include_baseline=True opts in); unique atomic-attack naming; num_attempts multiplies attacks; memory_labels propagation; dynamic technique-model shape (DEFAULT = {prompt_sending}, registry techniques available, delivery roles are not technique members).
Docs: doc/scanner/airt.py + doc/scanner/airt.ipynb jailbreak section rewritten to the 3-axis model (harmbench default, prompt_sending default technique, converter delivery, --num-templates / --jailbreak-names). Sources kept in sync with jupytext. Note: the notebook's executed outputs are not regenerated in this change (a live jupytext --to ipynb --execute runs the entire airt notebook against Azure targets); per the repo's "don't strip outputs" convention they're left in place and should be refreshed with live creds at merge time.
Ran: ruff check, ruff format --check, ty, pytest tests/unit/scenario/airt/test_jailbreak.py (30 passed), and the full tests/unit/scenario/ suite (794 passed).

…hniques x jailbreaks Restructure airt.jailbreak into the 3-axis selection model Rich described in microsoft#2045 and the 7/7-7/9 Teams thread: a run is the cross-product of a dataset (harmful objectives), attack techniques, and jailbreak templates. - Technique axis is now the registry attack techniques (build_technique_class_from_ factories), matching every sibling AIRT scenario. A scenario-local "just send" prompt_sending technique is the default; role_play_*, many_shot, tap, etc. are opt-in. The default technique set is intentionally just {prompt_sending} because jailbreak x technique explodes quickly (Rich, 7/9). - Jailbreaks are a separate selector (num_templates random sample / jailbreak_names explicit / small curated default [aim, dan_11]), declared as run parameters so they are settable from the CLI / config. - Each selected jailbreak template is delivered inline as a TextJailbreakConverter appended to every selected technique's request converters, so the objective is rendered into the template's {{prompt}} slot on the wire. Converter delivery keeps the scenario target-agnostic and composes with every technique, including the simulated-conversation ones (role_play / crescendo), which a prepended-seed delivery silently drops. - Default objectives switched airt_harms -> harmbench (max_dataset_size=4), mirroring adversarial / red_team_agent. - num_attempts repeats each (technique x template x objective); results group by jailbreak template (display_group) and atomic-attack names are unique per (technique x template x attempt). - Resume-stable: the resolved template set is persisted to ScenarioResult metadata and replayed on resume so a random num_templates sample does not diverge. - BASELINE_ATTACK_POLICY = Disabled (bare run omits the un-jailbroken baseline; opt in with include_baseline=True). VERSION 1 -> 2. Deferred follow-ups (Rich-sanctioned): the system-vs-user delivery-role axis and the variation / translation delivery methods, plus the baseline-skip optimization.

rlundeen2

Looking good! I think with the changes I commented it should be pretty close.

Refine the standardized Jailbreak scenario: - Enable the baseline attack policy by default (was Disabled). - Rename run parameters to num_jailbreaks / num_jailbreak_attempts. - Build the technique pool from every registered factory plus the scenario-local delivery extras, instead of a curated "core" set. - Default to a random jailbreak sample rather than a curated name list. - Default techniques to prompt_sending + jailbreak_system_prompt. Native system-prompt delivery is capability-gated (requires editable history and system-prompt support) and degrades to converter-based delivery otherwise, so the objective is never silently dropped. - Order the system-framing seed at sequence=-1 so a caller-supplied seed group carrying a user prompt at the default sequence 0 does not raise a same-sequence role collision when system delivery merges in. Update the scanner docs/notebook and the scenarios instruction example to match.