FEAT Plug-in mechanism for private scenarios

[ValbuenaVC]

Adding Private Scenario Support via Plug-Ins

Goal

Let an operator activate private scenarios and attack techniques from a stock PyRIT
installation — without forking PyRIT or passing per-run CLI flags — so a team's private
red-teaming package behaves like a built-in in the standard backend, GUI, and pyrit_scan
catalog.

# .pyrit_conf
plugins:
  - name: my_redteam
    source: /repos/my-redteam
    initializer: my_redteam.setup.MyInitializer

# A private technique through a public scenario
pyrit_scan airt.rapid_response --target openai_chat --techniques operation_foobar
# A private scenario the initializer registers
pyrit_scan my_redteam.violence --target openai_chat

What a plug-in is

A plug-in is a config-declared pointer to a private PyRITInitializer that lives outside
the PyRIT tree. It is one thin layer above --initialization-scripts:

• config-declared (backend/GUI honor it; no per-run flag);
• run first (privileged), before other initializers and before catalog/metadata
  warming, so lazy consumers see a complete registry;
• package-root-anchored — source is placed on sys.path, so a packaged initializer
  with intra-package imports (e.g. from my_redteam... import) loads correctly, which a
  loose --initialization-scripts path cannot do.

PluginSpec is {name, source, initializer}:

• name — an operator label used in logs/errors;
• source — the folder that contains your package, placed on sys.path (relative resolves
  against the config file);
• initializer — a dotted module.Class path to a concrete PyRITInitializer.

ConfigurationLoader prepends this as a privileged initializer automatically; operators
cannot list it under initializers: (it is not a registered initializer name).

Why not just order the config?

pyrit_scan is a thin REST client, and the backend resolves a scenario before it runs
any per-request initializers — so "list your important initializer first" is too late for
scanner discovery. The backend also initializes once at startup from its config file, not
per client command. And initializers: only resolves built-in registered names, while
initialization_scripts: loads a loose file without anchoring a package root. The plug-in
covers exactly this gap: startup-config-driven, guaranteed-first, and package-root-anchored.

The initializer owns registration

PyRIT discovers nothing on its own. The plug-in's initializer registers whatever it wants
discoverable, at whatever level of abstraction the private content requires:

• techniques via AttackTechniqueRegistry.register_from_factories(...) → --techniques;
• scenarios via ScenarioRegistry.register_class(cls, name=...) → pyrit_scan <name>;
• datasets via provider registration + memory load (private seeds stay in the
  operator's database, never published);
• default targets via set_default_value(...).

This is deliberately flexible: "private" is rarely all-or-nothing. Sometimes only a
dataset must stay private, sometimes a niche technique, sometimes an entire scenario
should not be exposed at all — the initializer publishes the generic parts and registers
only the sensitive ones.

Behavior

• Fail-closed: any failure (missing source, unresolvable/non-PyRITInitializer target,
  or an initializer that raises) aborts initialization with a precise error. A restart is
  required after fixing the config or source.
• Trust boundary: loading runs third-party Python with backend permissions. Whoever can
  write the config or the source can run code on the host. No transactional rollback of
  arbitrary side effects is claimed, and dependencies must already be installed in the
  backend environment.
• One plug-in per config.

Notable changes

• Restores RapidResponse to expose all registered techniques (drops a core-only
  filter, aligning with FIX: Aligning Scenarios with Technique Registry #2186), so an initializer-registered private technique surfaces via
  --techniques with no scenario-routing tag.
• Removes the earlier discovery-based machinery: source/scenario/technique discovery, the
  scenario:<name> technique_tags routing bridge, wheel extraction/versioning, the
  transactional registry snapshot, and the contributed/external registration APIs.

Testing

• Full unit suite: 10,417 passed, 120 skipped.
• Gated plug-in integration (RUN_ALL_TESTS=true): a source package whose initializer
  registers a real private scenario + technique is discoverable; a bad initializer fails
  closed.
• ty check pyrit: clean. Full pre-commit run --all-files: all hooks pass.

Limitations / out of scope

• Only one plug-in; no composition.
• No component discovery, no scenario: tags, no wheels/builder, no version resolution,
  no dataset/target/scorer/converter plug-in types (the initializer registers these).
• No hot reload; activation is at process/backend startup only.
• Built-in name shadowing is accepted (the trusted initializer owns its namespace); a
  shadow-warning is a possible follow-up.
• A private package must register its scenarios in its initializer and be on the current
  ScenarioTechnique API before it will load; PyRIT does no runtime API repair.

[Copilot]

Pull request overview

Adds a wheel-based plug-in loading mechanism to PyRIT initialization so operators can ship private scenarios (and dataset providers) outside the public repo, while ensuring plug-ins load before other initializers consume registries.

Changes:

• Introduces pyrit.setup.plugin_loader to extract a configured .whl into .plugin/, import it, run a bootstrap (register() or PyRITInitializer), and validate registration with rollback-on-failure.
• Hooks plug-in loading into initialize_pyrit_async as a guaranteed-first phase (after memory is set, before other initializers), with a plugin_fail_open override.
• Adds extensive unit coverage for loader behavior/guards and documents operator configuration via .env_example and .pyrit_conf_example; ensures .plugin/ is kept but ignored in git.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/unit/setup/test_plugin_loader.py	New unit test suite that builds a mock wheel and exercises loader spine + guardrails (fail-open/closed, rollback, collisions, cache, package inference).
pyrit/setup/plugin_loader.py	New plug-in loader implementation: extraction, import/submodule discovery, bootstrap execution, collision warnings, and rollback.
pyrit/setup/initialization.py	Adds the plug-in load phase into initialize_pyrit_async and exposes plugin_fail_open.
.pyrit_conf_example	Documents that plug-ins are not configured via .pyrit_conf and are loaded automatically during initialization.
.plugin/.gitkeep	Keeps .plugin/ directory present in the repo while allowing extracted artifacts to be ignored.
.gitignore	Ignores extracted plug-in artifacts under .plugin/ while retaining .gitkeep.
.env_example	Documents PLUGIN_WHEEL, PLUGIN_PACKAGE, PLUGIN_FAIL_OPEN, and PLUGIN_DIR configuration and trust boundary.

[fdubut]

Also as mentioned in our side conversation, I think it would be useful to share an example of how a customer would package their datasets/scenarios to make them available as a plugin.

[rlundeen2]

could we normalize this with how initializers do it since I think they discover filepaths?

[rlundeen2]

I don't dislike this solution, but I'm worried it's more complex than we need to solve the problem we're tackling.

[rlundeen2]

I'd recommend seeing if this makes our lives simpler or if you see it making our lives simpler going forward. My gut reaction is it will be easier to just include internal

• technique initializers
• datasets
• the pyrit_scan to run

[ValbuenaVC]

I've spent some time refactoring this and will have the changes in soon for additional review. A few notes:

• I want to scope this PR for just private scenarios and attack techniques. The groundwork will be there for other components, but I think they introduce enough complexity (especially datasets) that they deserve their own PRs.
• Earlier I conflated user discoverability with our actual implementation. Expecting users to build a well-formatted Python wheel for their plug-in is great for scaling plug-ins long term, but bad for live ops usage, even with a follow-up PR for a pyrit wheel builder. I think operators should be able to just write a config that includes the path to their current operation's scenarios/techniques and have PyRIT discover them.
• For simplicity's sake the config should be the source of truth for plug-ins. I don't think for this PR pyrit_scan should take any responsibility over plugins. The ConfigurationLoader and initialize_from_config_async components should handle them specifically because they can force precedence for plug-ins prior to anything else. Ideally pyrit_scan should remain very thin, and assume that by the time _dispatch_with_client_async is called, the plug-in components have already been added to their relevant registries.

[Copilot]

Pull request overview

Copilot reviewed 34 out of 35 changed files in this pull request and generated 3 comments.