Proposal: optional deterministic detection judge via shared ATLAS/OWASP axis (complements the LLM judge)

[eeee2345]

Proposal: a deterministic detection signal alongside the LLM judge (via a shared ATLAS/OWASP axis)

The attack-generation side is already broad — evaluators/agent/injection, mcp/tool-poisoning, supply-chain, etc. cover the surface well, and each evaluator.yaml already carries standards: { owasp-api, atlas }. That shared axis is what makes this proposal concrete, so I'm not suggesting new attack patterns (you have those) — I'm suggesting a complementary way to judge outcomes.

The gap: pass_criteria is natural-language, LLM-judged. For a lot of attack categories the "did the payload land / was the response malicious" question also has a deterministic answer, and for reproducibility + cost that's worth having.

Proposal: an optional deterministic judge backed by Agent Threat Rules (ATR — MIT, ~700 detection rules, each tagged with MITRE ATLAS + OWASP LLM/Agentic in metadata). Because your evaluators already declare atlas / owasp-api codes, ATR rules can be matched to an evaluator by that shared code — so for an mcp/tool-poisoning or injection run, ATR gives a deterministic verdict (rule id, category) on the generated exchange, alongside the LLM judge. Benefits mirror the LLM-judge tradeoffs in reverse: zero judge-model tokens, reproducible, and "attack X fired ATR rule Y" is a stable artifact for regression.

Two shapes, whichever fits your runner:

1. Deterministic judge module the evaluator can call in addition to (or as a pre-check before) the LLM pass_criteria.
2. A crosswalk doc only — evaluator id → ATLAS/OWASP → ATR rule(s) — if you'd rather keep judging LLM-only but want the mapping for users who run both tools.

I maintain ATR (COI: I author it; MIT, engine-neutral). Raising as an issue first to see whether a deterministic judge is something you'd want in the runner, or whether the crosswalk-only version is the better fit. Happy to follow CONTRIBUTING for whichever.

[achuvyas-kv]

Thanks for opening this — the shared standards: tags make this easy to connect, and we like the idea of a deterministic check alongside the LLM judge.

If this moves forward, our preference would be:

1. Run ATR first on the full attack exchange (user input, agent output, tool calls).
2. Collect any hits — mainly the pattern description for what matched (e.g. "System prompt and secret extraction requests").
3. Pass those descriptions into the LLM judge as structured evidence before it decides PASS/FAIL.

That way we get stable, repeatable signals in the report without maintaining two separate verdicts. The LLM judge stays the final call, but it's grounded on what was actually detected instead of reasoning from scratch.

Does that match how you meant ATR to be used?