Red Gate protocol, lexicon v2, gap analysis + fixes, and the published example gallery

[JRichlen]

A verifier-first operating protocol for agentic work, its research grounding, a working implementation, and the verification surfaces built around it — including a published before/after example gallery.

What this contains

1. docs/red-gate-protocol.md — the spec: ARM/TRACE/JUDGE rounds with classified gates, a generalized verifier (runnable · proven able to fail · pinned · independently executed), lazy recursion, pointer envelopes, and a corrections ledger from adversarial review.
2. docs/red-gate-glossary.md — the canonical lexicon (v2, adopted): ARM/TRACE/JUDGE, WITNESS, mandate, FAULT, scout/widen — plus the collision rulings and run-lifecycle verbs.
3. plugins/redgate/ — driver, criteria-contract (ARM), reconcile (JUDGE), the hooks enforcement layer, calibration, and the protocol references.
4. plugins/recurrence-detector/ — the growth loop's DETECT organ.
5. Research — the 88-pattern agentic corpus, the phase-structure prior-art study (which settled "keep 3 stages"), and a 14-agent gap analysis (34 skeptic-verified gaps).
6. Verification surfaces — a secret-scan gate, cross-plugin reference resolution, a context-tax budget, version-drift checks, untrusted-data provenance fences, a roster-level trigger-routing eval, a k-of-N statistical floor, and the published example gallery (real with-skill/without-skill model pairs, deployed to GitHub Pages).

Reflection, settled by research

Should rounds gain Reflect/Plan/Investigate phases? Researched against primary sources (PDSA, OODA, Polya, Kolb, the Scrum Guide, Beck's Canon TDD, ReAct, Reflexion, Self-Refine, CRITIC, LATS, ToT). Verdict: keep three stages. Reflection lives between iterations in every mature verified-work loop, and unenforced in-cycle phases evaporate. So it ships as gate obligations instead — a mandatory lesson field, a red verdict leaving a durable artifact the next ARM reads, and a widen cadence — plus an optional retro round type.

The blocking findings are fixed

An earlier re-review found four defects and this PR was parked with a "do not merge" banner. All four were re-verified as still live, fixed, and mutation-proven (reverting any one turns the cheap tier red):

1. A missing trailing newline turned the red gate green. The emitted check.sh read loop silently dropped a final unterminated line, so a failing criterion written last simply vanished and the gate exited 0. Fixed; a new coupled check feeds an unterminated failing criterion and asserts the exit code.
2. The evidence-freshness eval was uncoupled while labelled (exit-coupled) — it reused a fixture whose criteria failed anyway. Now runs on a passing fixture, where the gate is the only possible source of non-zero exit.
3. The hook guard had zero eval coverage — gutting its deny path left the tier green. Four new checks drive the real handler and assert exit codes.
4. Portability defects made the cross-harness claim false on macOS — GNU-only stat -c (every PASS read as stale), bare sed -i (half-pinned manifest on BSD), and a preflight hard-requiring timeout. All fixed with portable fallbacks.

Honest limit: CI is Linux, so the BSD/macOS branches are shape-checked (the fallback is present), not executed.

Also found and fixed: a live command injection

The adversarial verification pass on the gap fixes confirmed a real command injection in the graveyard delete-script generator — a repo name containing a double quote escaped the emitted BUNDLED="…" assignment and executed when the user ran the delete script, outside the bundle-existence guard that is the tool's entire safety premise. Reproduced end to end, then fixed by rejecting anything outside GitHub's [A-Za-z0-9._-] charset. The fuzz group now asserts defense-by-rejection across eight hostile name classes.

Example gallery

Real prompt/response pairs showing what each skill changes, with and without it, published to GitHub Pages beside the plugin docs. Every pair is a provenanced model run, never hand-written; the cheap tier refuses any snapshot lacking two real outputs plus provenance, and the page is regenerated deterministically from committed snapshots. A biweekly workflow captures fresh graded pairs and opens a review-gated PR.

Evals

Cheap tier: 1171 passed, 0 failed. Every new guard in this PR is mutation-proven. The behavioral tier gains a routing pack (deterministic, grader-free) with a k-of-N pass floor; the deep tier runs on the graveyard safety-path change.

🤖 Generated with Claude Code

https://claude.ai/code/session_011tuKEs7jU6FawKTbdHMzUU

[Copilot]

🟡 Changes recommended

The spec has an internal inconsistency in the BEGIN “preflight” definition that would block the intended red-gate behavior for greenfield binaries, and the new HTML artifact should declare a doctype to avoid quirks-mode rendering.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds Red Gate to the repository as documentation: a recursive BEGIN/MIDDLE/END protocol describing how the marketplace’s individual skills compose into a verifiable workflow, plus an accompanying interactive HTML map/simulator asset under docs/assets/.

Changes:

• Introduces docs/red-gate-protocol.md, a detailed spec for the Red Gate coordination protocol (invariant, stages, recursion, handoff envelope, skill map, and corrections ledger).
• Adds docs/assets/red-gate.html, a standalone interactive artifact that visualizes the protocol, skill map, refinements, fatal-flaw ledger, and missing-skill gaps.

File summaries

File	Description
docs/red-gate-protocol.md	Adds the Red Gate protocol spec (stages/invariants/recursion/handoff/skill map/corrections).
docs/assets/red-gate.html	Adds an interactive HTML visualization/simulator for the Red Gate protocol and mapped skills.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[JRichlen]

Demonstration — grill-me progressive disclosure (the anchor + path-consent change)

Per the demonstration discipline in AGENTS.md: this is the skill actually run on real input, before/after visible, with the rule that produced each change and the misses named. It is not a description of what the skill would do — it is the two openings the two prose versions actually produce.

Real input

The genuinely-unsettled decision I faced at the top of this turn (not a strawman — it is the plan this PR's final commit executed). Stated as it stood before I acted:

"The required behavioral tier is red. A single upstream 504 on one graveyard scenario made promptfoo exit 100, even though the model passed every scenario it was allowed to answer. I want to fix this so the tier stops going red on the weather. Options on the table: just re-run until it's green; disable/soften the required check; or build a statistical floor that tolerates transient faults. If a floor — how many repeats, what pass threshold, and how does it get wired so promptfoo's own exit code stops being the arbiter? Grill me before I build it."

Branches in that plan: (A) re-run vs. soften vs. statistical floor (the load-bearing one); (B) repeat count N; (C) pass threshold / floor; (D) how a transient 504 is distinguished from a real failure; (E) CI wiring so promptfoo's exit stops deciding the leg; (F) the cost of repeating a paid tier.

---

BEFORE — old prose (base ccd51df)

The old SKILL.md jumps straight to "## The frontier/round loop → Compute the current frontier. Ask it as one numbered round." There is no gate before questioning, so a faithful run opens like this:

❓ Q1 — Fix strategy: re-run, soften the check, or a statistical floor?
➡️ Statistical floor — re-running hides the signal, softening a verification-integrity check is self-refuting.
❓ Q2 — Repeat count: how many times does each scenario run?
➡️ 3 — smallest N that survives one fault and stays cost-conscious.
❓ Q3 — Floor: what per-scenario pass rate is required?
➡️ 0.6 (majority of 3).
❓ Q4 — Fault vs failure: how do you tell a 504 from a real rubric fail?
➡️ Exclude rows promptfoo marks as errors; score the floor over valid samples only.
❓ Q5 — Wiring: how does promptfoo's exit code stop being the arbiter?
➡️ `|| echo` the eval, let pass-rate.sh read results.json and decide.

Five questions fired in one round, before it was ever confirmed what "fixed" means or which of these the user actually wanted to open. Q2–Q5 are all downstream of Q1: if the user had said "just soften the check," every one of them is wasted. That is the exact waste the change targets.

AFTER — new prose (this PR)

The new "## Anchor, then consent to each path" section + references/path-consent.md forces a round 0 anchor, then a batched consent menu before any descent. Same input, faithful run:

Before I ask anything — the anchor, as I read it (correct me in one line):
• Goal: stop the required behavioral tier going red on transient upstream faults.
• Outcome: a 504 on one scenario can't fail a leg the model actually passed; a real rubric regression still does.
• Out of scope: changing which model/grader is used; the deep/pier tier.

One routing round — tell me which paths to spend questions on:

🧭 Path — Fix strategy (re-run / soften / statistical floor) [DEEP, ~3 q]
➡️ recommend: statistical floor — the only option that keeps the check meaningful.
🧭 Path — Fault-vs-failure classification (what counts as a transient) [DEEP, ~3 q]
➡️ recommend: exclude promptfoo error rows; fail closed if too few valid samples survive.
🧭 Path — CI wiring (who arbitrates the leg) [STANDARD, ~2 q]
➡️ recommend: let promptfoo exit non-zero; pass-rate.sh reads results.json and decides.

explore / accept / defer / out-of-scope?

(N=3, floor=0.6, and the paid-cost tradeoff are LIGHT confirm-or-skips hanging off "Fix strategy" — I'll fold them in only if you explore that path.)

The rule that produced the change: path-consent.md → "No branch questioning before three slots are set: Goal, Outcome, Out of scope" and "Before descending into any STANDARD or DEEP branch, present the branch as a header and let the user route… Batch the menu." The old run spent 5 questions on an unconfirmed goal; the new run spends zero until the user routes, and collapses B/C/F (the LIGHT knobs) under their parent instead of asking them as peers of the real decision.

---

The misses (this is not a sales pitch)

1. The anchor can be confidently wrong, and nothing catches it. I inferred "out of scope: the deep/pier tier." The plan never said that. The skill says infer-then-confirm-in-one-line, but a user skimming a plausible-looking anchor rubber-stamps it — and a wrong exclusion silently prunes a branch the user wanted. path-consent.md guards the inverse (descending into an off-anchor branch = "scope-silent exploration") but has no guard for wrongly excluding an in-scope branch via a bad anchor. That is the more dangerous direction and it is unhandled.

2. LIGHT branches still get no disclosure. N=3, floor=0.6, and the 3× paid-cost increase are real decisions with real consequences (0.6 vs 0.8 changes what fails), but as LIGHT confirm-or-skips they get no consent header — same silent-assume path as the old prose. The change improves STANDARD/DEEP disclosure and leaves the LIGHT floor exactly where it was. For this plan, the cost tradeoff arguably deserved a header; the skill's own rule ("a header would double their cost") suppresses it.

3. Leverage ordering is underdetermined. "Fix strategy" and "Fault-vs-failure classification" are both DEEP and both unsettled; the skill says order by stakes × unsettledness but gives no tiebreak, so the order above is my judgment, not the skill's. On a plan with several co-equal DEEP branches the "right order" the skill promises isn't actually determined by the prose.

4. On a mostly-settled plan the machinery is pure ceremony — and I confirmed this by running it on a second real input, docs/examples/PLAN.md, where Phases 1–4 are marked done. There the faithful run should hit "a plan that triages LIGHT end to end gets a brief 'no session needed'" and decline — but the new anchor step still fires first, so the skill sets up a round 0 anchor before reaching the sentence that tells it to bail. The decline short-circuit and the mandatory anchor are in slight tension; the anchor should be gated behind the LIGHT-end-to-end check, not ahead of it.

Net: the change removes the single most expensive failure mode (anchorless deep interrogation) cleanly and the before/after difference is real, but it leaves the LIGHT path undisclosed, has an asymmetric anchor guard, and adds a step that a fully-settled plan has to walk through before it's allowed to decline.

---

Generated by Claude Code