The gauntlet needs its own gauntlet: negative controls for the checks themselves

[klmtseng]

Love this skill. The evidence-first flow and the mutation stage are the strongest parts of this design.

One layer I could not find in the README or the demo: mutation testing plants bugs in the product code. I did not see anything that plants faults in the checkers themselves. If I missed an existing mechanism, happy to be corrected and this issue can be closed.

Background for why I think this layer matters: we spent the last two months running a similar gate system for AI-generated work, and the silent failures all lived in the checkers. Mutation testing tells you whether the tests catch a planted bug in the product. Nothing tells you whether a checker has quietly broken, unless you plant faults in the checkers too. A checker can break loudly, by crashing. The dangerous kind is fail-open: nothing crashes, and the check passes.

Two real cases from our failure ledger:

1. A shell gate whose own header said "this script never uses || true" had two of them on grep calls. When the manifest file was unreadable, grep returned rc=2, the check passed vacuously, and the gate printed PASS with exit 0. It survived two adversarial audits, because reviewers read the logic they were pointed at. It only died when we ran a negative control: chmod 000 the input and require the gate to fail.

2. All seven checks green, and the artifact was garbage: an ASR stage hallucinated a transcript from digitally silent audio, and the "transcript non-empty" check passed it. Every check measured "not structurally broken". None measured "fit for purpose". A non-empty check on ML output happily passes hallucinated output.

Concrete proposal, happy to help prototype any of these:

• A stage 0 "gauntlet self-test": for every check, one known-bad fixture that must FAIL, plus the known-good demo that must PASS. You already have planted bugs for mutation; this extends the same idea to each checker.
• A break-the-checker rule: if a checker crashes, times out, or reads an unreadable input, the gauntlet hard-fails. No fallthrough, no || true, no 2>/dev/null inside gate code.
• A skip-path audit: any item silently dropped from a loop (unparseable file, empty case) is collected and reported, not skipped. Dropped inputs are never checked, and that class of miss is invisible in green output.
• Per metric, one written answer to: "is there a post-process that produces this number without the property it claims to measure?" If yes, the metric needs a paired check from the other direction.

Methodology and the full ledger format we use: https://github.com/klmtseng/validity-audit

[AmazingAng]

Thank you — real gap, confirmed by our own ledger: tools/mutants.py originally counted any nonzero pytest exit as a kill, so a usage error would have faked a whole batch. A fail-open checker, exactly your class, caught only by external review.

One clarification: tests-as-checkers already have a trust chain (RED watches every test fail, immediate passes need a throwaway mutant, mutation checks assertions, randomized order checks determinism). What had no rules was what you pointed at: the non-test checkers.

Adopted in a0f54a1 (evidence rebound in d885815):

• Proposal 2 in full — new SKILL.md rule: home-grown checks fail closed; no || true, no 2>/dev/null, no silent skips (your proposal 3 folded in). The entry-point guide spells out the grep trap: rc 1 is the only pass, rc 0 and rc ≥ 2 both fail.
• Proposal 1 narrowed — every home-grown check needs a one-off negative control before its pass is trusted; no standing fixtures for off-the-shelf tools. The demo's two greps now run inside gauntlet.sh; all three controls (including your chmod-000 case) failed as required — see evidence.md. Bonus: the secret scan's first run flagged its own pattern literal; fixed with s[e]cret, not exclusion.

Not adopted: the per-metric written answer — that question is answered structurally here (coverage ↔ mutation, mutation ↔ strict exit codes, suite ↔ random order), and mandatory prose per metric didn't pass our bar. Happy to revisit with a concrete counterexample. The ASR case is what SKILL.md's opening caveat covers: the gauntlet proves the code meets the spec; it can't prove the spec captures what matters — that's what the human-approved SPEC and real execution are for.

[AmazingAng]

Feel free to PR!

[klmtseng]

Thanks. Happy to send a PR. I’ll keep it small and focus on the negative control and fail closed checker path.Sent from my iPhoneOn Aug 7, 2026, at 9:08 PM, 0xAA ***@***.***> wrote:AmazingAng left a comment (AmazingAng/old-coder#1) Feel free to PR! —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>