From ca160eb60370d98b0b790bf1934d4a5fa83a35ce Mon Sep 17 00:00:00 2001 From: Mike Crowe Date: Wed, 12 Aug 2026 08:42:13 -0400 Subject: [PATCH] Prefer the real mutation tool, and make a hand-rolled runner prove it ran The manual mutation procedure reads as an equal alternative to a mutation tool. It is not, and this repo's own six-round verification is the evidence: the hand-rolled runner reported kills for mutants it never executed, because two same-size mutants written in the same second shared a bytecode cache. What makes that failure worth a rule rather than a bugfix is its direction. It could only ever inflate the score. A mutant that silently does not run reports a kill, so the layer stays green precisely because it is broken, and no red gauntlet can surface it. Every other layer here fails loudly when it breaks; this one did not. Two changes follow. The SKILL.md cell now states a preference order rather than an equivalence: the project's tool generates mutants from the syntax tree and cannot skip one silently, so reach for it first. And the manual procedure now carries the execution-proof requirement, pointing at the guard tools/mutants.py already gained in the fix, so the next person writing a runner from this procedure knows it is required rather than incidental. The general form: a hand-written mutant list matched against source text is a second copy of the code, and it goes stale on every refactor of the thing it is meant to guard. (Our fork goes further and reports the layer UNAVAILABLE rather than falling back to a hand-rolled runner at all. That is a bigger change to this skill's posture, so it is not in this PR.) Co-Authored-By: Claude Opus 5 (1M context) --- skills/old-coder/SKILL.md | 2 +- skills/old-coder/references/gauntlet.md | 17 +++++++++++++++++ 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/skills/old-coder/SKILL.md b/skills/old-coder/SKILL.md index 966fc6e..acddfb4 100644 --- a/skills/old-coder/SKILL.md +++ b/skills/old-coder/SKILL.md @@ -109,7 +109,7 @@ or a tool is unavailable, record that in the evidence report with the reason. | Static types | whole classes of bugs | tsc / mypy / etc., zero new errors | | Lint + format | latent bugs, drift | project's linter, zero new warnings | | Coverage on changed lines | untested code paths | every changed/added line executed by a test; branch coverage where the tool supports it. Global % is vanity — changed-line coverage is the constraint. **This layer must exit nonzero when its threshold is missed** (`--cov-fail-under`, `diff-cover --fail-under`, equivalent): a layer that prints a percentage and exits 0 is a report, not a gauntlet layer, and it will sit there green while coverage falls | -| Mutation testing | tests that assert nothing | see `references/gauntlet.md`. No mutation tool? Do manual mutation: introduce 3–5 plausible bugs into the new code one at a time (flip a comparison, off-by-one a bound, drop a condition, return early); the suite must kill every one. Restore after | +| Mutation testing | tests that assert nothing | **prefer the project's mutation tool** (mutmut, cosmic-ray, Stryker, PIT…), which generates mutants from the syntax tree and cannot silently skip one. No tool available? Manual mutation, per `references/gauntlet.md` — introduce 3–5 plausible bugs one at a time; the suite must kill every one; restore after. A hand-rolled runner must **prove it executed each mutant**: a runner that can report a kill it never ran inflates the score and no red gauntlet will ever surface it | | Property-based tests | edge cases you didn't imagine | for parsing, math, serialization, anything with invariants (round-trip, idempotence, ordering) — add hypothesis/fast-check properties | | Complexity budget | unmaintainable output | new functions small and single-purpose; if a function needs a paragraph to explain, split it | | Real execution | "passes tests, doesn't run" | actually run the app/CLI/endpoint once on a realistic input, not only the test harness | diff --git a/skills/old-coder/references/gauntlet.md b/skills/old-coder/references/gauntlet.md index 72abd77..4d06fd1 100644 --- a/skills/old-coder/references/gauntlet.md +++ b/skills/old-coder/references/gauntlet.md @@ -120,6 +120,23 @@ dependency diff so the human can see exactly what the agent pulled in. ## Manual mutation procedure (any language, no tool) +**Reach for the project's mutation tool first.** A real tool generates mutants +from the syntax tree, so it cannot apply a mutant to code that has moved and it +cannot report a mutant it did not run. A hand-written mutant list matched +against source text is a second copy of the code: it goes stale on every +refactor of the thing it guards, and it fails in the one direction no gauntlet +can catch. Use the procedure below when no tool exists for the language, not as +a default. + +**A hand-rolled runner must prove it executed each mutant.** This is the sharp +edge, and this repo's own demo found it: two same-size mutants written in the +same second shared a bytecode cache, so the runner reported kills for mutants it +never executed. That class of defect can *only* inflate the score, which means +it can never surface as a red gauntlet — the layer stays green precisely because +it is broken. `tools/mutants.py` now guards against it (mtime pinning, a cache +check that aborts the run); any runner written from this procedure needs the +equivalent, and EVIDENCE should say which check proves execution. + Script this rather than hand-editing, and **persist the script in the repo** (e.g. `tools/mutants.py`): it holds the original source, applies each mutant by unique string replacement, runs the suite, and restores. Hand-editing N times