diff --git a/skills/old-coder/SKILL.md b/skills/old-coder/SKILL.md index 966fc6e..acddfb4 100644 --- a/skills/old-coder/SKILL.md +++ b/skills/old-coder/SKILL.md @@ -109,7 +109,7 @@ or a tool is unavailable, record that in the evidence report with the reason. | Static types | whole classes of bugs | tsc / mypy / etc., zero new errors | | Lint + format | latent bugs, drift | project's linter, zero new warnings | | Coverage on changed lines | untested code paths | every changed/added line executed by a test; branch coverage where the tool supports it. Global % is vanity — changed-line coverage is the constraint. **This layer must exit nonzero when its threshold is missed** (`--cov-fail-under`, `diff-cover --fail-under`, equivalent): a layer that prints a percentage and exits 0 is a report, not a gauntlet layer, and it will sit there green while coverage falls | -| Mutation testing | tests that assert nothing | see `references/gauntlet.md`. No mutation tool? Do manual mutation: introduce 3–5 plausible bugs into the new code one at a time (flip a comparison, off-by-one a bound, drop a condition, return early); the suite must kill every one. Restore after | +| Mutation testing | tests that assert nothing | **prefer the project's mutation tool** (mutmut, cosmic-ray, Stryker, PIT…), which generates mutants from the syntax tree and cannot silently skip one. No tool available? Manual mutation, per `references/gauntlet.md` — introduce 3–5 plausible bugs one at a time; the suite must kill every one; restore after. A hand-rolled runner must **prove it executed each mutant**: a runner that can report a kill it never ran inflates the score and no red gauntlet will ever surface it | | Property-based tests | edge cases you didn't imagine | for parsing, math, serialization, anything with invariants (round-trip, idempotence, ordering) — add hypothesis/fast-check properties | | Complexity budget | unmaintainable output | new functions small and single-purpose; if a function needs a paragraph to explain, split it | | Real execution | "passes tests, doesn't run" | actually run the app/CLI/endpoint once on a realistic input, not only the test harness | diff --git a/skills/old-coder/references/gauntlet.md b/skills/old-coder/references/gauntlet.md index 72abd77..4d06fd1 100644 --- a/skills/old-coder/references/gauntlet.md +++ b/skills/old-coder/references/gauntlet.md @@ -120,6 +120,23 @@ dependency diff so the human can see exactly what the agent pulled in. ## Manual mutation procedure (any language, no tool) +**Reach for the project's mutation tool first.** A real tool generates mutants +from the syntax tree, so it cannot apply a mutant to code that has moved and it +cannot report a mutant it did not run. A hand-written mutant list matched +against source text is a second copy of the code: it goes stale on every +refactor of the thing it guards, and it fails in the one direction no gauntlet +can catch. Use the procedure below when no tool exists for the language, not as +a default. + +**A hand-rolled runner must prove it executed each mutant.** This is the sharp +edge, and this repo's own demo found it: two same-size mutants written in the +same second shared a bytecode cache, so the runner reported kills for mutants it +never executed. That class of defect can *only* inflate the score, which means +it can never surface as a red gauntlet — the layer stays green precisely because +it is broken. `tools/mutants.py` now guards against it (mtime pinning, a cache +check that aborts the run); any runner written from this procedure needs the +equivalent, and EVIDENCE should say which check proves execution. + Script this rather than hand-editing, and **persist the script in the repo** (e.g. `tools/mutants.py`): it holds the original source, applies each mutant by unique string replacement, runs the suite, and restores. Hand-editing N times