The problem
test_kit_doctor.py::test_kit_repo_self_check_is_clean rehashes every kit-owned
file against kit-manifest.json. So any byte changed in an engine fails it —
including a deliberate mutation. A mutation-testing run therefore reports
"killed" for every mutant while nothing behavioural caught anything.
Reproduce in ten seconds — disable _flat's truncation entirely (a pure
behaviour change no drift check should care about):
def _flat(text: object, n: int = 999999) -> str: # was 120
FAILED scripts/tests/test_kit_doctor.py::test_kit_repo_self_check_is_clean
1 failed, 316 passed
Looks killed. Nothing behavioural touched it.
Why this is worse than a papercut
It is retroactive. docs/agentic-dev-kit/fallback-review-panel.md makes
mutation testing contract item 5, and "N mutants died" was used across #25, #28,
#29 and #31 as evidence that a property was pinned — and, more consequentially,
as a reason to stop reviewing. Any of those runs where the reviewer did not
exclude this test is worthless.
Measured on #31: one lens's first pass reported 17/17 killed. Re-run with the
drift test excluded: 7 had survived — including the exact guard the commit
message under review was advertising as fixed.
A false-negative testing tool is worse than no tool, because it is used to
justify confidence.
Proposed fix
Make the exclusion a documented one-liner rather than folklore:
@pytest.mark.driftcheck
def test_kit_repo_self_check_is_clean(): ...
- register the marker in pytest config
uv run pytest -m 'not driftcheck' becomes the documented mutation-testing
invocation
- reference it from
fallback-review-panel.md contract item 5 (which currently
warns in prose, helping the panel but not anyone mutating by hand) and from
wherever /code-review describes mutation testing
Alternative considered: regenerate the manifest after each mutation. Works,
but it is per-mutant bookkeeping every reviewer must remember, and forgetting it
fails silently in the confident direction.
Scope note
This is not specific to the panel — it applies to any agent or human
mutation-testing a kit-owned engine, which the kit now actively asks for.
The problem
test_kit_doctor.py::test_kit_repo_self_check_is_cleanrehashes every kit-ownedfile against
kit-manifest.json. So any byte changed in an engine fails it —including a deliberate mutation. A mutation-testing run therefore reports
"killed" for every mutant while nothing behavioural caught anything.
Reproduce in ten seconds — disable
_flat's truncation entirely (a purebehaviour change no drift check should care about):
Looks killed. Nothing behavioural touched it.
Why this is worse than a papercut
It is retroactive.
docs/agentic-dev-kit/fallback-review-panel.mdmakesmutation testing contract item 5, and "N mutants died" was used across #25, #28,
#29 and #31 as evidence that a property was pinned — and, more consequentially,
as a reason to stop reviewing. Any of those runs where the reviewer did not
exclude this test is worthless.
Measured on #31: one lens's first pass reported 17/17 killed. Re-run with the
drift test excluded: 7 had survived — including the exact guard the commit
message under review was advertising as fixed.
A false-negative testing tool is worse than no tool, because it is used to
justify confidence.
Proposed fix
Make the exclusion a documented one-liner rather than folklore:
uv run pytest -m 'not driftcheck'becomes the documented mutation-testinginvocation
fallback-review-panel.mdcontract item 5 (which currentlywarns in prose, helping the panel but not anyone mutating by hand) and from
wherever
/code-reviewdescribes mutation testingAlternative considered: regenerate the manifest after each mutation. Works,
but it is per-mutant bookkeeping every reviewer must remember, and forgetting it
fails silently in the confident direction.
Scope note
This is not specific to the panel — it applies to any agent or human
mutation-testing a kit-owned engine, which the kit now actively asks for.