Skip to content

kit_doctor's drift self-check makes mutation testing report false kills #33

Description

@topij

The problem

test_kit_doctor.py::test_kit_repo_self_check_is_clean rehashes every kit-owned
file against kit-manifest.json. So any byte changed in an engine fails it —
including a deliberate mutation. A mutation-testing run therefore reports
"killed" for every mutant while nothing behavioural caught anything.

Reproduce in ten seconds — disable _flat's truncation entirely (a pure
behaviour change no drift check should care about):

def _flat(text: object, n: int = 999999) -> str:   # was 120
FAILED scripts/tests/test_kit_doctor.py::test_kit_repo_self_check_is_clean
1 failed, 316 passed

Looks killed. Nothing behavioural touched it.

Why this is worse than a papercut

It is retroactive. docs/agentic-dev-kit/fallback-review-panel.md makes
mutation testing contract item 5, and "N mutants died" was used across #25, #28,
#29 and #31 as evidence that a property was pinned — and, more consequentially,
as a reason to stop reviewing. Any of those runs where the reviewer did not
exclude this test is worthless.

Measured on #31: one lens's first pass reported 17/17 killed. Re-run with the
drift test excluded: 7 had survived — including the exact guard the commit
message under review was advertising as fixed.

A false-negative testing tool is worse than no tool, because it is used to
justify confidence.

Proposed fix

Make the exclusion a documented one-liner rather than folklore:

@pytest.mark.driftcheck
def test_kit_repo_self_check_is_clean(): ...
  • register the marker in pytest config
  • uv run pytest -m 'not driftcheck' becomes the documented mutation-testing
    invocation
  • reference it from fallback-review-panel.md contract item 5 (which currently
    warns in prose, helping the panel but not anyone mutating by hand) and from
    wherever /code-review describes mutation testing

Alternative considered: regenerate the manifest after each mutation. Works,
but it is per-mutant bookkeeping every reviewer must remember, and forgetting it
fails silently in the confident direction.

Scope note

This is not specific to the panel — it applies to any agent or human
mutation-testing a kit-owned engine, which the kit now actively asks for.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions