story-085: A rerun only shadows a broader failure

[jerodw]

Story

A rerun only shadows a broader failure.

A declared suite run fails, the stage self-routes, and the rerun after it passes. The pass was simply the more recent result, so it became the current one and the workflow advanced — nothing asked whether the rerun had run as much as the failure did. A sequence that repairs the failure and a sequence that narrows the selector until the answer is convenient were, to the harness, the same sequence. story-083 recorded what each run was narrowed by; this story compares two of those records.

The rule

suite_run_shadows(passing_scope, failing_scope) — a later pass supersedes an earlier failure only when its recorded scope is a subset of the failure's, i.e. the rerun filtered at least as little. Its whole body is set containment.

• An unfiltered rerun shadows whatever failed.
• A rerun at the same scope shadows, so an ordinary fix-and-rerun completes exactly as before. This is not a requirement that a full-scope green precede completion.
• A rerun carrying a selection the failure did not carry established nothing about what it stopped running, so it shadows nothing and the failure still stands.

The limit is stated where the rule is written. A scope entry is compared as an opaque string; no selector syntax is parsed, normalised or interpreted here any more than anywhere else, to the rule story-040 and story-041 established. Two spellings of one selection are two strings, and a run made broader by rewriting a selection rather than by dropping one is not recognised as broader — it is treated as narrowing and refused. The way out is to rerun at a scope the comparison can see is no larger, not to teach the function a syntax. The standing module drives the rule with strings that are not selectors at all, to show the verdicts do not depend on their being any.

Where it fires

RunState.unshadowed_suite_failure holds the outstanding failure — stage, attempt, try, recorded scope, exit code, and the retained result and output paths — so a refusal is reconstructable from state.json alone. Defaulted like every field beside it, so an older state file reads as no failure outstanding. A resume does not clear it: it is evidence a failure is unrepaired rather than a live allowance, so it is restored while the live counters beside it are zeroed.

Both firing points are inside the existing declared-suite-run block, so neither can happen without that run happening:

• Non-zero exit — the field is populated from the run just made and the retained paths already computed there, then the existing SUITE_FAILED self-route runs as it did. Nothing else about a red run changed; its route, artifacts and statement are as story-083 and story-084 left them.
• Zero exit with a failure outstanding — the two scopes are compared. Shadowing clears the field and saves state at the decision, so a crash before the next stage's save cannot resurrect a superseded failure. Not shadowing refuses the advance through the same self_route on SUITE_FAILED — no failure category was invented, because what stands is the suite failure and it takes the route a suite failure already takes. The artifacts cited are the original failure's retained pair rather than the passing rerun's, which is what story-084's retention is for.

_complete is guarded: a run reaching the end still carrying the failure escalates naming it and its retained record, reading state.json and costing no suite run.

Two of the three suite runs are deliberately outside the rule, and neither writes nor reads the field: the clean-clone check always runs unnarrowed and could only ever shadow, and the revert check's selector runs are narrowed by design and answer a different question. The rule governs the declared suite runs only.

tests/test_suite_run_scope.py's "no decision under orchestration/ consults the scope" assertion gains a by-name exemption for the one rule that owns the comparison, held shut from both sides — a decision taken anywhere else is still reported, and a companion assertion requires the coordinator to still take that decision, so the exemption cannot outlive what it exempts.

Evidence

• Suite on the working tree: 3860 passed, 1 skipped (suite-run-result.json, exit 0)
• Suite in a fresh clone with the story committed: 3860 passed, 1 skipped (clean-clone-result.json, exit 0)
• Verification: passed, no blocking issues
• Revert-check nomination: tests/test_suite_run_scope.py::test_no_decision_under_orchestration_consults_the_scope — exit 1 reverted, exit 0 applied
• 31 tests written; new module tests/test_suite_run_shadow.py
• 0 retries; one self-route, both its failures in the tester's own in-progress work

The single skip is a parametrize skip ("asserted exactly above"), not a disabled test.

Notes for review

story-084's retention worked on its first live run. suite-run-result-tester-attempt-1-try-0.json and its output survived the rerun, so the self-route's citation resolved to what it actually saw — 2 failed, 3858 passed — rather than to the green run that ended the story. That is the first time the failing run behind a self-route was readable after the fact.

A guardrail was exempted rather than weakened, and it is worth a look. test_cost_ceiling.py's "state keeps no per-stage total" scan now passes over unshadowed_suite_failure by name. The exemption lapses if the value ever becomes keyed by a stage — with a planted control asserting exactly that — and a companion assertion requires the exempted name to still be a field a run writes, so a stale exemption cannot linger.

One unverified is a declined mutation check. The verifier intended to flip suite_run_shadows to constant True and observe the module go red; the command to edit the tree was declined. Its fallback reasoning — that the rule's assertions are positive rather than absence-based — holds on inspection: the module carries three is False assertions on the rule, each of which fails against that mutant.

The refusal path is not exercised in production today, since every declared suite run records an empty scope and shadows([], []) is always true. It goes live if declared runs are ever narrowed, which is the point of putting the guard in before that happens.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HkujNrgMcsRBFWNEKWYnmA