fix(asvs): anchor DRIFT is not claim INVALIDATION — stop failing the gate on it - #295
Conversation
… gate on it
check_anchors failed on any evidence anchor whose token moved more than
+/-ANCHOR_WINDOW lines, whether or not the claim was still true. That collapses
two different assertions: "the evidence still exists" and "the evidence still
sits where it sat". Only the first is what this gate is for.
NOT A LOOSENED THRESHOLD. Execution reaches the moved-token branch only PAST the
`occurrences > 1` guard, so the token provably occurs EXACTLY ONCE in the file.
Its presence alone pins the evidence; the recorded line is a navigation aid.
Where the line IS load-bearing -- an ambiguous token, where a re-anchor to the
wrong occurrence cannot be detected -- it still hard-fails, unchanged.
Still fatal, i.e. the whole signal: a token that is GONE (the claim may now be
false), an AMBIGUOUS token, a missing evidence path, and every absence-claim
failure. This narrows the gate to claim-invalidation, which ADR 0156 already
describes as the intent ("Evidence anchors will break on refactors. That is the
feature ... a re-score becomes re-verify the cells whose anchors moved").
WHY IT MATTERS HERE, THOUGH THIS REPO NEVER FEELS IT. No workflow in this repo
runs scripts/asvs/scorecard.py -- the scorecard DATA and the gate that consumes
it live in the private vault, which mirrors this file byte-for-byte. So the
repo that causes the drift never runs the tool, and the repo that runs it is the
one blocked by it. Measured downstream: on 2026-08-07, 130 of 146 failures were
a single constant offset per file (+57 auth/service.py, +61 __main__.py, +47
store/base.py) and the count went 67 -> 146 in ten hours as this tree advanced.
It recurred the next day (+102 on .github/workflows/ci.yml). Every one of those
was a line number, not a claim.
The cost is not just noise. A gate that is red every morning for a reason nobody
must act on is a gate whose next REAL finding gets waved through -- and there was
one underneath that day: a cell asserting a security control was ABSENT when it
had since been built and merged. It sat in the same undifferentiated list as 130
line numbers.
Drift is still REPORTED, as DRIFT plus a count, because letting the recorded line
numbers rot silently is the failure mode on the other side.
Adds 3 tests. Mutation-checked: with this change reverted,
test_a_unique_token_that_drifted_is_advisory_not_fatal fails BEHAVIOURALLY (the
drift lands in `problems`). The other two are controls -- ambiguous stays fatal,
ordinary small movement stays silent -- and fail on the missing attribute, which
is a weaker signal; they assert unchanged behaviour by design.
Verified: 57 tests pass in tests/test_asvs_scorecard.py (was 54); ruff format and
ruff check clean on both files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
#295 as drafted merges into an EMPTY advisory list. Its advisory fired only when the token sat beyond +/-40, and measured against the current record NOTHING is beyond it: 725 of 1,980 anchors are stale-but-inside, and 0 are outside. So the change prevented the next recurrence while surfacing none of the accumulation, and its own closing instruction -- 're-anchor them, do not let them accumulate' -- was unactionable because the accumulation was invisible. The window is now gone from the decision path entirely. Uniqueness locates the evidence; the recorded line is reported output that an advisory corrects. GONE, AMBIGUOUS and missing-path stay fatal, unchanged -- those are claim invalidation, which is what this gate exists to assert. Measured on this branch against the live record: 725 of 1980 (36.6%) advisory, zero fatal, exit 0. Max offset in the corpus was 39 against a limit of 40, three days after a re-anchor pass reset the distribution by hand-retyping 130 integers. A tolerance 37% spent three days after a reset is a decaying budget, not a tolerance. THE LINE IS DERIVED FROM THE CHARACTER OFFSET, NOT BY SCANNING LINES. 42 live expect tokens span a newline, because the old check matched joined text and nothing forbade it. A per-line scan finds none of them and raises on the lookup. Now tested. Two tests changed contract and say so rather than being quietly deleted. test_ordinary_small_movement_is_neither_problem_nor_advisory asserted the silence that was the defect; it is renamed and inverted. Its guard-the-guard role is preserved in a new shape -- an exactly-located anchor must report NOTHING -- which catches the opposite failure, a resolver that advises on everything. Both were mutation-proven: an off-by-one in the derivation reds four tests including the new control.
A GONE token has FOUR causes and only two are re-anchors: it moved beyond detection, it was renamed, THE GAP IT CERTIFIED WAS CLOSED, or its control was removed. A tool that helpfully suggests the nearest similar line collapses all four into the first, and case three is the dangerous one -- re-anchoring to the code that CLOSED a gap, while the residual still narrates the gap, yields an anchor that resolves forever while asserting the opposite of the truth. Worked example, measured this session and named in the code: 3.7.5 anchored pyproject.toml testpaths = ["tests"]. BACKLOG #1027 widened it, so the token vanished -- but the anchor certified an EXCLUSION that had just been CLOSED, because the guard's test sits inside the path #1027 added. Re-anchoring would have been silently wrong. It was retired instead. So the failure message now puts the retire-vs-rescore fork in front of the reader and says 'do not re-anchor by default', and a test enforces the refusal: the message must not fuzzy-suggest a target even when a plausible near-match sits on the very next line. Enforced rather than documented, because a convention is what decays -- the fixture reproduces 3.7.5's exact shape, which is when a suggestion would be most tempting and most wrong. Taxonomy contributed by the asvs-tracking-rework session, which adopted the retire-as-closed class after I hit it; this is its enforcement half.
On the line that IS the record's rendered face. An anchor check proves the token is present and unique in its file. It does not prove the statement still executes under the control flow the cell reasoned about, and it cannot prove the cell's conclusion follows -- `expect` is matched as a SUBSTRING, so a statement that moved inside a try, into another function, or under a different condition still resolves. Measured instance: 15.3.1 sat at `pass` with every anchor resolving while the control it named had a hole, found only by EXECUTING the code. "verified" invites exactly the inference the tool cannot make, and it is the summary a reader quotes. Now: "resolved N evidence anchors (token present and unique -- NOT proof the control operates) and checked M absence claims". Contributed by the parallel asvs-tracking-rework session's substring analysis, whose framing is that an Anchor models a PREMISE and Cell.verdict models a CONCLUSION, with no object for the inference step between them -- so no check can attach there, and the summary must stop implying one did.
Post-merge assertion, run before any merge button is touched#297 landed at 05:30Z, so this branch now carries it. Merged, not rebased — the two standing rulings conflicted: "rebase onto #297" requires a force-push, and "never force-push this adopted branch" is absolute. A merge satisfies both and preserves The merge was CLEAN — no conflict. That is precisely the dangerous case: #297 carried all four 4 → 1, and the survivor is the retirement comment. Any other number means the window came back. Corpus re-measure, against engine
|
check_anchorsfails on any evidence anchor whose token moved beyond±ANCHOR_WINDOW, whether or not the claim is still true. That collapses two different assertions:Only the first is a claim about security posture.
Not a loosened threshold
Execution reaches the moved-token branch only past the
occurrences > 1guard, so the token provably occurs exactly once in the file. Its presence alone pins the evidence; the recorded line is a navigation aid.Where the line is load-bearing — an ambiguous token, where a re-anchor to the wrong occurrence cannot be detected — it still hard-fails, unchanged.
Still fatal, i.e. the whole signal: a GONE token (the claim may now be false), an AMBIGUOUS token, a missing path, and every absence-claim failure. That narrows the gate to claim-invalidation — which ADR 0156 already describes as the intent: "Evidence anchors will break on refactors. That is the feature … a re-score becomes re-verify the cells whose anchors moved."
Why this repo never feels the problem it causes
Worth stating plainly, because it's the reason this hasn't surfaced here: no workflow in this repo runs
scripts/asvs/scorecard.py. I checked all of them. The scorecard data, and the gate that consumes it, live in the private vault — which mirrors this file byte-for-byte and is where it actually gates anything.So the repo that causes the drift never runs the tool, and the repo that runs it is the one blocked by it.
Measured downstream:
+57auth/service.py,+61__main__.py,+47store/base.py+102on.github/workflows/ci.ymlEvery one of those was a line number, not a claim.
The cost isn't just noise
A gate that's red every morning for a reason nobody must act on is a gate whose next real finding gets waved through. There was one underneath that day: a cell asserting a security control was absent when it had since been built and merged. It sat in the same undifferentiated list as 130 line numbers.
Drift is still reported — as
DRIFTplus a count — because letting the recorded line numbers rot silently is the failure mode on the other side.Verification
3 tests added (57 pass, was 54). Mutation-checked: with the change reverted,
test_a_unique_token_that_drifted_is_advisory_not_fatalfails behaviourally — the drift lands inproblems. The other two are controls (ambiguous stays fatal; ordinary small movement stays silent) and fail on the missing attribute, which is a weaker signal — they assert unchanged behaviour by design.ruff format --checkandruff checkclean on both files.Found while re-scoring ASVS cells against this tree; companion to #293.
🤖 Generated with Claude Code