Skip to content

fix(asvs): anchor DRIFT is not claim INVALIDATION — stop failing the gate on it - #295

Merged
wshallwshall merged 5 commits into
mainfrom
fix/asvs-anchor-drift-advisory
Aug 9, 2026
Merged

fix(asvs): anchor DRIFT is not claim INVALIDATION — stop failing the gate on it#295
wshallwshall merged 5 commits into
mainfrom
fix/asvs-anchor-drift-advisory

Conversation

@wshallwshall

@wshallwshall wshallwshall commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

BRANCH ADOPTED, NOT AUTHORED. The base commit 3770380c (2026-08-08 18:49:46 -0500) is not
mine. I picked this branch up because the change was correct and the work needed finishing; my
commits sit on top of it and rewrite nothing. If you authored 3770380c, your history is intact.

Do not force-push this branch. An adopted branch is the one case where a force-push can destroy
someone else's only copy of their work.

How ownership was checked, and how half my check was wrong: I ran git worktree list (nothing held
the branch) and list_sessions (no running session). The second half is invalid -- isRunning:false
means PAUSED, not finished, so a paused owner is invisible to it. The durable rule, agreed with the
parallel session: branch ownership is answered by git worktree list plus
git log -1 --format='%an %ad', never by session liveness. Sessions are invisible; refs are not.


check_anchors fails on any evidence anchor whose token moved beyond ±ANCHOR_WINDOW, whether or not the claim is still true. That collapses two different assertions:

  • the evidence still exists — what this gate is for
  • the evidence still sits where it sat — a line number

Only the first is a claim about security posture.

Not a loosened threshold

Execution reaches the moved-token branch only past the occurrences > 1 guard, so the token provably occurs exactly once in the file. Its presence alone pins the evidence; the recorded line is a navigation aid.

Where the line is load-bearing — an ambiguous token, where a re-anchor to the wrong occurrence cannot be detected — it still hard-fails, unchanged.

Still fatal, i.e. the whole signal: a GONE token (the claim may now be false), an AMBIGUOUS token, a missing path, and every absence-claim failure. That narrows the gate to claim-invalidation — which ADR 0156 already describes as the intent: "Evidence anchors will break on refactors. That is the feature … a re-score becomes re-verify the cells whose anchors moved."

Why this repo never feels the problem it causes

Worth stating plainly, because it's the reason this hasn't surfaced here: no workflow in this repo runs scripts/asvs/scorecard.py. I checked all of them. The scorecard data, and the gate that consumes it, live in the private vault — which mirrors this file byte-for-byte and is where it actually gates anything.

So the repo that causes the drift never runs the tool, and the repo that runs it is the one blocked by it.

Measured downstream:

Date Effect
2026-08-07 130 of 146 failures were one constant offset per file — +57 auth/service.py, +61 __main__.py, +47 store/base.py
same day count went 67 → 146 in ten hours as this tree advanced
2026-08-08 recurred — +102 on .github/workflows/ci.yml

Every one of those was a line number, not a claim.

The cost isn't just noise

A gate that's red every morning for a reason nobody must act on is a gate whose next real finding gets waved through. There was one underneath that day: a cell asserting a security control was absent when it had since been built and merged. It sat in the same undifferentiated list as 130 line numbers.

Drift is still reported — as DRIFT plus a count — because letting the recorded line numbers rot silently is the failure mode on the other side.

Verification

3 tests added (57 pass, was 54). Mutation-checked: with the change reverted, test_a_unique_token_that_drifted_is_advisory_not_fatal fails behaviourally — the drift lands in problems. The other two are controls (ambiguous stays fatal; ordinary small movement stays silent) and fail on the missing attribute, which is a weaker signal — they assert unchanged behaviour by design.

ruff format --check and ruff check clean on both files.

Found while re-scoring ASVS cells against this tree; companion to #293.

🤖 Generated with Claude Code

wshallwshall and others added 5 commits August 8, 2026 18:49
… gate on it

check_anchors failed on any evidence anchor whose token moved more than
+/-ANCHOR_WINDOW lines, whether or not the claim was still true. That collapses
two different assertions: "the evidence still exists" and "the evidence still
sits where it sat". Only the first is what this gate is for.

NOT A LOOSENED THRESHOLD. Execution reaches the moved-token branch only PAST the
`occurrences > 1` guard, so the token provably occurs EXACTLY ONCE in the file.
Its presence alone pins the evidence; the recorded line is a navigation aid.
Where the line IS load-bearing -- an ambiguous token, where a re-anchor to the
wrong occurrence cannot be detected -- it still hard-fails, unchanged.

Still fatal, i.e. the whole signal: a token that is GONE (the claim may now be
false), an AMBIGUOUS token, a missing evidence path, and every absence-claim
failure. This narrows the gate to claim-invalidation, which ADR 0156 already
describes as the intent ("Evidence anchors will break on refactors. That is the
feature ... a re-score becomes re-verify the cells whose anchors moved").

WHY IT MATTERS HERE, THOUGH THIS REPO NEVER FEELS IT. No workflow in this repo
runs scripts/asvs/scorecard.py -- the scorecard DATA and the gate that consumes
it live in the private vault, which mirrors this file byte-for-byte. So the
repo that causes the drift never runs the tool, and the repo that runs it is the
one blocked by it. Measured downstream: on 2026-08-07, 130 of 146 failures were
a single constant offset per file (+57 auth/service.py, +61 __main__.py, +47
store/base.py) and the count went 67 -> 146 in ten hours as this tree advanced.
It recurred the next day (+102 on .github/workflows/ci.yml). Every one of those
was a line number, not a claim.

The cost is not just noise. A gate that is red every morning for a reason nobody
must act on is a gate whose next REAL finding gets waved through -- and there was
one underneath that day: a cell asserting a security control was ABSENT when it
had since been built and merged. It sat in the same undifferentiated list as 130
line numbers.

Drift is still REPORTED, as DRIFT plus a count, because letting the recorded line
numbers rot silently is the failure mode on the other side.

Adds 3 tests. Mutation-checked: with this change reverted,
test_a_unique_token_that_drifted_is_advisory_not_fatal fails BEHAVIOURALLY (the
drift lands in `problems`). The other two are controls -- ambiguous stays fatal,
ordinary small movement stays silent -- and fail on the missing attribute, which
is a weaker signal; they assert unchanged behaviour by design.

Verified: 57 tests pass in tests/test_asvs_scorecard.py (was 54); ruff format and
ruff check clean on both files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
#295 as drafted merges into an EMPTY advisory list. Its advisory fired only when
the token sat beyond +/-40, and measured against the current record NOTHING is
beyond it: 725 of 1,980 anchors are stale-but-inside, and 0 are outside. So the
change prevented the next recurrence while surfacing none of the accumulation,
and its own closing instruction -- 're-anchor them, do not let them accumulate'
-- was unactionable because the accumulation was invisible.

The window is now gone from the decision path entirely. Uniqueness locates the
evidence; the recorded line is reported output that an advisory corrects. GONE,
AMBIGUOUS and missing-path stay fatal, unchanged -- those are claim
invalidation, which is what this gate exists to assert.

Measured on this branch against the live record: 725 of 1980 (36.6%) advisory,
zero fatal, exit 0. Max offset in the corpus was 39 against a limit of 40, three
days after a re-anchor pass reset the distribution by hand-retyping 130 integers.
A tolerance 37% spent three days after a reset is a decaying budget, not a
tolerance.

THE LINE IS DERIVED FROM THE CHARACTER OFFSET, NOT BY SCANNING LINES. 42 live
expect tokens span a newline, because the old check matched joined text and
nothing forbade it. A per-line scan finds none of them and raises on the lookup.
Now tested.

Two tests changed contract and say so rather than being quietly deleted.
test_ordinary_small_movement_is_neither_problem_nor_advisory asserted the silence
that was the defect; it is renamed and inverted. Its guard-the-guard role is
preserved in a new shape -- an exactly-located anchor must report NOTHING -- which
catches the opposite failure, a resolver that advises on everything. Both were
mutation-proven: an off-by-one in the derivation reds four tests including the new
control.
A GONE token has FOUR causes and only two are re-anchors: it moved beyond
detection, it was renamed, THE GAP IT CERTIFIED WAS CLOSED, or its control was
removed. A tool that helpfully suggests the nearest similar line collapses all
four into the first, and case three is the dangerous one -- re-anchoring to the
code that CLOSED a gap, while the residual still narrates the gap, yields an
anchor that resolves forever while asserting the opposite of the truth.

Worked example, measured this session and named in the code: 3.7.5 anchored
pyproject.toml testpaths = ["tests"]. BACKLOG #1027 widened it, so the token
vanished -- but the anchor certified an EXCLUSION that had just been CLOSED,
because the guard's test sits inside the path #1027 added. Re-anchoring would
have been silently wrong. It was retired instead.

So the failure message now puts the retire-vs-rescore fork in front of the
reader and says 'do not re-anchor by default', and a test enforces the refusal:
the message must not fuzzy-suggest a target even when a plausible near-match sits
on the very next line. Enforced rather than documented, because a convention is
what decays -- the fixture reproduces 3.7.5's exact shape, which is when a
suggestion would be most tempting and most wrong.

Taxonomy contributed by the asvs-tracking-rework session, which adopted the
retire-as-closed class after I hit it; this is its enforcement half.
On the line that IS the record's rendered face. An anchor check proves the token
is present and unique in its file. It does not prove the statement still executes
under the control flow the cell reasoned about, and it cannot prove the cell's
conclusion follows -- `expect` is matched as a SUBSTRING, so a statement that
moved inside a try, into another function, or under a different condition still
resolves.

Measured instance: 15.3.1 sat at `pass` with every anchor resolving while the
control it named had a hole, found only by EXECUTING the code. "verified" invites
exactly the inference the tool cannot make, and it is the summary a reader quotes.

Now: "resolved N evidence anchors (token present and unique -- NOT proof the
control operates) and checked M absence claims".

Contributed by the parallel asvs-tracking-rework session's substring analysis,
whose framing is that an Anchor models a PREMISE and Cell.verdict models a
CONCLUSION, with no object for the inference step between them -- so no check can
attach there, and the summary must stop implying one did.
@wshallwshall

Copy link
Copy Markdown
Collaborator Author

Post-merge assertion, run before any merge button is touched

#297 landed at 05:30Z, so this branch now carries it. Merged, not rebased — the two standing rulings conflicted: "rebase onto #297" requires a force-push, and "never force-push this adopted branch" is absolute. A merge satisfies both and preserves 3770380c.

The merge was CLEAN — no conflict. That is precisely the dangerous case: #297 carried all four ANCHOR_WINDOW occurrences forward as unchanged context, so a keep-both resolution would have reinstated the window inside the very function this PR strips it from, with nothing to flag. Hence the assertion:

$ grep -c ANCHOR_WINDOW scripts/asvs/scorecard.py
1

$ grep -n ANCHOR_WINDOW scripts/asvs/scorecard.py
70:#: RETIRED 2026-08-09. There was an ``ANCHOR_WINDOW = 40`` here: a token found within 40 lines of its

$ grep -c _headline_caveat scripts/asvs/scorecard.py
2          # confirms #297's content is actually present, not just claimed

4 → 1, and the survivor is the retirement comment. Any other number means the window came back.

Corpus re-measure, against engine main post-merge

731 of 1980 anchors (36.9%) carry a stale line number
0 fatal
exit 0

advisory 0 was the tell for a silent reinstatement; it reads 731, matching the parallel session's independent measurement. 60 scorecard tests green, ruff and mypy clean.

One addition since review

f4738ce3 corrects the summary line from "verified N evidence anchors" to "resolved N evidence anchors (token present and unique — NOT proof the control operates)". An anchor check proves presence and uniqueness; it cannot prove the statement still executes under the control flow the cell reasoned about, because expect is a substring match. 15.3.1 is the measured instance — pass, every anchor resolving, control had a hole, found only by executing. The line that IS the record's rendered face was stating a false premise about itself.

@wshallwshall
wshallwshall merged commit 8cd0d16 into main Aug 9, 2026
34 checks passed
@wshallwshall
wshallwshall deleted the fix/asvs-anchor-drift-advisory branch August 9, 2026 08:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant