fix(research): the paper's own verifier ran nowhere and finished nowhere - #728
Open
gHashTag wants to merge 1 commit into
Open
fix(research): the paper's own verifier ran nowhere and finished nowhere#728gHashTag wants to merge 1 commit into
gHashTag wants to merge 1 commit into
Conversation
`research/arxiv_tnf/verify_numbers.py` re-derives every headline number in the TNF paper from the committed records. Its own docstring says why it exists: "so somebody else can refute the numbers". Two defects meant nobody could. **It was collected by nothing.** The ratchet globs `research/audit_*.py`, `research/witness_*.py`, `research/verify_*.py` and `research/block/*.py`. This file is named for one of those three roles but sits one directory down, in `research/arxiv_tnf/`, so the glob never saw it. Neither did gate-ratchet.yml nor paper-numbers.yml, which runs a different script. A gate excluded by where it happens to live is precisely the implicit exclusion `gate_status_ratchet.py`'s docstring refuses to have -- SKIP_BLOCK_PREFIX exists so that leaving a gate out is auditable. Discovery now includes `research/arxiv_tnf/verify_*.py`. **It died before the end.** `confirm_w990.json` carries no `die_reads_canonical` field, so the script raised KeyError at line 963 and stopped -- taking the entire W991 competitor block with it. Those 27 checks had never run at all. Verifying 506 of 533 numbers and exiting on a traceback is not a partial pass; it is a verification whose scope nobody knows. A missing field is now reported as a divergence. It is deliberately not routed through `check()`'s `got is None` branch: that path increments neither counter and leaves the exit code at zero, so an absent measurement would read exactly like a satisfied one. The script now completes: **533 agree, 3 diverge, 0 blocks skipped**. All three divergences say one thing -- W990's dot4 confirmation sweep was still running when the record was written, and the script encodes the intended finish: * `dot4: подтверждено размещений` -- record says 1, script cites 2 * `dot4: обе площадки различны` -- record has 1 distinct site, script cites 2 * `чтения канонизированы` -- `die_reads_canonical` is absent from the record Either the sweep finishes at two placements or the citation drops to one. That is a call for whoever owns W990, not a number to quietly adjust here. The baseline records this gate as `findings`, which is what it measures today and not an endorsement of the three. Only that one line was added: `--update` would re-run all 89 gates on this machine and write my environment's answers over CI's, which is the failure mode half this repository's audits exist for.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two ways to not be verified
research/arxiv_tnf/verify_numbers.pyre-derives every headline number in the TNFpaper from the committed records. Its docstring states the purpose plainly — "so
somebody else can refute the numbers". Neither half of that was true.
Collected by nothing. The ratchet globs three role-named prefixes in
research/plus
research/block/. This file is namedverify_*, matching the convention, butlives in
research/arxiv_tnf/— one directory down, outside every glob.grep -rn verify_numbers .github/workflows/returns nothing;paper-numbers.ymlruns adifferent script (
tools/check_paper_numbers.py). So no automation has everexecuted it, which is why nobody noticed the second defect.
A gate left out by where it happens to sit is the implicit exclusion that
gate_status_ratchet.py's own docstring refuses to have —SKIP_BLOCK_PREFIXexists specifically so that omitting a gate is auditable.
Finished nowhere.
confirm_w990.jsonhas nodie_reads_canonicalfield, soline 963 raised
KeyErrorand the run stopped there — taking the whole W991competitor block with it. Those 27 checks had never executed. Before: 506 checks
and a traceback. After: 533 agree, 3 diverge, 0 blocks skipped.
A missing field is now a divergence, deliberately not routed through
check()'sgot is Nonebranch — that path increments neither counter and leaves the exitcode at zero, so an absent measurement would read exactly like a satisfied one.
The three divergences, which are left standing
All three are the same fact: W990's dot4 confirmation sweep was still running when
the record was written, and the script encodes the intended finished state.
dot4: подтверждено размещенийplacements_done: 1dot4: обе площадки различнычтения канонизированыEither the sweep finishes at two placements, or the citation drops to one. That is
a call for whoever owns W990 — adjusting the expected values here to match the
records would be moving the ruler to fit the board.
For contrast,
gft_signed_macis genuinely done: 4 placements, 3 distinct BSCANsites, identical word
0xa5a5337fin every build.dot4sits at 1 and 1.Baseline
One line added:
verify_numbers.py: findings. That is what it measures today, notan endorsement of the three divergences — the ratchet's contract is may-improve,
may-not-degrade.
--updatewas deliberately not run: it re-executes all 89 gates on whichevermachine invokes it and writes those answers over CI's. Six gates in this baseline
are
TIMEOUTat 120 s on a two-core runner and are not TIMEOUT here; rewriting thefile from a ten-core laptop would silently record a fleet of green that CI cannot
reproduce. Half this repository's audits exist because of exactly that failure.