claude-opus-5 on nine splits: the annapolis displacement does not generalise (#139) - #146
Open
jonfroehlich wants to merge 5 commits into
Open
claude-opus-5 on nine splits: the annapolis displacement does not generalise (#139)#146jonfroehlich wants to merge 5 commits into
jonfroehlich wants to merge 5 commits into
Conversation
…ts (#139) 984 panoramas across bend, budapest_district5, clovis, gainesville, morgantown, paterson, richmond and sao_paulo, run 2026-08-18. With the annapolis leg already on main this takes `claude-opus-5-effort-low` to nine of the ten benchmark splits. Committed on its own, ahead of the scoring and doc work, because these are paid artifacts that existed only as untracked files in a worktree. The sibling worktree that produced #141 was deleted the same day; its commits survived only because they had been pushed. These had no such protection. `manual_gold` was NOT run and is not merely pending -- see the split's own issue for why a Claude-only run there cannot settle the question it appears to address. KNOWN GAP, and it is not recoverable in-repo: the API spend for these 984 panoramas was never written to analysis_out/usage_log.jsonl. The ledger holds $0.34 -- the sonnet annapolis leg and two richmond smoke tests -- and nothing for this run. Per compare.py's own note, the detection cache stores points only, so a re-run makes zero API calls and has no usage to report; the token counts cannot be back-filled from anything in this repository. The measured smoke rate was $0.058/pano, which puts the true figure near $57, but that is an inference and is recorded as one. Only Vertex billing / Cloud Monitoring can give the real number. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…that says The eight-split claude-opus-5 run wrote no row to analysis_out/usage_log.jsonl. `vertex_usage.py --days 3` recovered it the next day: 11,988,993 input / 418,503 output = $70.41. Layer 1 failed, layer 3 did its job. Written up in docs/model_comparison.md next to the Gemini history it parallels, and reported on #143, which had assumed the paid-API half needed no further work. This CORRECTS the previous commit's claim that the spend was "not recoverable" and its inferred figure of ~$57. That inference divided a smoke test's cost by `panos_scored` (4) when only 3 panos had actually been billed -- the fourth was cached. The real per-pano rate is $0.077, not $0.058. Two results worth keeping, both of which fall out of the recovery: - Estimate input from geometry; never estimate output. An Opus pano is exactly 12,186 input tokens (6 views x 2,031), identical across both smoke records, so 984 panos predicts 11,991,024 against 11,988,993 billed -- 0.02% out, a single call, re-derivable from the committed detections with no cloud access. The same extrapolation applied to output was 62% high ($16.92 against $10.46), because output is thinking plus box count and the two smoke panos were unusually verbose. Input is checkable; output is not. - Recovery is per-model per-day, so per-split attribution is permanently gone even on a successful pull. That is tolerable here only because the eight splits were one contiguous run of one model. And the deadline, which is the part that matters for next time: Cloud Monitoring retention is ~6 weeks. Had this gone unnoticed until October the number would have been unrecoverable at any price. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…neralise (#139) claude-opus-5 at low effort now has 7/7 pooled coverage, so it enters the headline and by-split tables and leaves the partial one. scoreboard.py --check is green again; it went red the moment the detections landed in e2f66fc. The result the issue was opened to get: on annapolis Opus led gemini-3.1-pro by +0.021 F1, the only time anything had displaced the top challenger. Pooled over seven city splits it TRAILS by -0.021 (0.588 vs 0.608). The one split that made the result look real was the one split that flattered it -- annapolis is the survey-grade Trimble MX7 city, and the issue pre-registered that as the risk. gemini-3.1-pro remains best challenger; RampNet's lead is unchanged at 0.219. What survives is the shape, not the ranking, and it matters more than the F1: Opus trades -0.083 precision for +0.041 recall, which makes it the highest-recall chat VLM on the board (0.614, above every Gemini leg). It loses on the aggregate and wins on the axis operating_point.md says to optimise. The scoreboard's own third finding is "precision is not the differentiator; recall is", so quoting the ranking alone gets the direction right and the reason wrong. Added as finding 4 rather than a footnote for that reason. standing stays False, and that is not an oversight. roster.py raises at import for any pinned leg that is standing -- a scored entry has to be what a bare --models spec reproduces, and both efforts of claude:claude-opus-5 share one spec. It is also unnecessary: standing gates the roster tables (fp_taxonomy, null_recall, the frozen WITNESS_POOL_46), not the scoreboard, which keys off coverage. gemini-3.7-flash and the three YOLO pano arms sit in the same position. Documented in the doc's own caveat list so the next reader does not try the flip and hit the assertion. export_model_cache.py --verify: 9/9 pairs score IDENTICALLY to the cache. Three tests failed on this change and all three were right to. Two encoded "Opus low is a single-split leg"; they now assert the opposite, with a note saying that a reappearance means lost coverage rather than a test needing a nudge. The third is the published-corpus ledger in replication.md, which exists because that count silently drifted four times: 114 -> 122 files, 23.1 -> 23.2 MB. manual_gold remains deliberately unrun (#144). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
) Review of #146. Eight findings, and one of them had a clock on it. **The "unrecoverable" cost claims were wrong, and are now recovered.** Both `replication.md` and `model_comparison.md` said the four #122 legs' token counts "cannot be recovered", giving a reason that only rules out re-running. This branch had just demonstrated the actual path, so it was applied to the older gap: `vertex_usage.py --days 7` returns claude-opus-5 $21.47 and claude-sonnet-5 $7.79 for 2026-08-15, $29.26 against the console's $28.82 -- the published figures were right to 1.5%. The per-leg split stays permanently lost (monitoring is per model per day and both efforts ran the same day), so the console table stands as the only per-leg record. The method validated itself on the one leg that did log: the 2026-08-18 Sonnet re-run appears in monitoring as 12,594/480, token-for-token identical to its `usage_log.jsonl` record. **The ledger breakdown summed to 114 under a heading saying 122.** Only the total was updated when the eight files landed; the per-leg rows were not. Fixed, and `test_the_ledger_count_matches_the_directory` now asserts the decomposition sums to the directory, not just the total -- the total alone is not a ledger. **The coverage matrix still said "annapolis only" and "the other nine splits have not been run".** That is the device this repo uses to keep an omission distinguishable from a withheld result, so a stale entry there does real damage. Restated per leg, with the $57 estimate reconciled against the measured $70.41 (~$8.80/split for Opus). **The cost row double-counted the smoke panos.** Read literally, "eight splits + 4 smoke panos" implies 988 panos, which would put the input-geometry check 0.42% out rather than the 0.02% the finding rests on. Both smoke records carry `bundle: richmond` and the export covers all 124 richmond panos, so those four are inside the 984 and were billed once; the ledger's $0.31 is a subset of the $70.41, not a line to add. Also states the residual precisely -- billed is 2,031 x 5,903, one view short of the 5,904 the geometry demands, unexplained -- and adds an independent confirmation: the annapolis leg's $8.94/125 panos scales to $70.4. **"Does not generalise" was doing more work than the data supports.** Per split the two models trade wins: Opus takes three of the seven pooled splits (four of nine overall) and the deficit is carried by gainesville and richmond. The annapolis margin was smaller than four of the seven per-split gaps, so the transferable result is that a single-split margin under ~0.07 F1 is unresolved until pooled -- stronger than the ranking correction, and stated as the headline now. Also: the nine-split `--verify` command is written down in both docs (it was claimed but not reproducible); "highest-recall chat VLM on the board" is qualified, since opus/high reaches 0.656 on annapolis four paragraphs below; and `test_the_annapolis_displacement_does_not_survive_pooling` pins the whole result, which was asserted nowhere -- the doc-currency tests only catch a forgotten regeneration, not a silently regenerated wrong number. Corroborating #143 while recovering the spend: `REPO_ROOT` derives from `__file__`, so a worktree breaks the read side too -- `vertex_usage.py` cannot find the repo-root `.env` and exits asking for `--project`. Same root cause, both directions. pytest 1,304 passed / 1 skipped; `scoreboard.py --check` green; `export_model_cache.py --verify` 9/9 identical, re-run independently against `.model_cache`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…#139, #143) The previous commit said the per-leg split was "unrecoverable, and always will be", because Cloud Monitoring reports per model per day and both efforts ran the same day. That was wrong for the same reason the original "cannot be recovered" was: it named a limit of the default query, not of the data. Daily alignment is a query parameter. Re-query at 60s and the legs separate. There is no `effort` label -- effort is a request parameter and never reaches the metric -- so the split rests on two facts the repo already holds. Input is deterministic (12,186 tokens per Opus pano, 6 views x 2,031), so total input pins the pano count exactly: the 08-15 Opus day is 251.00 panos, two 125-pano legs plus the one re-run pano, and the input half divides by geometry with no inference. Effort bills as output, so the high leg has both a higher output ratio and lower throughput. One unknown remains, and the minute series shows the predicted shape: the legs ran CONCURRENTLY, holding ~5 panos/min until 18:32 UTC, then dropping 2.54x to ~1.7 as the output ratio doubles (0.0675 -> 0.1203) -- low finishing and leaving high alone. anchor low high sum tail ratio = pure high (0.1203) $9.21 $12.20 $21.41 low ratio = 0.0349, from the #139 leg $8.95 $12.47 $21.41 console output, recorded at run time $8.94 $12.46 $21.40 The rate anchor reproduces the run-time record to 0.1% from a wholly independent source. The anchor came from the #139 leg's measured output rate before either number was compared, so that is a check and not a fit. The docs still quote $8.94/$12.46 -- the run-time record -- with this as corroboration. **Sonnet does not separate, and the script refuses rather than guessing.** Its ratio is flat across the whole run: throughput drops only 1.63x and the ratio moves the WRONG WAY (0.0365 -> 0.0281), because its high leg spent 17,820 thinking tokens against Opus's 127,227. A mixture solver handed that series returns "high effort cost less than low", which is false and entirely plausible-looking, so `MIN_RATIO_LIFT` makes it print NOT SEPARABLE instead. That is the transferable result: a per-effort split is recoverable exactly when effort changed the model enough to be worth splitting, and the telemetry is not the limiting factor. New: `scripts/analysis/vertex_effort_split.py` (read-only, same ADC as `vertex_usage.py`) and `tests/test_vertex_effort_split.py`, which pins the changepoint detector and the refusal on synthetic series whose answer is known by construction -- the cloud half needs credentials, the decision half does not. Also corrected: the guards paragraph no longer implies a warning could not have helped. Everything here was reconstructed five days late and worked only because nobody waited six weeks; which case you are in is unknowable until after the money is spent, so layer 1 is still the only layer that always works. pytest 1,309 passed / 1 skipped; scoreboard --check green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #139.
claude-opus-5ateffort=lowran the eight remaining city splits (984 panoramas). Thispublishes the detections, scores them, and answers the question the issue was opened to
settle.
The annapolis displacement does not generalise
claude-opus-5(low)gemini-3.1-pro-previewThe one split that made this worth running was the one split that flattered it. annapolis is
the survey-grade Trimble MX7 city — the least representative rig in the benchmark — and the
issue pre-registered that as the risk.
gemini-3.1-proremains the best challenger andRampNet's lead is unchanged at +0.219 F1.
The shape survives even though the ranking doesn't, and it is the more useful result
Opus trades −0.083 precision for +0.041 recall, which makes it the highest-recall chat
VLM on the board (0.614, above every Gemini leg). It loses on the aggregate and wins on the
axis
operating_point.mdsays to optimise. The scoreboard's own third finding is "precisionis not the differentiator; recall is" — so quoting the ranking alone gets the direction right
and the reason wrong. It is written up as finding 4 under the headline table rather than as a
footnote, for that reason.
budapest replicates as Opus's collapse split (0.378, worst by a wide margin), consistent with
every other chat VLM. paterson is its only precision-strong split (0.752).
standingstaysFalse, deliberatelyroster.pyraises at import for any pinned leg that is standing — a scored entry has to bewhat a bare
--modelsspec reproduces, and both efforts ofclaude:claude-opus-5share onespec. It is also unnecessary:
standinggates the roster tables (fp_taxonomy,null_recall,the frozen
WITNESS_POOL_46), not the scoreboard, which keys off coverage. Opus entered theheadline table on regeneration by itself.
gemini-3.7-flashand the three YOLO pano arms sit inexactly the same position. Recorded in the doc's caveat list so the next reader doesn't try the
flip and hit the assertion.
Cost: $70.41, and layer 1 didn't record it
The run wrote no row to
analysis_out/usage_log.jsonl. Recovered from Cloud Monitoring thenext day (11,988,993 in / 418,503 out) and written up in
docs/model_comparison.md§ "Costaccounting". Reported on #143, which had assumed the paid-API half needed no further work — the
--usage-logpath derives fromREPO_ROOT, so a leg run from a scratch worktree writes itsledger inside a directory that then gets deleted.
Two findings that generalise, both in the doc:
tokens/pano, so 984 panos predicts 11,991,024 against 11,988,993 billed — 0.02% out,
re-derivable from the committed detections with no cloud access. The same extrapolation for
output was 62% high.
successful pull. Retention is ~6 weeks.
Verification
export_model_cache.py --verify— 9/9 pairs score IDENTICALLY to the cachescoreboard.py --check— green (it went red the moment the detections landed, mid-branch)pytest -q— 1,303 passed, 1 skippedThree tests failed on this change and all three were right to. Two encoded "Opus low is a
single-split leg" and now assert the opposite, with a note that a reappearance means lost
coverage rather than a test needing a nudge. The third is the published-corpus ledger in
replication.md, which exists because that count silently drifted four times: 114 → 122 files.Not in scope
manual_goldwas not run, and that is a decision rather than a pending task —gemini-3.1-prohas no
manual_goldrow either, so a Claude-only run there would have no peer to compareagainst. Stated in full in #144.
🤖 Generated with Claude Code (claude-opus-5[1m])