Record time as well as money for every non-free run (#143) - #147
Open
jonfroehlich wants to merge 6 commits into
Open
Record time as well as money for every non-free run (#143)#147jonfroehlich wants to merge 6 commits into
jonfroehlich wants to merge 6 commits into
Conversation
…worktree (#143) Three defects in the cost accounting, one of them actively losing data. **Time was not measured at all.** No perf_counter anywhere in the comparison harness; usage_log.jsonl carried a completion timestamp and no duration, so the ledger could answer $/pano and never s/pano. The runtimes we have survive only as prose in docs/model_comparison.md. score_model now fills in a caller-owned timing dict -- model-load seconds, inference seconds, panos actually put through the model -- and main stops the clock in the finally, so a leg that dies partway still reports the time it spent. panos_called counts attempts, not successes: a call that raised burnt wall-clock and, on a paid provider, still billed. **Free legs left no record at all.** report_usage returned early when a detector had no usage dict, which is every local GPU model -- OWLv2, Grounding DINO, Qwen, Molmo, YOLO. Those are free in API terms and cost real GPU-hours, and they were invisible to the one ledger we have. Every leg that spends something now writes a row, with a `paid` flag and the host/GPU it ran on. Legs that spent nothing (a fully cached re-score, a leg that never loaded) still write nothing, and the `--models rampnet` arm is excluded outright: it replays committed detections, so its row would be a zero appended to a committed file on nearly every run. **The ledger could be written somewhere that does not outlive the run.** DEFAULT_USAGE_LOG derived from REPO_ROOT, which in a linked worktree is the worktree -- so a leg run from a scratch worktree wrote its ledger there and lost it when the worktree was removed. That is how the #139 claude-opus-5 leg spent $70.41 and left no row, recovered only because Cloud Monitoring still had it. #119's guard cannot see this: it proves a log path was accepted, not that the file survives. The default now resolves through `git rev-parse --git-common-dir`, which every worktree shares, so they all append to one canonical ledger; an explicit --usage-log pointing into a worktree warns; and each logged leg prints the absolute path plus the running total, so a run that logged somewhere unexpected is visible while someone is still watching rather than six weeks later when the provider's telemetry has aged out. Falls back to REPO_ROOT whenever git cannot answer (a tarball, an HF clone): bookkeeping must never be the reason a run refuses to start. 9 new tests, 171 passing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
) The API half of an experiment's cost has been recorded per run since #119. The compute half was not recorded anywhere -- Tillicum's $0.90/GPU-hour and klone's GPU-hours lived only as prose in docs/tillicum.md, written by hand, per job, when someone remembered. Unlike API tokens this half is partly back-fillable from `sacct`, which is the reason to build it now rather than later: the retention window is finite. scripts/analysis/slurm_usage.py parses `sacct` (live on a login node, or a saved dump via --from-file for a machine that cannot reach the cluster) into analysis_out/compute_log.jsonl. Verified against the two numbers docs/tillicum.md already carries, which were arrived at independently of this code: the data-prep job's 4.67 GPU-hours and $4.20 come back exactly. Three things it gets right that a hand tally does not: - **`-D`.** Slurm shows only the LAST incarnation of a requeued job by default, and our klone runs live on the preemptable ckpt partition -- the paper's Stage 2 run was 15 preemptions, 44.7 h of compute across 74.6 h of calendar. Rows are therefore keyed on (cluster, job id, start), not the job id, or all but the last incarnation's compute disappears. - **GPU-hours = elapsed x N GPUs**, which is how Tillicum bills: an idle GPU in a 2-GPU job costs exactly as much as a busy one. The generic `gres/gpu=N` and the typed `gres/gpu:a40=N` are the same GPUs reported twice, so the generic count wins rather than being added. - **Idempotent.** Re-running appends only what is new; a job first seen while RUNNING is re-appended once terminal, and readers take the last row per key. pricing.py gains COMPUTE_PRICING under the same verified-only discipline as the token table: rate, as-of date, source, and Slurm's per-QoS UsageFactor. Tillicum's `debug` entry carries the unresolved conflict with it -- Slurm bills that QoS at factor 0 while hyakusage charged the smoke job $0.03 anyway -- so nobody can quote a bare "free" from it without the source. An unpriced cluster returns None rather than $0, because "we checked and it is free" (klone) and "we have no rate" are different statements and only one of them is safe to put in a paper. canonical_repo_root and the JSONL read/append helpers move to rampnet/ledger.py so both ledgers share them -- otherwise the new script would have reintroduced the worktree defect it exists to help measure. 14 new tests; full suite green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ld (#143) The standing rule lived in docs/model_comparison.md, which reads as a convention of that one harness, and said nothing about time or about non-free compute. It is now a repo-level rule beside the replication and record-in-GitHub ones: both units, both cost types, recorded at run time, ledgers committed into the main checkout. docs/model_comparison.md's cost section is rewritten to match what the code now does -- the per-leg timing fields, the free-leg rows and the two legs that deliberately write nothing, the fourth (compute) ledger, and a subsection on the worktree defect that is precise about why the #119 guard could not catch it: it proves a log path was accepted, never that the file survives. Promoting the rule would not have caught it either, and saying so is the point. docs/tillicum.md now says hyakusage is a live view rather than a record, and points at the ledger. docs/replication.md lists compute_log.jsonl with its gap stated: the sacct back-fill has not been run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…#143) The comment on #143 is right that nothing in the plan caught the failure that actually happened. #119's guard proves a log path was ACCEPTED; it says nothing about whether the file survived, which is how the #139 leg billed $70.41 and left no row. Layers 1 and 3 measure the same spend two ways and until now nothing compared them. `vertex_usage.py --reconcile` totals the committed ledger per model over the same window as the metric query and prints them side by side. On the real #139 numbers it reports claude-opus-5 as MISSING with 11,988,993 input tokens unaccounted for, and says what to do about it: the metric retains ~6 weeks, recovery is per-model per-DAY, so per-split attribution is gone even on a successful pull. Deliberately asymmetric: billed > ledger is spend with no record and is called out; ledger > billed is odd but harmless. Tolerance 2%, because Cloud Monitoring's daily rows are 24 h windows ending at the query's time-of-day rather than calendar days, so a leg straddling the boundary moves either way. Free legs carry no token keys and are skipped -- they have no bill to reconcile against, and counting them as zero-token models would invent a MISSING verdict. The comparison functions are pure and tested against the real incident's numbers; nothing in the new tests touches the network or needs cloud credentials. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…worth 4.3x (#143) 3,991 job allocations since 2026-07-02, written from a real `sacct` pull while the klone control master was up. The raw dump is committed alongside, so every number in docs/compute_cost.md is re-derivable from a clean clone with no cluster account -- the same reason usage_log.jsonl is committed while vertex_usage.py needs cloud credentials. **`sacct -D` is worth 4.35x on this workload, and that is the finding.** Slurm reports only the last incarnation of a requeued job by default, and 96% of our allocations end in PREEMPTED (3,780 of 3,991) because the work lives on ckpt. The #51 YOLO baseline is 2,046.9 GPU-hours across 3,857 incarnations with -D, and 470.5 across 27 job ids without it. Any hand tally from a default sacct is short by that factor and nothing in the output says so. It validates the one number the repo already had rather than contradicting it. docs/tillicum.md records 496.5 GPU-hours on the baseline as of 2026-07-30; summing this ledger over jobs ENDING from 07-24, the running total crosses 496.5 at 2026-07-29T21:17 -- the evening before that figure was written. Reproduces to the hour, and confirms the original query was duplicate-inclusive. Two fixes found by running it for real: - --save-raw did not create its parent directory, so the replication input it exists to write failed on a fresh path. - Rows embedded the whole pricing entry, which at 3,991 rows was 1.7 MB of the 2.6 MB file. They now carry the rate and its as-of date; the table with its caveats stays versioned in pricing.py. - --cluster silently restamped rows whose Cluster column disagreed, which would price another cluster's jobs at the wrong rate and attribute their hours to the wrong machine. It now warns. (Nothing was mislabeled here: all 3,991 rows report klone, and the 456 H200 allocations are klone's own ckpt-g2 nodes.) The ledger records every job on the account, not only RampNet's -- 28 sal-*/arch_* allocations, 1.3% -- deliberately, so the artifact is a complete measurement and attribution is the reader's call rather than a filter chosen once and baked in. Gaps stated in the doc: Tillicum is NOT back-filled (Duo, master down), the window starts where this account's retention does, and the paper's own runs were on another user's account and are outside this query entirely. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The eight-split claude-opus-5 leg of 2026-08-18 spent real money and left no row, so every cost table built from usage_log.jsonl under-reported the benchmark's Claude spend by about 200x -- $0.34 against a true $70.44. Layer 3 recovered the number the next day; until now it existed only as prose on the #139 branch. The row is the billed total for that model-day MINUS what the two surviving richmond smoke rows already account for: 11,940,249 in / 415,751 out / $70.0950. Written that way the ledger sums to $70.4375 against a $70.44 bill, instead of double counting the smoke legs. The subtraction, the billed totals it came from and the recovery command are all in the row, so the arithmetic is checkable without cloud access. What made this worth more than an append is the direction the two requirements pull. Cost totals must INCLUDE recovered spend or they are wrong by the whole amount. Reconciliation must EXCLUDE it, because a recovered row was read off the same bill it would be reconciled against -- counting it as "logged" makes the bill agree with itself and reports ok for exactly the gap that check exists to find. So ledger_totals_by_model now skips kind == "recovered" explicitly rather than by the accident of which keys a row happens to carry, and ledger_totals reports the recovered share separately so it never reads as an as-run number. A recovered row carries no bundle and no elapsed_s. Recovery is per-model per-day, so which of the eight splits spent what is permanently gone; that limitation travels inside the row rather than in someone's memory. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #143 (except the Tillicum back-fill — see Blocked below).
We recorded money for paid APIs and nothing else. This adds the other three quadrants:
time for paid legs, time for free GPU legs, and money+time for cluster compute — and fixes
the defect the comment on #143 identified, which was actively losing data.
The bug that was live
DEFAULT_USAGE_LOGderived fromREPO_ROOT, which in a linked worktree is the worktree. Aleg run from a scratch worktree wrote its ledger there and lost it when the worktree was
removed. That is how the #139
claude-opus-5leg spent $70.41 and left no row.#119's guard cannot see this, and the reason matters: it proves a log path was accepted,
never that the file it wrote survives. The operator followed the rule and lost the record
anyway, so promoting the rule into
CLAUDE.mdwould not have caught it either.Both ledgers now resolve through
git rev-parse --git-common-dir, which every worktree of arepo shares, so they all append to one canonical file in the main checkout. An explicit
--usage-loginto a worktree warns; each logged leg prints the absolute path and therunning total; and it falls back to the local checkout when git can't answer (a tarball, an HF
clone) — bookkeeping must never be why a run refuses to start.
What else changed
Time is measured. There was no
perf_counteranywhere in the harness.score_modelnowfills a caller-owned timing dict (model-load / inference / panos actually called) and
mainstops the clock in the
finally, so a leg that dies partway still reports what it spent.panos_calledcounts attempts — a call that raised burnt wall-clock and, on a paid provider,still billed.
Free legs are no longer invisible.
report_usagereturned early whenever a detector had nousagedict — i.e. OWLv2, Grounding DINO, Qwen, Molmo, YOLO, the entire GPU half of theroster. They now write the same row with
paid: false. Two legs deliberately write nothing: afully cached re-score, and
--models rampnet, which replays committed detections rather thanrunning a model.
A compute ledger.
scripts/analysis/slurm_usage.py→analysis_out/compute_log.jsonl,priced from a new verified-only
COMPUTE_PRICINGinpricing.py. Back-filled from a realsacctpull: 3,991 allocations, 2,684.4 GPU-hours on klone since 2026-07-02, with the rawdump committed so it re-derives without a cluster account.
Reconciliation.
vertex_usage.py --reconcilecompares the committed ledger against CloudMonitoring per model. This is the only check that catches a silent no-write; on the real #139
numbers it reports
claude-opus-5MISSING with 11,988,993 input tokens unaccounted for.The rule is now repo-level, in
CLAUDE.mdbeside replication and record-in-GitHub: bothunits, both cost types, ledgers committed into the main checkout.
Finding:
sacct -Dis worth 4.35x96% of our klone allocations end in
PREEMPTED(3,780 of 3,991), so the defaultsacctview —last incarnation only — discards nearly everything:
sacct -D-DIt validates the one figure the repo already had.
docs/tillicum.mdrecords 496.5GPU-hours as of 2026-07-30; summing this ledger over jobs ending from 07-24, the running
total crosses 496.5 at 2026-07-29T21:17 — the evening before that line was written.
Blocked
it's up; the gap is stated in
docs/compute_cost.mdnext to the numbers, not left implicit.data/claude-opus-5-nine-splits-139. Worth deciding whether recovered totals belong inusage_log.jsonlmarked as layer-3 recoveries.Overlap
Touches the same section of
docs/model_comparison.mdasdata/claude-opus-5-nine-splits-139, in a different hunk (that branch narrates the incident;this one documents the mechanism and the fix). Should merge clean.
Tests
39 new, 1,334 passing, still CPU-only and network-free. The reconciliation tests use the real
#139 numbers; the compute tests are checked against
docs/tillicum.md's independently-derived$4.20 / 4.67 GPU-h.
🤖 Generated with Claude Code (claude-opus-5[1m])