data(retrieval): score artifacts and public corpora - #176
Merged
Conversation
The evidence behind the published tables, so that verifying them does not require
re-running hours of measurement. 27 score artifacts from the single-project field
measurement, each carrying its per-arm summaries and all 108 case rows, plus the
corpora for the public upstreams.
The tables in full-field-got.md regenerate from exactly these files:
node scripts/context-retrieval/field-report.mjs \
benchmarks/context-retrieval/results/full-field-got
Two artifacts are prefixed EXCLUDED- and kept deliberately. One is a void
measurement of an arm whose own cleanup command was disabled by config and failed
silently, so its shared graph accumulated 110 stale worktrees until 19 of every 20
results pointed into earlier cases. The other is an arm that needs a long-lived
per-repository server the per-case protocol cannot provide. Both are retained as
the record of a retraction rather than deleted, because a deleted zero is
indistinguishable from a zero that was never measured.
Every artifact reports its reliability gate as FAILED. This is a false alarm and
HANDOFF.md leads with it: arms were measured one per process so that a
42-second-per-query tool could not gate twenty-four others, which leaves no single
artifact holding the controls. The gates must be recomputed over the union, and
there they pass.
Private-repository corpora are not included. This repository is public and those
corpora carry commit subjects and file paths from private codebases.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sarthakagrawal927
added a commit
that referenced
this pull request
Aug 23, 2026
The evidence behind the published tables, so that verifying them does not require
re-running hours of measurement. 27 score artifacts from the single-project field
measurement, each carrying its per-arm summaries and all 108 case rows, plus the
corpora for the public upstreams.
The tables in full-field-got.md regenerate from exactly these files:
node scripts/context-retrieval/field-report.mjs \
benchmarks/context-retrieval/results/full-field-got
Two artifacts are prefixed EXCLUDED- and kept deliberately. One is a void
measurement of an arm whose own cleanup command was disabled by config and failed
silently, so its shared graph accumulated 110 stale worktrees until 19 of every 20
results pointed into earlier cases. The other is an arm that needs a long-lived
per-repository server the per-case protocol cannot provide. Both are retained as
the record of a retraction rather than deleted, because a deleted zero is
indistinguishable from a zero that was never measured.
Every artifact reports its reliability gate as FAILED. This is a false alarm and
HANDOFF.md leads with it: arms were measured one per process so that a
42-second-per-query tool could not gate twenty-four others, which leaves no single
artifact holding the controls. The gates must be recomputed over the union, and
there they pass.
Private-repository corpora are not included. This repository is public and those
corpora carry commit subjects and file paths from private codebases.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Last of the stack. Depends on #175 (the data-path exclusion) to pass the size gate.
What this is
The evidence behind the published tables: 27 score artifacts from the single-project field measurement, each carrying per-arm summaries and all 108 case rows, plus corpora for the public upstreams. 72 files, ~2.6 MB.
Committing it means verifying the numbers does not require re-running hours of measurement. The tables in
full-field-got.mdregenerate from exactly these files:I checked that output against the committed markdown: identical apart from one trailing newline.
Two artifacts are prefixed
EXCLUDED-on purposeEXCLUDED-code-graph-context-stale-db.json— a void measurement. That tool's own cleanup command is disabled by config and failed silently, so its shared graph accumulated 110 stale worktrees until 19 of every 20 results pointed into earlier cases' trees.EXCLUDED-seagoat.json— needs a long-lived per-repository server the per-case protocol cannot provide.Both are kept rather than deleted, because a deleted zero is indistinguishable from a zero that was never measured. The reporter skips
EXCLUDED-files.The thing that will confuse a reviewer
Every artifact reports
RELIABILITY GATE FAILED — controls absent. It is a false alarm, andHANDOFF.mdleads with it. Arms were measured one per process so a 42-second-per-query tool could not gate twenty-four others, so no single file holds the controls — they are in sibling files. Recomputed over the union, the gates pass. The union recomputation is what to verify.Not included
Private-repository corpora. This repo is public and they carry commit subjects and file paths from private codebases.
HANDOFF.mdsays how to rebuild them from local checkouts.🤖 Generated with Claude Code