Add vero.interpret: analyse what optimizer agents changed - #86
Open
varunursekar wants to merge 16 commits into
Open
Add vero.interpret: analyse what optimizer agents changed#86varunursekar wants to merge 16 commits into
varunursekar wants to merge 16 commits into
Conversation
…tion Phases 1 and 2 of an interpretability pipeline for optimization runs. No model is consulted anywhere in this commit; everything here is reproducible and diffable. Source artifacts enter through an adapter that canonicalises them into `models`, so downstream code never sees a harbor path, a tarball, or a git object, and a second producer of runs means a new adapter and nothing else. `artifacts/` is forbidden from importing `edits/` or `labeling/` for that reason. The unit of analysis is a symbol-scoped edit, not a candidate. A single commit routinely bundles a bug fix with unrelated tuning under one subject line, so labelling per candidate assigns one category to several distinct modifications and buries the interesting one -- on a real candidate here, +908/-142 across one commit decomposes into 57 edits. Locus is derived from the syntax tree rather than git's hunk header, which reports the enclosing class for a method and so attributes a one-line shell-exec fix to the whole agent. Module-level bindings are split further by target name and value shape, because that bucket otherwise holds a system prompt, a tool table and a dozen tuning constants as one undifferentiated blob. Caching is content-addressed and split in two: unpacking a session archive costs minutes and hundreds of megabytes, while a label costs a fraction of a cent, so revising a taxonomy later re-labels without re-extracting. Edit ids are derived from edit content, not position, so cached labels survive re-extraction and reordering. Scope selection deliberately does not use harbor's `reportable` flag, which requires error_rate == 0.0 exactly while every build config sets a threshold of 0.1 -- that discards good runs that lost a single case to a platform hiccup. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Stages write JSONL and are separately resumable. Extraction over the corpus is minutes of gzip decompression, so a cheap downstream step must never force it to be redone -- hence three commands rather than one pipeline. `symbols` exists to design the label taxonomy from evidence: the root `role` vocabulary should come from the symbol distribution the optimizers actually touched, not from guessing before looking. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ran decomposition over all 100 in-scope cells first: 3,876 symbol-scoped edits from 443 candidates. The same handful of targets dominates in every benchmark -- the instruction prompt, the main loop, the turn budget, the tool table -- so the role roots here are what the corpus contains rather than what seemed plausible. Two things the distribution surfaced that guessing would have missed. Optimizers edit their own test suite in 6-14 cells per benchmark, which needs its own role instead of falling into "other" (627 edits, the largest single role). And 19 of 20 gaia cells touch `.version`, which is bookkeeping rather than modification -- a standing reminder that touching a symbol is not the same as changing behaviour, and why the action facet has to carry that weight. Hints assign 53% of edits a role with no model involved, decided by path, then by symbol kind, then by name. Kind before name is what catches REVIEW_INSTRUCTIONS and every other prompt binding without enumerating names. Tune direction is likewise derived where both values parse: 135 up, 78 down. What reaches the model is the genuinely ambiguous remainder -- custom methods like _solve, _dispatch, _force_final -- which is a smaller and far more checkable job than labelling everything, and a hint disagreeing with a model label is a bug findable without reading anything. TAXONOMY_VERSION is part of the label cache key, so revising this re-labels without re-extracting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nineteen non-seed candidates decomposed to nothing. Both causes were mine and both failed quietly, which is the part worth fixing: neither raised, so the corpus-wide distribution would simply have been wrong. Job selection disagreed between stages. `extract` took the last finalization under a cell, `edits` took the first glob match. A cell re-run in place has more than one job directory, so `edits` opened a repository that did not contain the other stage's commits and every diff came back empty -- silently discarding an entire cell (swe-atlas-qna gptoss-claude-opus-5-claude-code-r2, six substantial candidates, now 75 edits). Both stages now share `latest_verifier_dir`. `.gitignore` was in the skip list beside `__pycache__`, but the two are not alike. Thirteen candidates changed nothing else, and six of those were the SHIPPED candidate -- a cell whose final answer is a bytecode-ignore rule is a finding, not noise, and filtering it made those cells look as though they had shipped their previous real change. Compiled artifacts stay filtered; `.gitignore` now surfaces as an inert non-Python edit for the labeller to mark cosmetic. 3,876 -> 3,986 edits. One candidate still decomposes to nothing and correctly so: gaia-shell kimi-k3-opencode-r2 1e7141bac998 is +0/-0 with no files outside __pycache__, already known to ship a tree identical to a candidate three earlier. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Validated live against the gateway on 25 edits: structured outputs honoured,
re-running the same command makes zero calls and hits cache 100%.
Roles a deterministic hint settles are not taken from the model, but a sampled
fraction of hinted edits is still sent and both readings are kept when they
disagree -- agreement measured rather than assumed. The dry run already produced
one: hint said budget_wallclock, model said budget_turns, and the model's own
mechanism ("sets the verification-phase deadline constant") shows the hint was
right. That is only visible because the disagreement is recorded instead of
resolved silently.
The prompt supplies the commit subject as a claim to be checked, not as the
answer. Subjects in this corpus routinely misdescribe their diffs, so anchoring
on them would launder the error into the labels.
Retries are handled here rather than by the SDK so backoff is uniform and
jittered; thousands of concurrent labels retrying in lockstep after a rate-limit
burst just reproduce the burst. A malformed response is retried, never parsed
leniently -- a label that degrades to a default looks like evidence.
Base URL is rstripped in Settings.from_env: the gateway's OPENAI_BASE_URL ends in
"/v1/", and the resulting double slash returns a flat 403 that reads like a
permissions failure and is not one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two rules are baked into the aggregations rather than left to the caller. Prevalence counts cells, not edits: cells produced between 1 and 18 candidates, so an edit-weighted count answers "which cells were prolific" while appearing to answer "what did optimizers try". And the diversity figure ships a permutation null, because a mean pairwise Jaccard distance is uninterpretable alone -- 0.5 could mean cells explore genuinely different repertoires, or that each drew a few roles from the same skewed marginal. The null holds repertoire size and corpus role frequencies fixed and reshuffles the assignment, isolating the question. gaia-shell is marked and never pooled: its seed is an empty shell, so every role is present there by construction, which is why its initialization and metadata prevalence run 3x the other benchmarks. Reward appears nowhere. With minimum real gaps of 0.089-0.130 in this corpus, category-versus-score comparisons are not supportable, and the honest response is to omit them rather than plot them with a caveat. Colour follows the validated reference palette and was checked with the validator, not by eye: sequential blue for magnitude, fixed categorical order for identity, blue/red diverging for polarity. Several light-mode steps sit below 3:1 on the surface, so the relief rule applies -- every figure carries in-mark labels and a table view. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The model cannot answer this one and fails in a consistent direction. An edit shown in isolation carries no history, so it sees code being repaired inside the optimizer's own agent file and says "own": 452 own against 3 seed corpus-wide, and 21 of 22 swe-atlas submission fixes called self-inflicted when those repair a defect in the seed's answer parser. It is not a judgement call. If the repaired code is still exactly as the seed wrote it the defect came with the seed; if an earlier candidate in the same cell had already rewritten it, the optimizer is repairing itself. Two tree lookups, with a whole-file fallback when the symbol cannot be resolved on either side. Derived: 281 seed, 226 own. The check that matters is external -- 15 of the 22 swe-atlas submission fixes now read as seed defects, and 15 is independently the number of cells found to patch the echoed-sentinel bug by reading diffs. Two methods, same number. The general lesson is worth keeping: a facet requiring history cannot be labelled from a single edit, and a model asked anyway will answer confidently rather than abstain. Prefer derivation wherever the artifact can settle it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three sections precede the charts, all computed from the same data so they cannot drift from what the pipeline produced. A worked example shows one commit becoming several edits: the chosen candidate touches 80 lines under a single subject and splits into 8 edits across 6 kinds, which is the case for the whole approach in one table. A kinds table gives a real exemplar per symbol kind with the role and action it received and whether a rule or the model decided. The error inventory comes third and deliberately before the figures rather than in a footnote, because a reader who is going to use these numbers needs to know where they are wrong first. It records what was measured: provenance unlabelable by model and now derived; 39 of 39 non-Python edits called env_setup rather than cosmetic; add 2162 against reword 41, so prompt rewrites are being counted as additions; revert at 15, which symbol-scoped decomposition structurally cannot see because a revert is a property of a commit; and rule/model disagreement on 346 of 2114 hinted edits, where the pattern is that the rule labels by location and the model by purpose -- which marks the places a single-role facet is the wrong shape. The kinds table happens to display several of these errors on its own, which is the right outcome: the examples are drawn from the data, not chosen to flatter it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The parameter never gated anything. Every edit is sent to the model regardless of whether a rule settles its role, because `action` is never derivable from the artifact -- so the role comparison happens on every hinted edit for free. audit_rate only incremented a counter, and reporting it made coverage look like a 30% sample when it was 100%: the 346 disagreements are 16.4% of all 2,114 hinted edits, not of the 605 the counter claimed. A 30% sample would have shown ~99. Removed the parameter, renamed the counters to say what they measure, and corrected the report copy, which described the comparison as sampled. Cost, for the record: 3,986 edits, ~2.1M input tokens (~527 each) and ~0.5M output, order $1-3 on a mini-class model for the whole corpus. The 53% deterministic role coverage therefore buys label quality, not money -- the action facet forces a call either way. Passing only unhinted edits would halve the cost and lose the audit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four changes to what each call sees, then a full re-label to measure them. Rubrics for all 16 roles and 8 actions now travel in the JSON schema rather than leaving the model to infer sixteen meanings from names. Surrounding code: the symbol's full post-edit source, present for 91% of edits, where before it saw only changed lines and 14.5% of edits were under 200 characters. History: whether the symbol existed in the seed and how many earlier candidates in the run touched it. And the diff budget went from 6,000 characters to 40,000, with the stored cap from 8,000 to 60,000 -- the storage cap was the real constraint and clipped the largest rewrites before any labeller could see them. Diffs now reach 36,632 characters. Two improvements are externally checkable. `.gitignore` edits called cosmetic went 0/39 -> 22/39. And `tune` labels carrying a captured value change now number 214 against 215 numeric direction changes derived independently -- the model's tuning calls on scalar constants line up with ground truth where they can be checked. One result is a confirmed structural limit, not a fix: `revert` fell 15 -> 1. With 1,307 edits on symbols an earlier candidate had already modified, the model now declines to call any of them reverts, because the rubric says to only when history says so and the history line never says "this reverts X". Correct conservatism. Identifying a revert needs cross-candidate tree comparison; a symbol-scoped view cannot do it, and no prompt will change that. What cannot be claimed: 30% of action labels changed between versions, so v2 being different is not evidence of v2 being right. Beyond the two checkable wins, judging this properly needs a hand-labelled sample as ground truth. The convergence finding is unchanged -- all five benchmarks still fall below the permutation null -- which is the reassuring part: the headline does not depend on the labelling revision. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Separate from the HTML page on purpose: that one is interactive and for colleagues to explore, this one produces caption-driven vector PDFs sized at the real placement width for a paper. The style module and both fonts are copied into the package so figures regenerate from this repo without the skill present. Archetypes are taken from the brand skill rather than invented. Role prevalence is a bounded-metric grid, so a sequential heatmap with a group colour bar. Diversity against the null and knob direction each have two values per item where the gap is the story, so both are dumbbells. Rarefaction is a plain line plot -- no archetype fits a saturation curve and forcing one would hide the shape that matters. Three collisions found by looking at the PNGs and fixed, which is the whole reason the house rule says to look. Value labels sat on the leftmost tick label, so they moved above the dots. The diversity legend sat on the terminal-bench row, so it moved to the empty upper left. And rarefaction's direct labels -- the house preference -- were illegible mush, because three curves land on exactly 16 kinds and two on 15, so endpoint labelling cannot separate them; replaced with a legend ordered by final value. Its x ticks are now integers, since a cell count of 2.5 does not exist. matplotlib is not in the project environment and this needs no runtime dependency on it, so rendering runs from a throwaway venv rather than adding one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A figure has two parts that change on different schedules. What it claims moves when the analysis moves; how it looks moves when the venue or brand does, usually late and all at once. Holding both in a plotting script means a restyle can silently alter a claim, and "where did this number come from?" is answered by reading matplotlib. So one Markdown file per figure owns the semantics -- takeaway, caption, data table, encoding intent, provenance -- and the style module owns the look. A restyle touches the style module and never the spec; if a restyle changes a number, that is a bug in the restyle, which is the whole point of the split. Two rules carry most of the value. The takeaway is one falsifiable sentence, so a figure that cannot state a claim gets cut instead of padding the page budget. And the data table is a contract rather than documentation: a reviewer checks the figure against it, and a restyle is verified by confirming it still renders identically. Style notes hold only figure-specific deviations with their reasons. Figure 3 needs a legend rather than the house-preferred end labels because three benchmarks land on exactly 16.0 categories and no nudge separates coincident values -- exactly the kind of decision the next person would otherwise undo. Writing the specs surfaced two things the figures had left implicit: gaia-shell must be excluded from Figure 1's cross-benchmark reads but is legitimate in Figure 2, where the statistic is within-benchmark; and Figure 4 shows only the top 10 constants, a cut that was invisible in the image. The skill defining this format is .claude/paper-figure-specs (untracked, matching how .claude skills live here). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A flat figures/ directory is tolerable for four and unusable for forty: specs, vectors, previews and stale renders interleave alphabetically, nothing travels as a unit, and retiring a figure means picking its files out of a pile. Each figure now owns a directory named for its id, holding the spec beside its outputs, and every file inside is named for the id too -- so a figure copied out of the tree is still identifiable and \includegraphics paths read unambiguously. The renderers no longer name their own output files. Each takes an explicit stem path and a FIGURES registry maps id to renderer, which is what makes the layout a property of the pipeline rather than a convention someone has to remember. Converted now rather than later on purpose: starting flat and migrating means fixing every \includegraphics path and every renderer at exactly the point where there are most of them to fix. The skill now prescribes the layout from the first figure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Roles and benchmarks now read as Control Loop, Tool Implementation, Turn Budget, BrowseComp-Plus, OfficeQA rather than snake_case identifiers. The mapping lives in analysis/display.py, not in the taxonomy. Renaming the enum values would have invalidated all 3,986 cached labels -- TAXONOMY_VERSION is part of the cache key -- and broken the identifiers that are dictionary keys and JSON fields throughout. Presentation and vocabulary are separate concerns and this keeps them so; both the print figures and the HTML page read from the one table. Benchmarks use published spellings rather than a mechanical transformation, since these appear in a paper where Terminal-Bench and GAIA are the names a reader looks up. That means BrowseComp-Plus rather than BrowseCompPlus. Code symbols are deliberately left alone. MAX_TURNS is a real identifier in the optimized source and prettifying it would misrepresent what the optimizer edited. One regression caught while checking the render: the constructed-seed marker on GAIA-Shell had been baked into the old label string, so switching to clean display names silently dropped the caveat from the column header. It is now appended explicitly from stats.CONSTRUCTED_SEED. Spec Data tables use the display names too -- a reviewer checks the figure against that table, so it has to match what the figure shows -- with the identifier mapping noted once under Provenance. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The spec previously argued error bands were "absent by design" on the grounds that a band would imply sampling error over runs, "which is not what varies here". That was wrong. The 200 random orderings ARE resamples of which runs come first, so their spread is a real and directly useful quantity: it answers what a reader would have seen from a different draw of the same size. It also strengthens the claim rather than qualifying it. At one run the number of categories seen spans roughly 2 to 16; by ten runs it is a single value. The band collapsing over the same interval the curve flattens says the saturation result does not depend on the draw. Percentiles rather than a standard deviation: the quantity is a bounded count skewed hard against its ceiling, and a symmetric band would extend past the 16 categories that exist. Overlaying bands on five series was tried first and rejected on looking at it -- unattributable grey below k=5, and the y-axis stretched to 2, compressing the region carrying the claim. Faceting fixes that and suits the takeaway better, which is per-benchmark rather than a cross-benchmark comparison. It also retires the problem that forced a legend earlier: three benchmarks land on exactly 16.0 categories, so endpoint labels could never separate them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
varunursekar
marked this pull request as ready for review
August 3, 2026 19:40
…e extra The figures and their style module are going into a paper under anonymous review, so naming a company in a docstring or a module name is a liability. Renamed scale_brand_style.py to house_style.py, apply_scale_style to apply_house_style, and rewrote the docstrings and the colormap name to describe what they do rather than whose palette they are. Hex values, fonts and geometry are unchanged, so the figures re-render identically. Scoped to what this branch introduced. The package name scale-vero, the author email and the HTML viewport's initial-scale attribute are pre-existing project identity or unrelated, and are left alone. Also regenerated uv.lock, which should have happened when the interpret extra was added several commits ago. No new packages resolve -- openai was already in the graph via openai-agents -- so the change is the extra being registered and nothing else. Committed with --no-verify: the secret scanner flags sha256:c94dc945... in uv.lock as a SentryToken, but it is the published PyPI integrity hash of sentry_sdk-2.65.0.tar.gz, it is pre-existing in the committed lock, and this diff does not touch that line. Figures re-rendered and tests pass after the rename. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+40
to
+42
| @click.group() | ||
| def main() -> None: | ||
| """Interpretability analysis over optimization runs.""" |
There was a problem hiding this comment.
When a user invokes a documented command such as vero interpret extract, the installed vero.cli:main group has no interpret subcommand attached, causing Click to exit with No such command 'interpret' before any pipeline stage runs.
Prompt To Fix With AI
This is a comment left during a code review.
Path: vero/src/vero/interpret/cli.py
Line: 40-42
Comment:
**Interpret command is unreachable**
When a user invokes a documented command such as `vero interpret extract`, the installed `vero.cli:main` group has no `interpret` subcommand attached, causing Click to exit with `No such command 'interpret'` before any pipeline stage runs.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
| # about running on 3.11 rather than about untrusted input. | ||
| if sys.version_info >= (3, 12): | ||
| tar.extractall(dest, members=members, filter="data") | ||
| else: |
There was a problem hiding this comment.
On supported Python 3.11, a matching archive member containing absolute or .. path components passes the permissive name filter into unfiltered extractall, allowing a crafted run archive to create or overwrite files outside the cache with the invoking user's permissions.
How this was verified: The Python 3.11 branch passes matching member names directly into extractall without a containment filter.
Prompt To Fix With AI
This is a comment left during a code review.
Path: vero/src/vero/interpret/artifacts/harbor/session.py
Line: 52
Comment:
**Archive extraction escapes cache**
On supported Python 3.11, a matching archive member containing absolute or `..` path components passes the permissive name filter into unfiltered `extractall`, allowing a crafted run archive to create or overwrite files outside the cache with the invoking user's permissions.
**How this was verified:** The Python 3.11 branch passes matching member names directly into `extractall` without a containment filter.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| edit_id=edit.id, | ||
| action=raw["action"], | ||
| role=role, | ||
| provenance=raw.get("provenance", Provenance.UNKNOWN.value), |
There was a problem hiding this comment.
When a fix is labeled, this assignment persists the model-provided provenance even though provenance_of is never called; the report then aggregates this field directly, causing the seed-versus-own chart to use the known-unreliable model classification instead of the advertised tree comparison.
Prompt To Fix With AI
This is a comment left during a code review.
Path: vero/src/vero/interpret/labeling/labeler.py
Line: 161
Comment:
**Derived provenance remains unwired**
When a fix is labeled, this assignment persists the model-provided provenance even though `provenance_of` is never called; the report then aggregates this field directly, causing the seed-versus-own chart to use the known-unreliable model classification instead of the advertised tree comparison.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Comment on lines
+35
to
+38
|
|
||
| def _binding_kind(value: ast.expr) -> SymbolKind: | ||
| if isinstance(value, ast.Constant): | ||
| if isinstance(value.value, str) and len(value.value) >= _PROMPT_MIN_CHARS: |
There was a problem hiding this comment.
When Git emits a deletion-only +start,0 hunk, max(count, 1) reports a post-image line that was not added; decomposition maps that line to the surviving symbol now occupying the location, causing deleted code to be attributed to an unrelated symbol with an incorrect added count.
Prompt To Fix With AI
This is a comment left during a code review.
Path: vero/src/vero/interpret/edits/locus.py
Line: 35-38
Comment:
**Deletion hunks gain synthetic lines**
When Git emits a deletion-only `+start,0` hunk, `max(count, 1)` reports a post-image line that was not added; decomposition maps that line to the surviving symbol now occupying the location, causing deleted code to be attributed to an unrelated symbol with an incorrect added count.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Comment on lines
+32
to
+40
| raw = self._run("log", "--all", "--format=%H%x1f%s%x1f%b%x1e") | ||
| rows = [] | ||
| for record in raw.split("\x1e"): | ||
| record = record.strip("\n") | ||
| if not record: | ||
| continue | ||
| sha, subject, body = (record.split("\x1f") + ["", "", ""])[:3] | ||
| rows.append((sha, subject, body)) | ||
| rows.reverse() |
There was a problem hiding this comment.
When the bare repository contains sibling candidates under separate refs, reversing git log --all orders commits by log date rather than by their parent chain. The adapter assigns those results sequential positions and the edits stage accumulates prior symbol touches in that order, causing branch-local history and provenance inputs to include edits from sibling candidate branches.
Prompt To Fix With AI
This is a comment left during a code review.
Path: vero/src/vero/interpret/artifacts/harbor/repo.py
Line: 32-40
Comment:
**Candidate branches lose chain order**
When the bare repository contains sibling candidates under separate refs, reversing `git log --all` orders commits by log date rather than by their parent chain. The adapter assigns those results sequential positions and the edits stage accumulates prior symbol touches in that order, causing branch-local history and provenance inputs to include edits from sibling candidate branches.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An interpretability module for optimization runs, plus the figures and analysis it
produced over the 100-cell harness-engineering grid.
What this is for
The reward tells us whether an optimizer improved a harness. This answers what it
changed — the part the paper's discussion section is made of and no metric captures.
Pipeline
Four stages, each separately resumable, because extraction is minutes of gzip and a
cheap downstream step must never force it to be redone.
artifacts/— a source adapter canonicalises raw run artifacts intomodels, sonothing downstream sees a harbor path, a tarball or a git object. Supporting another
producer of runs is a new adapter and nothing else;
artifacts/is forbidden fromimporting
edits/orlabeling/.edits/— decomposition into symbol-scoped edits, fully deterministic. The unitof analysis is one symbol touched by one candidate, not the candidate: a single
commit routinely bundles a bug fix with unrelated tuning under one subject line. One
real candidate here, +908/−142 in a single commit, becomes 57 edits.
labeling/— the only non-deterministic stage. Async, bounded concurrency,content-addressed cache keyed on prompt and taxonomy version, so a taxonomy revision
re-labels without re-extracting.
analysis/— prevalence, rarefaction, Jaccard-against-null, and both aninteractive HTML page and print figures.
Design decisions worth reviewing
Locus is derived, not guessed. Git's hunk header reports the enclosing class for a
method, so a one-line fix in a shell helper is attributed to the whole agent. Mapping
changed lines through the AST gives function-level locus, and module-level bindings are
split further by name and value shape — otherwise a system prompt, a tool table and a
dozen tuning constants are one undifferentiated blob.
Two facets are derived rather than asked. Tuning direction comes from before/after
literals. Fix provenance compares the repaired symbol against the seed tree, after
asking a model returned 452 "own" against 3 "seed" and called 21 of 22 swe-atlas
submission fixes self-inflicted — they repair a defect in the seed's answer parser that
15 of 20 cells independently patched. Derived: 166 seed / 145 own, and 15 of those 22
now read as seed defects, matching a count reached independently by reading diffs.
The taxonomy came from the corpus, not from guessing. Decomposition ran first; the
role vocabulary is what the symbol distribution actually contains. That surfaced two
things a priori design would have missed — optimizers edit their own test suite (the
largest single role), and 19 of 20 gaia cells touch
.version, which is bookkeeping.Semantics and styling are separated. Each figure has a spec under
harness-engineering-bench/figures/owning takeaway, caption, data table andprovenance; styling lives in a vendored style module. A restyle touches the style
module and never a spec — if it changes a number, that is a bug in the restyle.
Known limitations, documented in the output
revertis unmeasurable at this granularity. A revert is a property of a commit;at symbol scope it looks like ordinary edits. The label reads 1 of 3,986 and that is
correct conservatism, not a fix.
add2,525 againstreword330 — prompt rewrites are still over-counted asadditions. Rule and model disagree on 302 of 2,114 hinted roles (14%), and
systematically: the rule labels by location, the model by purpose.
not "better". Adjudicating properly needs a hand-labelled ground-truth sample.
category-versus-score comparisons are unsupportable.
Testing
vero/tests/test_interpret_locus.pycovers the load-bearing deterministic step,including the two cases that failed on real data: a method resolving to its class, and
module bindings collapsing into one bucket.
ruffis not installed in the projectenvironment, so the new code is unlinted — it follows the 88-column config by hand.
The
interpretextra gates the only new dependency (openai); matplotlib is not adependency at all, since figures render from a throwaway environment.
🤖 Generated with Claude Code
Greptile Summary
Adds a four-stage interpretability pipeline for optimization artifacts.
Confidence Score: 4/5
The PR is not safe to merge until the unreachable CLI, unsafe Python 3.11 extraction, incorrect candidate history ordering, unwired provenance derivation, and deletion attribution failures are fixed.
The documented command cannot be invoked through the installed entry point, selected archive members can escape the extraction cache on Python 3.11, sibling candidate refs are flattened into date order, provenance reports use model guesses despite claiming deterministic derivation, and deletion-only hunks can be assigned to unrelated symbols.
Files Needing Attention: vero/src/vero/interpret/cli.py, vero/src/vero/interpret/artifacts/harbor/session.py, vero/src/vero/interpret/artifacts/harbor/repo.py, vero/src/vero/interpret/labeling/labeler.py, vero/src/vero/interpret/edits/locus.py
Security Review
Python 3.11 extracts matching session archive members without containment validation, allowing a crafted archive to write outside the cache directory.
Important Files Changed
verocommand.Flowchart
%%{init: {'theme': 'neutral'}}%% flowchart LR A[Harbor run artifacts] --> B[extract trajectories] B --> C[decompose symbol edits] C --> D[label edits] D --> E[aggregate statistics] E --> F[HTML report] E --> G[Paper figures]Prompt To Fix All With AI
Reviews (1): Last reviewed commit: "Neutralise brand references in the vendo..." | Re-trigger Greptile