Add vero.interpret: analyse what optimizer agents changed

harness-engineering-bench/figures/figure_01_prevalence/figure_01_prevalence.md

@@ -0,0 +1,85 @@
+---
+id: figure_01_prevalence
+archetype: heatmap
+script: vero.interpret.analysis.paper_figures::fig_prevalence
+outputs: [figure_01_prevalence.pdf, figure_01_prevalence.png]
+status: review
+---
+
+## Takeaway
+
+Every optimizer edits the instruction prompt and the control loop, but which other
+parts of the harness they touch is dictated by the benchmark, not by the optimizer.
+
+## Caption
+
+Share of optimization runs that made at least one edit of each kind, per benchmark
+(20 runs each, 100 total). Cell shading is the share; the annotation is the count.
+Prompt and control-loop edits are near-universal, while the remaining categories vary
+sharply by benchmark — submission-path edits appear in 15 of 20 swe-atlas-qna runs but
+2 of 20 terminal-bench runs, and retrieval edits appear almost only in
+browsecomp-plus, the sole benchmark with a retrieval corpus. Counted per run rather
+than per edit, since runs produced between 1 and 18 candidates and an edit-weighted
+count would measure verbosity instead of coverage. The left colour bar groups
+categories. \textsuperscript{*}gaia-shell's seed is an empty skeleton, so every
+category is present there by construction and its column is not comparable with the
+others. Reward is not shown; see \Cref{tab:signal-validity} for why score-versus-category
+comparisons are unsupportable in this corpus.
+
+## What the reader should see
+
+- Read down the first two rows first: `prompt` and `control_loop` are dark across every
+  column. That is the universal behaviour.
+- Then read across `submission`, `retrieval`, `tool_impl` — the variance between columns
+  is the finding. Benchmark, not optimizer, selects the target.
+- Shading encodes the same quantity as the annotation, deliberately: shading carries the
+  pattern at a glance, the fraction carries the exact read. Nothing else is encoded.
+- `gaia-shell` is marked and excluded from any cross-benchmark claim.
+- Absent by design: reward, and any ordering of benchmarks by score.
+
+## Data
+
+Cells that made ≥1 edit of each kind, out of 20 per benchmark. Rows ordered by mean
+share across benchmarks, which is the order the figure uses.
+
+| role | BrowseComp-Plus | OfficeQA | SWE-Atlas-QnA | Terminal-Bench | GAIA-Shell‡ |
+|---|---|---|---|---|---|
+| Prompt | 20/20 | 19/20 | 20/20 | 20/20 | 18/20 |
+| Control Loop | 17/20 | 19/20 | 18/20 | 17/20 | 20/20 |
+| Turn Budget | 15/20 | 17/20 | 16/20 | 18/20 | 11/20 |
+| Tool Surface | 15/20 | 7/20 | 15/20 | 10/20 | 18/20 |
+| Tool Implementation | 16/20 | 7/20 | 11/20 | 10/20 | 18/20 |
+| Other | 14/20 | 8/20 | 9/20 | 9/20 | 20/20 |
+| Tests | 11/20 | 7/20 | 14/20 | 10/20 | 16/20 |
+| Model Client | 11/20 | 14/20 | 10/20 | 8/20 | 11/20 |
+| Metadata | 9/20 | 7/20 | 6/20 | 11/20 | 19/20 |
+| Submission | 7/20 | 8/20 | 15/20 | 2/20 | 18/20 |
+| Output Cap | 5/20 | 13/20 | 11/20 | 7/20 | 7/20 |
+| Wall-Clock Budget | 5/20 | 7/20 | 7/20 | 6/20 | 12/20 |
+| Context Management | 7/20 | 7/20 | 6/20 | 10/20 | 7/20 |
+| Initialization | 6/20 | 4/20 | 4/20 | 5/20 | 17/20 |
+| Environment Setup | 7/20 | 5/20 | 3/20 | 7/20 | 13/20 |
+| Retrieval | 9/20 | 0/20 | 1/20 | 0/20 | 2/20 |
+
+## Style notes
+
+- Column tick labels sit on top, not bottom: with 16 rows the eye enters at the top and
+  the header should be adjacent to the first row it applies to.
+- Benchmark names are abbreviated to keep all five columns inside the text width without
+  rotation; rotated column headers cost more legibility than the abbreviation does.
+
+## Provenance
+
+Identifiers are snake_case in the data (`control_loop`, `browsecomp-plus`) and
+title-cased for display via `analysis.display`; the table above uses the display
+names, which are what the figure shows.
+
+```
+vero interpret extract --runs runs/{officeqa,browsecomp-plus,terminal-bench,swe-atlas-qna,gaia-shell} \
+                       --cells-file scope100.json
+vero interpret edits
+vero interpret label --model gpt-5.4-mini
+python -c "from vero.interpret.analysis import stats; stats.prevalence(rows)"
+```
+3,986 symbol-scoped edits over 100 runs. Roles set by deterministic rule where the
+path, symbol kind or name settles it (2,114 edits) and by model otherwise (1,872).

harness-engineering-bench/figures/figure_02_diversity/figure_02_diversity.md

@@ -0,0 +1,67 @@
+---
+id: figure_02_diversity
+archetype: dumbbell
+script: vero.interpret.analysis.paper_figures::fig_diversity
+outputs: [figure_02_diversity.pdf, figure_02_diversity.png]
+status: review
+---
+
+## Takeaway
+
+Independent optimizers converge on more similar repertoires of edits than chance
+allows, in every benchmark.
+
+## Caption
+
+Mean pairwise Jaccard distance between the sets of edit categories that different
+optimization runs touched (filled marker), against a permutation null that holds each
+run's repertoire size and the corpus-wide category frequencies fixed and reshuffles the
+assignment (open marker, null mean; band, null 95\% interval). All five benchmarks fall
+below their null, so runs are more alike than independent draws from the same marginal
+would be. The raw distance alone carries no information without the null: a value near
+0.5 is equally consistent with genuine diversity and with every run sampling a few
+categories from one skewed distribution. 20 runs per benchmark, 190 pairs each.
+
+## What the reader should see
+
+- The filled marker sits left of the grey band in every row. That gap is the whole
+  finding: left of the null means more similar than chance.
+- Position on the x-axis is the only quantitative channel. The connector shows the size
+  of the departure from the null mean; it encodes nothing extra.
+- Open versus filled distinguishes null from observed. Colour does not carry benchmark
+  identity here, because the y-axis label already does.
+- Absent by design: a significance star or p-value. The null interval is shown directly
+  so the reader judges the margin rather than a threshold.
+
+## Data
+
+| benchmark | runs | observed | null 2.5% | null 97.5% | null mean | verdict |
+|---|---|---|---|---|---|---|
+| BrowseComp-Plus | 20 | 0.561 | 0.630 | 0.667 | 0.651 | converged |
+| OfficeQA | 20 | 0.564 | 0.663 | 0.709 | 0.687 | converged |
+| SWE-Atlas-QnA | 20 | 0.538 | 0.601 | 0.653 | 0.630 | converged |
+| Terminal-Bench | 20 | 0.572 | 0.643 | 0.691 | 0.670 | converged |
+| GAIA-Shell | 20 | 0.341 | 0.399 | 0.446 | 0.425 | converged |
+
+500 permutations per benchmark, seed 0.
+
+## Style notes
+
+- Value labels sit above the marker, not beside it: gaia-shell's observed value is at
+  the far left of the axis and a left-placed label collided with its tick label.
+- Legend is upper-left. Lower-right — the usual choice — overlapped the terminal-bench
+  row, whose null band extends furthest right.
+- gaia-shell is included here, unlike Figure 1, because this statistic is computed
+  within a benchmark and never compared across them, so its constructed seed does not
+  distort it.
+
+## Provenance
+
+Identifiers are snake_case in the data (`control_loop`, `browsecomp-plus`) and
+title-cased for display via `analysis.display`; the table above uses the display
+names, which are what the figure shows.
+
+```
+python -c "from vero.interpret.analysis import stats; stats.jaccard(rows, trials=500, seed=0)"
+```
+Same 3,986-edit label set as Figure 1.

harness-engineering-bench/figures/figure_03_rarefaction/figure_03_rarefaction.md

@@ -0,0 +1,76 @@
+---
+id: figure_03_rarefaction
+archetype: small_multiples
+script: vero.interpret.analysis.paper_figures::fig_rarefaction
+outputs: [figure_03_rarefaction.pdf, figure_03_rarefaction.png]
+status: review
+---
+
+## Takeaway
+
+Five independent optimizers exhaust almost the entire repertoire of edit categories;
+the next fifteen add nearly nothing.
+
+## Caption
+
+Distinct edit categories discovered as optimization runs are added, one panel per
+benchmark (16 categories available). Line, mean over 200 random orderings of the 20
+runs; band, the 10th-90th percentile across those orderings — the spread attributable
+to which runs happen to come first. Every curve is within one category of its final
+value by the fifth run and flat thereafter, and the band collapses over the same
+interval: at one run the number of categories seen spans roughly 2 to 16, by ten runs
+it is a single value. The marginal contribution of an additional independent optimizer
+is therefore close to zero, and that conclusion does not depend on the draw. Same
+convergence \Cref{fig:diversity} establishes against a null, seen as saturation rather
+than as a distance.
+
+## What the reader should see
+
+- The shape, not the ordering: every curve bends hard before x=5 and is flat by x=8.
+- Vertical position at the right edge is the ceiling each benchmark reached (15 or 16 of
+  16). The gap between curves is not the point and should not be over-read.
+- The band narrowing left-to-right is as much the finding as the curve flattening: it
+  says the result is insensitive to which runs you happened to have.
+- Colour distinguishes benchmark only, redundantly with the panel title. Panels share
+  both axes so heights are directly comparable across them.
+- Absent by design: a cross-benchmark overlay. The claim is per-benchmark, and five
+  overlapping bands produced grey that attributed to no series.
+
+## Data
+
+Mean distinct categories after k runs, of 16 available.
+
+Mean, with the 10th-90th percentile across orderings in brackets.
+
+| benchmark | k=1 | k=2 | k=5 | k=10 | k=20 |
+|---|---|---|---|---|---|
+| BrowseComp-Plus | 8.3 [2-16] | 12.0 [8-16] | 15.2 [13-16] | 16.0 [16-16] | 16.0 [16-16] |
+| OfficeQA | 7.7 [3-15] | 10.6 [6-14] | 13.9 [11-15] | 14.9 [15-15] | 15.0 [15-15] |
+| SWE-Atlas-QnA | 8.8 [3-14] | 11.7 [8-15] | 14.3 [12-16] | 15.4 [14-16] | 16.0 [16-16] |
+| Terminal-Bench | 7.5 [2-14] | 10.5 [6-14] | 13.8 [12-15] | 14.7 [14-15] | 15.0 [15-15] |
+| GAIA-Shell | 11.4 [8-15] | 13.9 [11-16] | 15.2 [14-16] | 15.8 [15-16] | 16.0 [16-16] |
+
+200 random orderings per benchmark, seed 0.
+
+## Style notes
+
+- Small multiples rather than one overlaid axis. Adding percentile bands to five
+  overlaid series produced unattributable grey below k=5 and stretched the y-axis to 2,
+  compressing the saturation region that carries the claim. Faceting also retires the
+  earlier problem that made a legend necessary — three benchmarks land on exactly 16.0
+  categories, so endpoint labels could not separate them.
+- Percentiles, not a standard deviation. The quantity is a bounded count skewed hard
+  against its ceiling; a symmetric band would extend past the 16 categories available.
+- x ticks are 1/10/20 only. Panels are ~1.2 in wide and a denser locator produced
+  half-steps, which is not a run count.
+
+## Provenance
+
+Identifiers are snake_case in the data (`control_loop`, `browsecomp-plus`) and
+title-cased for display via `analysis.display`; the table above uses the display
+names, which are what the figure shows.
+
+```
+python -c "from vero.interpret.analysis import stats; stats.rarefaction_bands(rows, trials=200, seed=0)"
+```
+Same 3,986-edit label set as Figure 1.

harness-engineering-bench/figures/figure_04_knob_direction/figure_04_knob_direction.md

@@ -0,0 +1,67 @@
+---
+id: figure_04_knob_direction
+archetype: dumbbell
+script: vero.interpret.analysis.paper_figures::fig_knob_direction
+outputs: [figure_04_knob_direction.pdf, figure_04_knob_direction.png]
+status: review
+---
+
+## Takeaway
+
+Optimizers raise turn and step budgets almost without exception, but move output caps
+and timeouts in both directions.
+
+## Caption
+
+Numeric constants most often changed, by direction of change (filled marker, raised;
+open marker, lowered). Turn and step budgets move overwhelmingly upward — `MAX_TURNS`
+raised in 63 edits against 14 lowered — while output-truncation caps and per-command
+timeouts are as often reduced as increased. Direction is derived by comparing the
+literal before and after values, not inferred from the commit message. Constants
+touched by reformatting without a value change are excluded. Counted per edit rather
+than per run, since one run may retune the same constant several times and each
+retuning is a separate decision.
+
+## What the reader should see
+
+- The top two rows against the rest: budgets go up, everything else is mixed.
+- `MAX_TOOL_OUTPUT_CHARS` is the notable inversion — lowered slightly more often than
+  raised, the only frequently-touched constant where that holds.
+- Horizontal position is a count, and the connector's length is the imbalance between
+  the two directions. Open versus filled is the only other channel.
+- Absent by design: the magnitudes of the changes. A constant moved 24→100 and 24→32
+  count the same here; direction is the claim, size is not.
+
+## Data
+
+| constant | raised | lowered |
+|---|---|---|
+| MAX_TURNS | 63 | 14 |
+| MAX_STEPS | 26 | 5 |
+| MAX_TOOL_OUTPUT_CHARS | 13 | 16 |
+| MAX_OUTPUT_CHARS | 3 | 3 |
+| SOFT_DEADLINE_SEC | 2 | 3 |
+| COMMAND_TIMEOUT_SEC | 1 | 4 |
+| N_ATTEMPTS | 1 | 2 |
+| MAX_HISTORY_CHARS | 0 | 2 |
+| MAX_CONCURRENT_SHELLS | 2 | 0 |
+| RESEARCH_DEADLINE_SEC | 0 | 2 |
+
+Corpus totals across all scalar constants: 137 raised, 78 lowered.
+
+## Style notes
+
+- Top 10 constants only, by total edits. The tail is single-digit and would add rows
+  without adding signal; the cut is stated here so it is not read as the full set.
+
+## Provenance
+
+Identifiers are snake_case in the data (`control_loop`, `browsecomp-plus`) and
+title-cased for display via `analysis.display`; the table above uses the display
+names, which are what the figure shows.
+
+```
+python -c "from vero.interpret.analysis import stats; stats.tuning_direction(rows, edits, top=10)"
+```
+Direction from `taxonomy.direction_of` over captured before/after literals; same
+3,986-edit set as Figure 1.

vero/pyproject.toml

@@ -29,6 +29,9 @@ harbor = [
 claude = [
     "claude-agent-sdk>=0.1.56",
 ]
+interpret = [
+    "openai>=1.0",
+]
 optimize = [
     "async-lru>=2.0.5",
     "beautifulsoup4>=4.14.2",

vero/src/vero/interpret/__init__.py

@@ -0,0 +1,26 @@
+"""Interpretability analysis over optimization runs.
+
+Source artifacts are canonicalised by an adapter (`artifacts`), split into
+symbol-scoped edits deterministically (`edits`), labelled with a model
+(`labeling`), and aggregated (`analysis`). Only `labeling` is non-deterministic.
+"""
+
+from vero.interpret.models import (
+    Candidate,
+    CellRef,
+    Corpus,
+    Edit,
+    EvalRecord,
+    SymbolKind,
+    Trajectory,
+)
+
+__all__ = [
+    "Candidate",
+    "CellRef",
+    "Corpus",
+    "Edit",
+    "EvalRecord",
+    "SymbolKind",
+    "Trajectory",
+]

vero/src/vero/interpret/analysis/__init__.py

@@ -0,0 +1,5 @@
+"""Aggregation and figures over labelled edits."""
+
+from vero.interpret.analysis import figures, stats
+
+__all__ = ["figures", "stats"]