Vistas at resolution parity: the handicap was real, and it was recall (#126) by jonfroehlich · Pull Request #137 · ProjectSidewalk/RampNet

jonfroehlich · 2026-08-18T22:03:05Z

Closes the open caveat on the Mapillary Vistas arm from #129: the published richmond numbers were
measured at 384×384 while every other tiled leg sees 1024×1024, so the 0.338 F1 gap to RampNet was
an upper bound, not a measurement. --vistas-input-size landed in #129 for exactly this run.

Docs only. No code, no detection artifacts, no roster change.

Pre-registered read

Posted to #126 before the scored output was read (comment 5334506091): gap closes by
< 0.05 F1 ⇒ "transfers but does not compete" stands, one split, stop. More ⇒ revise and cost
3–4 further splits.

It stands, and not narrowly

RampNet re-ran alongside as the comparability check and reproduced its committed richmond row to
every digit (0.964 / 0.768 / 0.855, AP 0.763, 238/9/72).

arm	model input	P	R	F1	AP	tp/fp/fn
rampnet (committed, reproduced)	—	0.964	0.768	0.855	0.763	238/9/72
vistas curb-cut — published (RTX 3070)	384×384	0.411	0.697	0.517	0.513	216/309/94
vistas curb-cut — same-env control (A40)	384×384	0.411	0.694	0.516	0.510	215/308/95
vistas curb-cut — parity (A40)	1024×1024	0.383	0.884	0.534	0.649	274/442/36

F1 0.516 → 0.534, +0.018 — about a third of the bar. RampNet's lead goes 0.339 → 0.321.

The mechanism is the finding, and it is not what the caveat predicted

The handicap was real and large. It was almost entirely a recall handicap:

Recall 0.694 → 0.884 (+0.190); misses fall 95 → 36, a 62% reduction. At parity this arm
out-recalls RampNet (0.884 vs 0.768) while never having seen one of our labels.
AP 0.510 → 0.649 (+0.139), 27% relative.
Precision 0.411 → 0.383 (−0.028) — slightly worse. FPs rise faster than TPs.

F1 stayed flat only because precision is the binding constraint and resolution does nothing for
it. That sharpens the conclusion #126 was built to test: the concept is findable, the
discrimination is not now holds against the supervised arm at equal input, no longer confounded
with how many pixels the model was given.

An env control that wasn't in the plan, and earned its place

The published run was on an RTX 3070 on an older transformers; makelab2 is a major version on,
and the transformers version is not in the detection signature — a hazard the doc already
flags. So parity-vs-published would have differed in two things.

Re-running 384 in the parity env separates them: it lands within one detection out of 523 of
the published run (215/308/95 vs 216/309/94, F1 0.516 vs 0.517). The 4.x→5.15 jump is benign for
this checkpoint, the residual is fp16 kernel nondeterminism, and the whole parity delta is
attributable to input size. That also retires "an upgrade could have changed every mask under an
unchanged cache key" as a measurement rather than an assurance.

Two claims at 384 that parity corrects

The YOLO AP comparison said richmond AP 0.748 / 0.724 / 0.536 are "all above 0.513". At parity
the arm is 0.649, so y26_pano (0.536) no longer clears it. The sentence's conclusion
holds — the two stronger YOLO arms still lead — its arithmetic does not.
min_area_px=16 is inert at 384 but not at 1024. At 384 the mask upsamples 10.67× so the
smallest blob is ~114 px; at 1024 it upsamples 4× so the smallest blob is ~16 px, exactly the
floor. The two rows share the value, not its meaning. Left untouched so as not to confound the
one variable.

Deliberate omissions

Parity detections are not published. --vistas-input-size does not change the arm's label,
so exporting would overwrite mask2former-vistas-curb-cut__richmond.json. Publishing them needs
a distinct name (the --publish-as pattern from Wire Claude into the model-comparison harness (#122): Opus 5 at low effort takes the top challenger slot on annapolis #123). Out of scope here.
richmond only, per the decision rule.
richmond only for the complementarity read below too.

Verification

Ran on makelab2 (A40, fp16, transformers 5.15.0 / torch 2.13.0+cu130) in a scratch worktree with
a private --cache-dir, so nothing shared a cache directory with the published detections.
Instrument checked before the run, not assumed: the override reaches the model —
pixel_values (1, 3, 384, 384) → (1, 3, 1024, 1024), mask logits (1, 100, 96, 96) →
(1, 100, 256, 256). A silently no-opping processor.size on a new major version would have made
"parity" a second 384 run wearing a different cache key. Also re-confirmed the 65-class head
(class 9 Curb Cut, class 2 Curb).
Cost measured, not extrapolated — the estimate going in was 3–4× and was wrong in the cheap
direction: full 124-pano run 3m38s on one A40; GPU forward 0.078 → 0.092 s per view
(1.17×), because Swin's windowed attention scales far better than pixel count. Peak GPU
memory 1.64 GB.
pytest -q: 1238 passed, 1 skipped, including Actually fix the two collapsed runbook snippets from #129, and guard the class #136's runbook line-continuation test against
the new shell block.

Follow-on in this PR: complementarity (the #35 gate)

The parity result raised the recall-first question, so it is answered here rather than filed.
Scoring-side only, on the same cached detections.

	vistas @384	vistas @1024 (parity)
challenger ONLY (rampnet-miss ∩ hit)	21	54
found by NEITHER	51	18
of rampnet's 72 misses, recovered	21 (29%)	54 (75%)
null on that subset	0.090	0.143
attributable after the null	~15	~44
oracle-union recall	0.835	0.942
boxes/pano · above chance	4.5 · 0.657	6.2 · 0.864

Discounted for chance, a free zero-training model finds ~44 of the 72 ramps RampNet misses —
61%. The null is measured on the miss subset, not extrapolated from the split-wide one, because
RampNet's misses are a biased sample and that is where density differs; it landed at 0.143 against
0.145, so the extrapolation would have been fair — a result, not an assumption.

The 384 control is the interesting half. Parity moved F1 by +0.018 and was correctly judged not
to change the ranking — but it nearly tripled the attributable complementary gain (~15 → ~44)
and shrank the found-by-nobody core from 51 to 18. A flat headline metric hid a large change in
the structure underneath it.

A naive union is still dead: F1 0.555 vs RampNet's 0.855. The 54 ramps arrive with 442 false
positives — ~8.2 FPs per recovered ramp, against the 9 FPs RampNet pays for 238 TPs. The oracle
0.942 is recorded as the ceiling it is, with the real union arithmetic beside it.

The cascade was then run — see below.

Tooling changes

complementarity.py generalized past its Gemini-only form (it called
build_detector("gemini", ...), so this arm could not go through the gate at all). Takes a model
spec now; the documented positional form still works. Verified against the committed paterson
numbers — 36 of 124, 88 found by no model, oracle-union 0.777, reproduced exactly.
--vistas-* args added to null_recall.py. build_detector reads them via getattr, so
without them the null there was silently the 384 arm even when analysing parity.
Provider defaults now come from roster.PROVIDER_DEFAULTS via the same shim shape
fp_taxonomy uses, so the reconstructed cache key cannot drift from the one compare.py wrote
under — a wrong default does not crash, it silently misses every entry and reports zero.
The oracle-union number now prints its counterweight: the FP bill a real union pays and the
resulting P/R/F1. On paterson/gemini-pro the union genuinely wins (0.821 vs 0.805), which is the
check that the line is not hardcoded to one verdict.

pytest -q: 1238 passed, 1 skipped after every step.

Also in this PR: the operating-point correction, and the cascade gate

The complementarity numbers above are against a RampNet we do not recommend

The committed bundle detections are the shipped operating point (≥ 0.5519 on richmond); this
document has recommended 0.30 since #54/#55 (PR #79). Verified op_cache is the same source
before re-basing: filtered at ≥ 0.5519 it reproduces the published row exactly (238/9/72). At 0.30
it gives 257/28/53, P 0.9018 / R 0.8290 / F1 0.8639, matching committed corrected_at_0.3.csv.

	rampnet @0.55	rampnet @0.30 (recommended)
rampnet recall / F1	0.768 / 0.855	0.829 / 0.864
rampnet misses	72	53
challenger recovers	54 (75%)	38 (72%)
attributable after null	~44	~30
naive union F1	0.555	0.549

~14 of the recoveries are ramps RampNet already has at the recommended point. Deployable gain
is ~30, not ~44 — the correction is applied in the doc, next to the original figure rather
than replacing it. The recovery rate barely moves (75% → 72%): fewer misses to find, not a
different kind of miss.

`cascade_gate.py` (new): the cascade is live, ceiling ~19 ramps

Partitions all 310 GT ramps into the four cells and reads RampNet's heatmap at each, reusing #46
Phase 1's instrument verbatim. Read pre-registered on #126 before running. At rampnet@0.30, of the
38 genuinely-complementary ramps:

what RampNet has there	n	meaning
floor peak in radius, 0.05–0.30	19	promotable — the cascade's real target
floor peak ≥0.30 but unmatched	4	greedy matcher gave it to an adjacent GT — matcher/σ (#130), not threshold
no floor peak in radius	15	nothing to promote; `act` 0.215 is unpeaked mass

Ceiling ≈ 19 ramps, +6.1 recall points (0.829 → 0.890), FP cost unmeasured. Far below the 54
the raw complementarity suggested.

Negative worth keeping: RampNet's activation does not discriminate — challenger_only null
percentile 0.88 vs the hard core's 0.925, so the ramps nobody finds look stronger on raw heatmap
mass. No self-gating shortcut exists.

An instrument bug I caught mid-run

nearest_peak expects floor peaks (≥0.05); I first fed it the shipped ≥0.5519 detections, so
"is there a peak near this missed ramp?" was really "is there a 0.55+ detection near it?" — close
to definitionally no for a miss. It made every miss look peakless (nearest peak a median 104 px
away) and I nearly wrote up "the cascade is dead". Fixed in dcb9c32, artifacts regenerated. The
greedy match that defines the miss set still uses the bundle records, since that is what
produced the published 238/9/72 — two inputs, two questions, now documented.

New committed artifacts: analysis_out/cascade_gate.json, analysis_out/cascade_gate_op030.json
(both whitelisted in .gitignore, written with newline="" so a Windows re-run stays
byte-comparable).

🤖 Generated with Claude Code (claude-opus-5[1m])

…#126) The published Vistas richmond numbers were measured with the checkpoint's own 384x384 preprocessor in force, so every 1024x1024 view was downsized to ~1/7 the pixel area while every other tiled leg sees the full view. That made the 0.338 F1 gap to RampNet an upper bound rather than a measurement. #129 added --vistas-input-size for exactly this run; this is it. Read pre-registered on #126 before the scored output was read: gap closes by < 0.05 F1 => "transfers but does not compete" stands, one split, stop. Result: it stands, and not narrowly. F1 0.516 -> 0.534 (+0.018), about a third of the bar. RampNet's lead goes 0.339 -> 0.321. The mechanism is the actual finding, and it is not what the caveat predicted. The handicap was large but almost entirely a RECALL handicap: recall 0.694 -> 0.884 (+0.190; misses 95 -> 36, a 62% reduction) AP 0.510 -> 0.649 (+0.139, 27% relative) precision 0.411 -> 0.383 (-0.028, slightly worse) At parity this arm out-recalls RampNet (0.884 vs 0.768) while never having seen one of our labels. F1 stayed flat only because precision is the binding constraint and resolution does nothing for it -- which sharpens the conclusion #126 was built to test: "the concept is findable, the discrimination is not" now holds against the supervised arm at equal input, no longer confounded with pixels. Also ran a same-env 384 control, which was not in the plan and earned its place. The published run was on an RTX 3070 on an older transformers; makelab2 is a major version on, and the transformers version is not in the detection signature -- a hazard this document already flags. So parity-vs-published would have differed in two things. The control lands within ONE detection out of 523 of the published run (215/308/95 vs 216/309/94), so the 4.x->5.15 jump is benign for this checkpoint and the whole delta is attributable to input size. That retires the "an upgrade could have changed every mask under an unchanged cache key" worry as a measurement. Two claims stated at 384 are corrected: y26_pano's richmond AP 0.536 no longer clears this arm at 0.649 (the YOLO sentence's conclusion holds, its arithmetic does not), and min_area_px=16 is inert at 384 but sits exactly at the smallest achievable blob at 1024, so the two rows do not share that setting's meaning. Parity detections are deliberately not published: --vistas-input-size does not change the arm's label, so exporting would overwrite the published richmond file. Publishing them needs a distinct name (the --publish-as pattern from #123). Measured cost, since the estimate going in was 3-4x and wrong in the cheap direction: the full 124-pano run is 3m38s on one A40, GPU forward 0.078 -> 0.092 s per view (1.17x). Verified before the run that the override reaches the model (pixel_values 384 -> 1024, mask logits 96x96 -> 256x256) rather than silently no-opping on a new major version. Docs only; no code, no detection artifacts, no roster change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The #35 gate script was hardcoded to build_detector("gemini", ...), so the parity Vistas arm could not be run through it at all. Generalized to a model spec (provider or provider:model_id), with the documented positional form preserved -- a bare non-provider token still reads as a Gemini model id, which is how three call sites in docs/model_comparison.md spell it. Verified against the committed paterson numbers: 36 of 124 misses recovered (29%), 88 found by no model, oracle-union 0.777, all reproduced exactly. Provider defaults now come from roster.PROVIDER_DEFAULTS via the same shim shape fp_taxonomy and null_recall use, so the reconstructed cache key cannot drift from the one compare.py wrote under -- a wrong default there does not crash, it silently misses every entry and reports zero detections. Also added the --vistas-* args to null_recall.py. build_detector reads them off the namespace with getattr, so without them the vistas arm there was silently the published 384x384 one even when the run being analysed was at parity -- a wrong-arm read that looks like a valid answer. And printed the counterweight to the oracle-union number: the FP bill a naive union actually pays, with the resulting precision/F1 against rampnet alone. The oracle ceiling assumes a combiner that keeps every right call, which does not exist; on paterson/gemini-pro the union genuinely wins (0.821 vs 0.805), which is the check that the line is not hardcoded to one verdict. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

#126) The headline of a complementarity run is "the challenger recovers N of rampnet%s M misses". At any real box density some of that N is the match radius handing out hits for free, and the repo doctrine is to discount every recall-ceiling and union-oracle claim before believing it. null_recall.py already measures that, but over the WHOLE split. Applying its number to the miss subset assumes the coincidence rate is uniform across GT, and it need not be: rampnet%s misses are a biased sample (far-field, adjacent pairs) and those are exactly where box density differs. So this measures the null on the subset the claim is about, using null_recall%s own cyclic-shift construction -- pano A%s missed GT against pano B%s predictions, averaged over every non-identity shift, so box count and clustering stay real and only the pairing is wrong. On the paterson/gemini-pro regression case the committed numbers are unchanged (36 of 124, 88 found by no model) and the new line reads: null 0.081, so ~10 of the 36 are coincidence and ~26 attributable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…till loses (#126) Runs the #35 gate on the parity arm, plus the 384 arm as the control. Answers the recall-first question the parity result raised, and answers it against chance rather than at face value. Discounted for the null, a free zero-training model finds ~44 of the 72 ramps RampNet misses on richmond (61%). Raw is 54 of 72 (75%); the null on that subset is 0.143. The recall is real detection, not density -- at 6.2 boxes/pano the arm above-chances 0.864, higher than RampNet own 0.754, nothing like OWLv2 0.733 at 74 boxes/pano. The control is the interesting half. Parity moved F1 by +0.018 and was correctly judged not to change the ranking -- but it nearly TRIPLED the attributable complementary gain (~15 -> ~44 ramps) and shrank the found-by-nobody core from 51 to 18. A flat headline metric hid a large change in the structure underneath it, and only the complementarity read surfaced it. A naive union stays dead and it is not close: F1 0.555 against RampNet 0.855. The 54 ramps arrive with 442 false positives, ~8.2 FP per recovered ramp against the 9 FP RampNet pays for 238 TP. Recorded the oracle-union 0.942 as the ceiling it is, with the real union arithmetic beside it so the two cannot be confused. What is left open is a gated cascade -- this arm candidates as a spatial prior to locally relax RampNet threshold -- which is decidable but NOT run: #131 measured most silent misses as adjacent-tail or faint, but nobody has checked whether sub-threshold signal exists at these 54 locations specifically. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The complementarity result rules out a union but leaves a gated cascade -- the challenger candidates as a spatial prior to locally relax RampNet threshold. That has one precondition and it is measurable without building anything: RampNet must already produce sub-threshold response AT those ramps. If the heatmap is flat there the miss is genuine absence and no prior can raise it. Partitions every GT ramp into the four complementarity cells and reads the heatmap at each, with #46 Phase 1 instrument imported rather than reimplemented -- site_profile, null_percentile, nearest_peak, class_of and its ABSENT_MAX/PEAK_FLOOR cutoffs -- so the numbers are directly comparable to that phase 8% absent / 62% adjacent-tail / 30% faint decomposition, and a fix to the probe fixes both analyses. The three-way class is the point, not the raw activation. faint_local with the argmax at the site is signal a prior could raise. tail is an ADJACENT mode reaching in, so relaxing a threshold there fires on the neighbour rather than the ramp -- that argues for sigma/NMS work, not for a prior, even though it reads as "signal present" if you only look at act. Read pre-registered on #126 before running. Nulls computed only for the two RampNet-miss cells; the hit cells are high by construction and are the positive control. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ons (#126) Self-inflicted, caught before the write-up. nearest_peak expects peak_local_max output down to the 0.05 floor. I fed it records[pid]["detections"], which on richmond are all >= 0.5519 -- the SHIPPED operating point. So "is there a peak near this missed ramp?" was really asking "is there a 0.55+ detection near it?", which for a missed ramp is close to definitionally no, and it made every miss look peakless with the nearest peak a median 104 px away. analysis_out/op_cache/richmond.json is committed and holds the floor peaks: 537 for richmond, 270 of them below 0.55. That is the input the sub-threshold question needs. The greedy match that DEFINED the miss set still uses the bundle records, because that is what produced the published 238/9/72 -- two different inputs to two different questions, now documented as such. Also replaced the class_of column in the printed table with peak_in_radius. class_of is #46 Phase 1 cutoffs, and there "tail" could only mean an outside mode reaching in because that population was SILENT misses by definition. This population is every RampNet miss, so act >= 0.05 has two causes the class cannot separate: a floor peak inside the radius (the model localized it and the detection was lost downstream -- recoverable WITHOUT a second model) versus unpeaked heatmap mass (nothing for a prior to promote). The old table also printed "100% tail" for the two hit cells, which is meaningless there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The committed bundle detections are the SHIPPED operating point -- on richmond every one scores >= 0.5519 -- while this document own recommendation since #54/#55 (PR #79) is 0.30. For a complementarity read those are different models, and the difference decides who gets credit for a recovery. Verified the op_cache is the same source before trusting it: filtered to >= 0.5519 it reproduces the published richmond row exactly (P 0.9636 R 0.7677 F1 0.8546, 238/9/72). At 0.30 the same peaks give P 0.9018 R 0.8290 F1 0.8639, 257/28/53 -- matching the committed corrected_at_0.3.csv. So RampNet at the recommended point already finds 19 of the ramps the shipped point misses. --rampnet-op-threshold re-sources RampNet from analysis_out/op_cache/<split>.json at a given threshold. Default is the bundle records, so the published roster numbers and the committed #35 gate results are unchanged -- paterson/gemini-pro still reproduces 36 of 124, 88 nobody finds, oracle-union 0.777. Also regenerates analysis_out/cascade_gate.json, which was first committed from a run whose nearest_peak input was the shipped detections rather than the floor peaks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… too (#126) same reasoning as the complementarity flag: the bundle detections are the shipped point and the recommendation is 0.30, and the cells move between them. peak_in_radius and the heatmap probe are threshold-independent, but which cell a ramp lands in is not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…e gain was ours (#126) Two findings, one of which corrects my own headline from earlier in this PR. 1. THE OPERATING-POINT CORRECTION. The complementarity read scored RampNet from the committed bundle detections, which are the SHIPPED point (>=0.5519 on richmond), while this document has recommended 0.30 since #54/#55 (PR #79). Verified op_cache is the same source before re-basing: filtered at 0.5519 it reproduces the published row exactly (238/9/72). At 0.30 it gives 257/28/53, P 0.9018 R 0.8290 F1 0.8639, matching committed corrected_at_0.3.csv. Re-based, the challenger recovers 38 of 53 misses, ~30 after the null -- not 54 of 72 / ~44. About 14 of the ramps it got credit for are ramps RampNet already has at the point we recommend. The recovery RATE barely moves (75% -> 72%), which is the honest read: there are simply fewer misses to find. 2. THE CASCADE CEILING. Of the 38 genuinely-complementary ramps at 0.30: 19 floor peak in radius at 0.05-0.30 -> promotable, the real target 4 floor peak >=0.30 but unmatched -> greedy matcher gave it to an adjacent GT; a matcher/sigma problem (#130), not a threshold one 15 no floor peak at all -> nothing to promote; act 0.215 is unpeaked mass So the cascade is LIVE with a ceiling of ~19 ramps, +6.1 recall points (0.829 -> 0.890), before an unmeasured FP cost. A long way below the 54 the raw complementarity suggested. Also a negative worth keeping: RampNet activation does NOT discriminate. challenger_only sits at null percentile 0.88 and the hard core at 0.925 -- the ramps nobody finds look stronger on raw heatmap mass. There is no self-gating shortcut that skips the second model. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

This work branched at 5e20d11, before eccadda (wrap the 360 seam in the matcher) and f4c71c8 (peaks_to_dets dropped peaks beside the seam) landed on main. Rather than argue that away, measured it. The greedy match used here does not wrap. score_pano own docstring records that wrapping moves no metric on any committed split while #130 duplicate ground truth masks it, so aggregates are safe -- but a cell assignment is finer-grained than an aggregate. And analysis_out/op_cache/richmond.json is UNCHANGED by those commits, so it was never regenerated after the peaks_to_dets fix and can still be missing peaks beside the seam; that can only UNDERSTATE the promotable count. Measured: 6 of richmond 310 GT ramps straddle the seam, and only ONE is in challenger_only. The other five are in both, where neither fix can move the partition in a direction that matters. Worst case for the headline is one ramp in 38 and no conclusion turns on it. Re-running on a post-#140 main retires the caveat instead of bounding it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

jonfroehlich mentioned this pull request Aug 18, 2026

Mapillary Vistas as a second supervision source: off-the-shelf baseline, and a path to non-ramp label types #126

Open

jonfroehlich and others added 9 commits August 18, 2026 15:48

Navigation Menu

Uh oh!

Vistas at resolution parity: the handicap was real, and it was recall (#126) - #137

Vistas at resolution parity: the handicap was real, and it was recall (#126)#137
jonfroehlich wants to merge 10 commits into
mainfrom
exp/vistas-parity-126

jonfroehlich commented Aug 18, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

jonfroehlich commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-registered read

It stands, and not narrowly

The mechanism is the finding, and it is not what the caveat predicted

An env control that wasn't in the plan, and earned its place

Two claims at 384 that parity corrects

Deliberate omissions

Verification

Follow-on in this PR: complementarity (the #35 gate)

Tooling changes

Also in this PR: the operating-point correction, and the cascade gate

The complementarity numbers above are against a RampNet we do not recommend

cascade_gate.py (new): the cascade is live, ceiling ~19 ramps

An instrument bug I caught mid-run

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jonfroehlich commented Aug 18, 2026 •

edited

Loading

`cascade_gate.py` (new): the cascade is live, ceiling ~19 ramps