A Mapillary Vistas supervised-transfer baseline, and what it says about the precision gap (#126)

[jonfroehlich]

Advances #126 (does not close it — part 2, non-ramp label types, is untouched).

Stacked on #128, which is stacked on #123. Base is feat/challenger-registry-122; it needs
the registry's label_for because this is the one provider whose label cannot be derived from
its spec. Retarget down the stack as each lands.

The gap in the roster this fills

Every one of the nine challengers is zero-shot — a prompted general VLM or an
open-vocabulary detector. So the roster answers "can a general model be prompted to do this?"
and #51 answers "architecture vs data, within our dataset". Neither answers the third
question: do somebody else's supervised curb-cut labels transfer to deployment panoramas?

Result: richmond, 124 panos, 310 GT ramps

RampNet's row reproduces its committed numbers exactly (0.964 / 0.768 / 0.855, 238/9/72), which
is the check that this run is comparable. The leg also ran twice across a signature change and
agrees to every digit.

model	P	R	F1	AP	tp/fp/fn
rampnet	0.964	0.768	0.855	0.763	238/9/72
gemini-3.1-pro-preview	0.631	0.700	0.664	–	217/127/93
gemini-3.6-flash	0.626	0.642	0.634	–	199/119/111
mask2former-vistas-curb-cut	0.419	0.697	0.524	0.513	216/299/94
molmo2-8B	0.410	0.516	0.457	–	160/230/150
mask2former-vistas-curb-cut+curb	0.127	0.648	0.213	0.089	201/1377/109
owlv2	0.033	0.971	0.064	0.104	301/8799/9
grounding-dino-base	0.028	0.852	0.053	0.032	264/9321/46

Supervised transfer fixes most of the precision problem and does not close the gap. Against
the open detectors — the comparison #126 set up, the concept is findable, the discrimination is
not — real labels buy 12.7× OWLv2's precision (0.419 vs 0.033) for 0.274 of its recall. So
that discrimination failure is about supervision, not about the concept being intrinsically
hard to localize. It is still 4th of nine on F1 and RampNet leads by 0.331, the wide end of
this benchmark's 0.12–0.34 range. These labels transfer; they do not compete.

On AP it is the best non-RampNet model here — 0.513, five times OWLv2's 0.104. The chat VLMs
above it on F1 have no AP at all: unscored boxes, one operating point, untunable. A tunable model
at AP 0.513 is a more useful starting point than an untunable one at F1 0.664, and that is the
part worth carrying forward.

Two mechanisms, measured not assumed

• The curb-cut+curb union is a clean negative result. Run to test whether recall hides on
  the other side of Vistas' ramp/curb boundary. It does not: the union loses recall
  (0.697 → 0.648) while precision collapses to 0.127 on 1,377 FPs. Curb fires along every kerb
  line, and since points come from connected components it also fuses adjacent ramps into
  one — the same adjacent-pair merge Qualitative failure taxonomy: bucket each model's FP/FN from cached detections #46 found from the other direction.
• The Mask the nadir/hood region in the reprojection rig; measure ΔAP #47 hood worry inverted. fp_taxonomy.py puts 1.9% of this arm's FPs in the
  ego-vehicle band vs OWLv2's 15.0% — eight times better, not worse, despite masks being the
  output most likely to bleed into the hood.

Two of the issue's premises were wrong

Both load-bearing, both corrected on the issue:

• box_gallery.py does not reproject. Its --fov sizes an axis-aligned crop of the
  equirectangular image, so distortion is preserved — the wrong input for a perspective-trained
  checkpoint. The real library is equirect_tiling.py, already called by _detect_tiled, so
  this arm needs no new projection code and gets the identical six-view rig as every other
  tiled leg.
• The classes are Curb Cut (9) and Curb (2) in a 65-class v1.2 head, not
  construction--flat--curb-cut from v2.0. So: no training, no dataset download, and no new
  dependency — Mask2Former is in-library and scikit-image is already core.

Instrument checks, before any number

• The einsum semantic map agrees with post_process_semantic_segmentation's argmax on 99.73%
  of pixels. It is assembled here because that helper returns argmax only, and this arm needs a
  per-pixel confidence to have a PR curve at all.
• The MISSING-weight warning on this checkpoint is noise. transformers reports
  swin.layernorm.{weight,bias} as newly initialized; scrambling them to (7.5, −3.25) changes
  0 of 1,048,576 pixels. Recorded so nobody re-derives it.
• Overlay first. All four predictions in the yaw-240 view of pano 1273933840289887 land on
  the yellow detectable-warning pads, no coordinate offset, nothing on the hood. --masks dumps
  the class mask per view, which the points cannot substitute for — a wrong class id gives
  confident points on the wrong object and reads as an ordinary bad model.

Stated as gaps, not left implicit

• Richmond only. One split, one imagery tier (OOD Mapillary). This benchmark's rankings are
  robust but not invariant across splits, so both arms are registered off-roster.
• The paper's driveway prediction is not confirmed. 79.6% of the arm's FPs are isolated,
  which is an upper bound on hallucination, not a measurement — driveways, crosswalks and stairs
  all land there. Confirming it needs Qualitative failure taxonomy: bucket each model's FP/FN from cached detections #46's gallery half pointed at this arm.
• This is a baseline, never a supervision source. The paper (arXiv 2508.09415) already
  rejected these labels as too broad; that is cited, not rediscovered.
• Part 2 of Mapillary Vistas as a second supervision source: off-the-shelf baseline, and a path to non-ramp label types #126 (crosswalk / obstacle label types) is deliberately untouched — it needs the GT
  protocol per label type and should not ride on this.

Verification

pytest -q — 1045 passed, 2 skipped (14 new). Detections published and verified: 2/2 pairs
score identically to the cache. Densities measured, not assumed: 4.48 boxes/pano for curb-cut
(RampNet's class, 4.2); the union's 13.31 sits between the sparse group and the open detectors,
so its density is left unclassified on purpose and density_of refuses it rather than
rounding.

🤖 Generated with Claude Code (claude-opus-5[1m])

[jonfroehlich]

Rebased onto the restacked #128 — no result changes

#128 gained a leg model (pins / published_as) and a test that checks benchmark/model_detections/ against the registry in both directions, so this branch was rebased onto its new tip. One follow-on commit, 200286e, all mechanical:

• vistas joins detectors.PROVIDERS, so the --models help, compare.py's docstring and build_detector's error message all name it from one list instead of three hand-typed ones. This branch predates that consolidation, which is why it was adding vistas to each by hand.
• The ledger counts the two Vistas files: 114 files, 23.1 MB, with a row for the richmond-only class-set arms.
• One real fix. test_every_published_leg_is_named_in_the_ledger knew that slug() maps / to __ (a HF repo id) but not that it maps + the same way (a class set), so it read mask2former-vistas-curb-cut+curb as a published file named in no document.

The Vistas roster entries themselves needed no change — curb-cut measured sparse, curb-cut+curb still None — and they pass the new bijection test as they stand. Suite: 1,112 passed, 2 skipped.

🤖 Generated with Claude Code (claude-opus-5[1m])

[jonfroehlich]

Review: 15 findings — three of them change what this PR claims

/code-review 129 high. Every finding was re-verified by hand before posting, and two are re-stated below because the review's framing needed correcting. Nothing here is fixed yet; this is the record.

The headline: the results table compares two different operating points, and the two most quotable sentences in the write-up are wrong because of it. The underlying arm is fine — arguably better than claimed on recall — but the comparison as printed does not hold.

#	severity	file	what
1	high	docs/model_comparison.md:1215	the table mixes operating points; the "12.7×" headline is a cross-op comparison
2	high	docs/model_comparison.md:1233	"best non-RampNet AP on this split" is false — three YOLO arms beat it, in this same doc
3	high	detectors.py:1720	the checkpoint downsizes every view to 384×384; the doc says it gets 1024×1024
4	medium	detectors.py:1698	_check_class_ids runs after self._model is set, so a caught failure disables it
5	medium	detectors.py:1685	dtype is in the signature but only applied on CUDA — CPU and GPU runs share a cache key
6	medium	rampnet/roster.py:226	bare vistas resolves to two different labels depending on which resolver runs
7	medium	detectors.py:1727	~1.1 GB of transient tensors per view to carry 96×96 of information
8	medium	dump_detections.py:204	the three --vistas-* args default to None instead of PROVIDER_DEFAULTS
9	medium	detectors.py:1609	VISTAS_CLASS_SETS and LABEL_OVERRIDES are two hand-kept tables of one derivable fact
10	low-med	detectors.py:276	min_area_px=16 is calibrated against the wrong resolution and can never fire
11	low-med	detectors.py:1675	four signature keys spell one fact; every extra key is a future orphaning hazard
12	low	docs/model_comparison.md:1230	"fourth of nine challengers" is third
13	low	compare.py:637	the two lists of score-emitting detectors were not updated for vistas
14	low	docs/model_comparison.md:1156	both runbook snippets have their line-continuations collapsed to spaces
15	—	detection files	see below — the review misattributed this one

1. The table mixes operating points

Re-scored from the committed benchmark/model_detections/ against benchmark/richmond:

model	op 0	op 0.30
mask2former-vistas-curb-cut	0.411 / 0.697 / 0.517 — 216/309/94	0.419 / 0.697 / 0.524 — 216/299/94
owlv2	0.033 / 0.971 / 0.064 — 301/8799/9	0.146 / 0.110 / 0.125 — 34/199/276
grounding-dino	0.028 / 0.852 / 0.053 — 264/9321/46	0.030 / 0.090 / 0.045 — 28/897/282

Bold is what the table prints. The Vistas row is the op-0.30 scoring; the OWLv2 and Grounding DINO rows are op-0 numbers carried over from the main roster table. So line 1226 — "real labels buy 12.7× the precision of OWLv2 (0.419 vs 0.033) for 0.274 of its recall" — divides a number measured at 0.30 by one measured at 0.

At a common 0.30 it is 2.9×, and Vistas has more recall than OWLv2 (0.697 vs 0.110), not 0.274 of it. The reproduction command at line 1156 passes --op-threshold 0.30 and therefore does not produce the table printed beneath it.

Worth saying plainly: fixing this makes the arm look better, not worse. At a common operating point Vistas beats OWLv2 on precision and recall. What it costs is the "12.7×" number. The unscored rows (gemini, Qwen, Molmo) are unaffected — they have no confidences, so no operating point applies.

2. "Best non-RampNet AP on this split" is contradicted 1,000 lines up

scripts/model_comparison/yolo_baseline/benchmark_eval/richmond.txt, committed 2026-08-14 and summarized in this same document:

arm	richmond AP	F1 @ conf 0.25
y11x_pano_h200	0.748	0.547
y11l_pano	0.724	0.595
y26_pano	0.536	0.491
mask2former-vistas-curb-cut	0.513	0.524

All three supervised arms beat it on AP, and y11l_pano beats it on F1 too. The write-up never compares against the supervised baseline at all — even though "#51 answers architecture versus data" is the framing for why this arm exists. The most directly comparable supervised model on the same split is the one omitted.

3. The model sees 384×384, not 1024×1024

From the checkpoint's own preprocessor_config.json: size = {'height': 384, 'width': 384}, do_resize = True. Every 1024×1024 view is downsized to ~1/7 the pixel area before the model sees it; mask logits live at 96×96 and are bilinearly upsampled 10.67× back to the view. The claim at line 1172 that this arm gets the "identical six-view rig every other tiled leg uses (… 1024×1024 …)" is true of the rig and false of what the model receives.

That matters beyond the wording: the conclusion "supervised transfer does not compete" is confounded with an uncontrolled resolution handicap, in a benchmark whose far-field/resolution story (#46, #21) is load-bearing. And nothing records it — from_pretrained has no revision pin and no size= override, and neither the effective input size nor the processor config is in signature(), so a transformers upgrade can silently change every mask under an unchanged cache key.

Findings 10 and 7 follow from this one. min_area_px=16 is calibrated against the upsampled 1024×1024 map, but the smallest component a 10.67× upsample can produce is ~114 px — so the "cache floor" is inert at the resolution it actually runs at, while still sitting in the cache signature where changing it orphans both arms' detections. And _raw_detect upsamples all 100 query masks to full view size before the einsum (verified in the code): ~419 MB + 419 MB + 272 MB of transient tensors per view, to carry information that only exists at 96×96. Reducing over classes first gives the same points at roughly 1/50th the memory.

4–14, briefly

• 4. self._model = model.to(self._device).eval() runs on the line before self._check_class_ids(model). If the label set has drifted the check raises — but self._model is already set, so the if self._model is not None: return guard makes every later _ensure_ready() a no-op and the leg proceeds segmenting class 9 of the wrong label set. That is precisely what the check's own docstring says it exists to prevent. Validate before publishing self._model.
• 5. if self._device == "cuda" and self.dtype == "float16" — but dtype is in the signature and the device is not. A CPU run stays fp32 while reporting float16, so a laptop run and a cluster run write different masks under the same cache key. The comment three lines up asserts the opposite.
• 6. LABEL_OVERRIDES keys on vistas:curb-cut, not on bare vistas. Verified: label_for("vistas") → facebook/mask2former-swin-large-mapillary-vistas-semantic, while build_detector("vistas", None, args) → mask2former-vistas-curb-cut, and a test asserts the bare token is supported. fp_taxonomy --models rampnet,vistas therefore looks for a file named after the checkpoint, finds nothing, and reports the arm as unpublished instead of scoring it.
• 8. Line 183 of that same file says defaults come from PROVIDER_DEFAULTS "so this overlay reproduces exactly what compare.py cached rather than a near-miss" — and the three vistas args pass None, falling through to build_detector's literals. The literal 16 appears three times and the checkpoint twice.
• 9. The LABEL_OVERRIDES values are exactly "mask2former-vistas-" + class_set, which is the string VistasDetector.__init__ already computes as its own default. Add a third class set and forget the override, and label_for returns the bare class set — not a model name — which slug() turns into a filename with a + in it. Two sources of truth for one derivable name, with no test tying them.
• 11. prompt, class_set, class_ids, class_names are four spellings of one datum, all derivable from class_set plus the pinned ids. cache_key hashes the whole signature, so dropping any one later to simplify would orphan both arms' paid detections.
• 12. Ranking the nine challengers in the table by F1 puts Vistas third (0.524, behind gemini-3.1-pro 0.664 and gemini-3.6-flash 0.634). "Fourth" counts the table row — but row 1 is rampnet, which the roster's own note calls "the subject, not a challenger". Same error in the PR body.
• 13. Both the module docstring and --sweep's help enumerate "RampNet, OWLv2, Grounding DINO, YOLO" as the detectors that get AP / PR curves / sweeps. Vistas carries confidences and this PR reports AP 0.513 for it. test_every_provider_is_listed_everywhere_a_user_looks passes because it only checks that "vistas" appears somewhere in the docstring — which it does, on the unrelated --models line.
• 14. Both new runbook snippets have five literal spaces where the surrounding snippets use a trailing \ and a newline. They execute, but a reader copying the wrapped form gets a broken command.

15. Where the review was wrong

It reported that the two new detection files are "missing the published_as field that the committed exporter always writes", framed as something this PR introduced. Checked: 110 of the 114 files lack that key, including every file committed before this PR. The two Vistas files are not anomalous — they match the corpus. The underlying point stands (a regenerated copy would differ from the committed one, against CLAUDE.md's content-hash rule), but it is a corpus-wide staleness predating this branch, not a defect in #129, and fixing it here would mean re-exporting 110 files. Filed as such rather than as a finding against this PR.

🤖 Generated with Claude Code (claude-opus-5[1m], effort: high)

[jonfroehlich]

All 15 findings addressed — 104ebb6 + 5d3adf6

Suite 1,114 passed, 1 skipped. Three findings changed what this PR claims; four are deliberately not changed, with the reason recorded in the code rather than left implicit.

#	finding	resolution
1	table mixed operating points	fixed — every row at one point; conf-0.30 numbers in their own table; the reproduction command now produces the table above it
2	"best non-RampNet AP" false	fixed — rewritten as "best zero-training model", with the YOLO comparison stated
3	model sees 384×384, not 1024×1024	fixed as a caveat — stated on the headline; --vistas-input-size / --vistas-revision added, recorded deviation-only; parity run named as not done
4	_check_class_ids after self._model	fixed — validate, then publish
5	dtype applied only on CUDA	fixed — refuses rather than silently downgrading
6	bare vistas resolved two ways	fixed — label_for derives the label from the class set
7	~1.1 GB/view upsample-before-reduce	not changed — reason in the code
8	dump_detections vistas defaults	fixed in 104ebb6
9	two hand-synced label tables	fixed — LABEL_OVERRIDES is empty again; the string is derived
10	min_area_px inert	not changed — documented; recalibrating orphans both arms
11	four redundant signature keys	not changed — documented in signature()'s docstring
12	"fourth of nine"	fixed — third
13	scored-detector lists missing vistas	fixed — both lists, and the test now checks them
14	collapsed line continuations	fixed
15	missing published_as	not this PR's — 110 of 114 files predate it; see the review

The operating-point fix makes the arm look better

Re-derived from the committed detections, which is now what the doc prints:

	op 0	conf ≥ 0.30
mask2former-vistas-curb-cut	0.411 / 0.697 / 0.517	0.419 / 0.697 / 0.524
owlv2	0.033 / 0.971 / 0.064	0.146 / 0.110 / 0.125
grounding-dino	0.028 / 0.852 / 0.053	0.030 / 0.090 / 0.045

At a common floor of none it is 12.4× OWLv2's precision; at 0.30 it is 2.9× the precision with six times the recall. So at the deployment threshold the supervised arm dominates the open detectors on both axes rather than trading one against the other — a stronger claim than the one that shipped. What it costs is the "12.7×" headline, which was 0.30 divided by 0.

What I did not do, and why it is your call

No parity run. Finding 3 means every number in this section was measured at ~1/7 the pixel area every other tiled leg gets, so the gap to RampNet is an upper bound on this arm, not a measurement of it at equal input. The knobs to fix that now exist and are cache-safe; running it costs GPU time, which is a decision rather than a code fix. It is written up as the first thing to try if this arm is revisited.

No re-export. Fixing findings 3, 5, 6 and 9 was done without changing any published detection's cache key — the two new knobs enter signature() only when set, which is the deviation-only pattern from #123. Both richmond files are untouched and still verify.

One correction to my own earlier summary

My note on this arm recorded "AP 0.513 = best non-RampNet" and the "12.7×" figure as results. Both were wrong for the reasons above; I have corrected them at the source rather than leaving the claim to propagate.

🤖 Generated with Claude Code (claude-opus-5[1m], effort: high)

[jonfroehlich]

Rebased onto #128's canonicalization (afe5804). The two Vistas detection files got the same envelope backfill as the other 108 — published_as added, detections byte-identical, verified against HEAD before committing. Note the + survives correctly: mask2former-vistas-curb-cut+curb is recovered from the file's own model field, not from the slugged filename.

Suite 1,117 passed, 1 skipped.

🤖 Generated with Claude Code (claude-opus-5[1m], effort: high)

[jonfroehlich]

Correction to my resolution comment above: finding 14 was reported fixed and was not.

The patch script was a quoted bash heredoc, which strips one level of backslash — so the replacement text's trailing continuation became a Python line continuation and the newline was removed, leaving the same collapsed line it was meant to repair. The --op-threshold correction in the same replacement did apply, which is why the diff looked right.

Actually fixed in #136, together with a test that walks every shell block in the runbook docs and rejects run-on spaces mid-command, so the class cannot recur. It found nothing beyond those two lines.

🤖 Generated with Claude Code (claude-opus-5[1m], effort: high)