Feat/bench core abstraction

[Abhinavexist]

Every new model used to mean copy-pasting a benchmark runner and patching it per provider — an N×M fork explosion (~40 near-identical files) where copied parts silently drifted and produced wrong scores: dropped RefCOCO runners, a live wrong Gemini thinking budget, and audio_url silently dropped to OpenRouter causing WER ≈ 1.0.

This replaces all of that with a single abstraction: benchmark logic says what to measure, adapters say how to talk to a model, and config says which models to run and when.

Adding a model is now just a YAML entry in src/targets.yaml — no code changes. Everything runs through one CLI:

python -m src run --target X --benchmark Y

The core lives in src/. Capabilities are represented as data, with declared reasoning style, media shape, and response shape per target. At runtime, execute() has a fallback ladder that self-heals declared capability gaps such as thinking-floor and temperature coupling, then emits a hint so the working behavior can be promoted into YAML.

Provider adapters live in src/providers/: openai_compat, anthropic, and gemini, plus reducto for document parsing. There is one client builder and a uniform encode/decode/classify interface.

There is now one runner handling token-bucket rate limiting, a bounded worker pool for memory safety, retry with backoff, resume from per-row append+fsync checkpoints, and ordered cross-provider failover with the actual host recorded for each row.

All results follow one contract:

results/<benchmark>/<target>/{responses,metrics,run}.json

Result discovery is content-based.

All eight benchmarks have been collapsed to one bench.py each under benchmarks/<name>/: GPQA, MMMLU, MMMU-Pro, OCRBench v2, olmOCR, RefCOCO, VoxPopuli ASR, and Spider2-Lite. Each has a parity test proving there is no score drift from the previous implementation.

For heavy benchmarks, --sample N streams only N rows instead of downloading the whole split. Smoke runs are isolated under results/_smoke/, so they cannot overwrite a full score and are hidden from the report.

Full image runs are also RAM-bounded using lazy-by-index loading. RefCOCO val, for example, stays around 1.7 GB flat instead of heading toward ~14 GB and OOMing CI.

CI under .github/workflows/ now includes an offline test + lint gate on every push and PR, plus a benchmark runner supporting manual dispatch, a weekly regression smoke, and a full run when a targets.yaml change is merged. There is also a UI form for adding a model via PR.

There is deliberately no default target, so nothing runs unless a target is explicitly selected.

This removes ~40 per-provider forks, the commons_{gemini,openai,anthropic,fireworks,openrouter} shims, BFCL, JigsawStack, the demo entrypoint, and the one-time results backfill. results/ is no longer tracked because run artifacts are generated output.

185 tests pass, covering parity replays plus capability, adapter, runner, and failover units. ruff check . and ty check src are clean.

All three adapter paths have also been live-verified against real APIs, including audio transport and failover host recording.

The package is currently named src, so the CLI is python -m src. That's unconventional, and it can be moved to a src/bench_core/ layout if preferred.

olmocr and spider2 are excluded from CI because they require poppler/chromium and roughly 4 GB of data. Vendored scorers are excluded from ruff.

reducto lives in src/providers/, but it is a function module rather than a ProviderAdapter, since it represents a document-parsing API rather than a chat provider.