Add classical baselines (tesseract, pp-ocrv6) + 3 new VLMs to the model registry#34
Merged
Merged
Conversation
…R-2, Unlimited-OCR, olmOCR-2 Two classical (non-VLM) baselines and three current VLMs: - tesseract (cpu-upgrade) — the legacy baseline; first CPU-flavor entry in the registry - pp-ocrv6 (t4-small, --model-tier medium) — modern classical det+rec pipeline; cheapest model in the benchmark - deepseek-ocr-2 (l4x1) — successor to deepseek-ocr already on the boards - unlimited-ocr (l4x1, baidu's dedicated vllm-openai image tag) — currently the top-trending OCR model on the Hub - olmocr-2 (l4x1, FP8) — top scorer on olmOCR-bench old_scans in our experiments Scripts all live on uv-scripts/ocr; tesseract's lands there via uv-scripts-for-ai#74. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…results 5-sample smoke runs against a scratch repo validated each entry: - deepseek-ocr-2: needs image-mode (script's vLLM nightly cu129 wheels want libnvrtc.so.13, absent from the default uv image) and --prompt-mode free (grounded default interleaves <|det|> markup the judge would penalize). Both verified clean on re-run. - unlimited-ocr: pass --strip-grounding for the same markup reason (script already supported it). Verified clean. - olmocr-2: DROPPED — its script predates the --config/--create-pr contract, so ocr-bench cannot launch it; documented locally, re-add after a script update. Registry 14 -> 13; tesseract, pp-ocrv6, paddleocr-vl-1.6 passed smoke unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
Author
|
Smoke-validated every entry with 5-sample Jobs against a scratch repo before asking for merge — all five now proven end-to-end (schema checked the way
Registry lands at 13 entries. Merging this unblocks the real 50-sample Britannica runs. |
Smoke-verified on bare-image l4x1 (4m28s, correct schema, real text). Requires #75 merged + synced before launching. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
Author
|
olmocr-2 is back in (registry lands at 14): its script contract gap is fixed in uv-scripts-for-ai#75 and smoke-verified on a real bare-image l4x1 job (4m28s, schema + real text confirmed — no image-mode needed). Merge order: uv-scripts#75 first (sync workflow puts the fixed script on the Hub), then this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Expands MODEL_REGISTRY from 9 to 14 for the Britannica benchmark round (issue #1):
Classical baselines — the "does any of this beat what libraries already run?" anchor:
tesseract(cpu-upgrade) — Tesseract 5 via the new script in uv-scripts-for-ai#74; merge that first (registry loads fromuv-scripts/ocrmain, populated by the sync workflow)pp-ocrv6(t4-small, medium tier, 34.5M params) — modern classical det+rec pipeline, cheapest entry in the benchmarkNew VLMs (scripts already on
uv-scripts/ocr):deepseek-ocr-2(3.4B, l4x1) — v1→v2 comparison with deepseek-ocr already on the boardsunlimited-ocr(3.3B, l4x1, Baidu'svllm/vllm-openai:unlimited-ocrimage per the script's docstring) — top-trending OCR model on the Hub right nowolmocr-2(8.3B FP8, l4x1) — top scorer on olmOCR-bench old_scans inexperiments/Registry count test updated 9 → 14.
pytest283 passed,ruff+tyclean.🤖 Generated with Claude Code