Skip to content

fix: validate OJBench across the full corpus - #5

Open
aabbdev wants to merge 3 commits into
mainfrom
research/ojbench-dmoj-20260806
Open

fix: validate OJBench across the full corpus#5
aabbdev wants to merge 3 commits into
mainfrom
research/ojbench-dmoj-20260806

Conversation

@aabbdev

@aabbdev aabbdev commented Aug 7, 2026

Copy link
Copy Markdown
Owner

Summary

  • pin OJBench's image to Bookworm/GCC 12 so every official Testlib checker, interactor, and output validator compiles unchanged
  • add a sanitized arm64 full-configuration preflight and document zero infrastructure failures across all 232 Python and 232 C++ paths
  • record a complete pinned 3,712-generation RWKV-7 run with real Pass@1/Pass@8 results and retain alpha pending a second or Inspect-native model stack
  • refresh the long-term TIR, VitaBench, and Codeforces-Elo audits; qualify current bilingual VitaBench v1 live as the next executable candidate

Validation

  • pre-commit run --all-files
  • pytest: 476 passed, 11 skipped
  • targeted OJBench mypy: passed
  • Bookworm arm64 image: CPP17/PYPY3 self-tests passed
  • full sentinel preflight: 464/464 normal verdicts, zero IE/Skip/exceptions
  • real model run: 3,712/3,712 unique judgments, zero infrastructure errors
    • Python: Pass@1 0.4310%, Pass@8 1.7241%
    • C++: Pass@1 0.2155%, Pass@8 0.8621%
  • VitaBench source smoke: all eight 100-task language/domain manifests loaded; 66-tool cross-domain environments constructed

Notes

Generated programs and hidden tests are not included. The committed evidence contains only aggregate counts, hashes, verdict status, timing, and protocol metadata. OJBench remains alpha because the full generation path used one direct pinned Transformers stack rather than an Inspect-native provider.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant