diff --git a/benchmark/wave2-code/matrix.tsv b/benchmark/wave2-code/matrix.tsv index 6ffb446f..40214ee6 100644 --- a/benchmark/wave2-code/matrix.tsv +++ b/benchmark/wave2-code/matrix.tsv @@ -1,6 +1,7 @@ candidate case rep metric resource status notes ojbench registry_baseline 1 0 0 unsupported No implementation or registry ID at baseline 7f11867 -ojbench official_protocol_audit 464_prompts_232_problems 0 dmoj+g++17+pypy3+git_lfs_testdata unsupported Official OJBench repo commit 5e94480b1e135b98855cf5bc81213c256aff5b17 and HF testdata HEAD 61cf9986f22c25d08e1657b03742124099c74353 expose 464 prompts with sha256 bcc8c94eb1fefb856355aa8b5a3e20cc0a2112f5436c5d83ab686edb417bce2c, but faithful judging requires DMOJ 4.1.0 at judge-server commit f098cd3a49a60186d1fadde5132329ec5f4f2213 plus g++17/PyPy3 and LFS problem zips; no hardened OpenBench Docker image has been validated for DMOJ under cap-drop/no-network, and OpenCompass only loads prompts without scoring. +ojbench official_protocol_audit 464_prompts_232_problems 0 dmoj+g++17+pypy3+git_lfs_testdata unsupported Official OJBench repo commit 5e94480b1e135b98855cf5bc81213c256aff5b17 and HF testdata HEAD 61cf9986f22c25d08e1657b03742124099c74353 expose 464 prompts with sha256 bcc8c94eb1fefb856355aa8b5a3e20cc0a2112f5436c5d83ab686edb417bce2c. The initial audit established that faithful judging required DMOJ 4.1.0 at judge-server commit f098cd3a49a60186d1fadde5132329ec5f4f2213 plus g++17/PyPy3 and LFS problem zips, while OpenCompass only loaded prompts without scoring; the subsequent alpha adapter row records the completed hardened-runtime work. +ojbench openbench_alpha_adapter 232_python+232_cpp_x8 0 lazy_pinned_testdata+dmoj_arm64 alpha_arm64_runtime_validated Added ojbench_python and ojbench_cpp with eight official samples per problem, pinned prompts/testdata, and lazy per-problem downloads. The arm64 DMOJ image passed CPP17/PYPY3 self-tests and judged both languages over 20 real loj-2083 cases; an attempted hidden-test file read received IR. The container is network-none/read-only/no-new-privileges with bounded memory/PIDs, cap-drop ALL, and only DMOJ-required SYS_PTRACE restored. Full 7.85GB corpus validation and real model scoring remain future gates, so both IDs are alpha. tir_bench registry_baseline 1 0 0 unsupported No implementation or registry ID at baseline 7f11867 tir_bench official_protocol_audit 1215_examples_13_tasks 0 agentic_image_harness+gpt4.1_extractor unsupported Official repository commit f79c7562b59e4f8142b0437fc725eb3ee1aec76c and HF dataset revision ae9976e81e86c4797fd0d28f9119393b37d9d006 publish data and post-generation scoring, but the released scripts require a GPT-4.1 answer extractor and do not publish the agentic image/tool generation harness underlying with-CI/without-CI runs. codeforces_elo registry_baseline 1 0 0 unsupported No implementation or registry ID at baseline 7f11867 diff --git a/benchmark/wave2-code/ojbench.md b/benchmark/wave2-code/ojbench.md index 1d65f8e6..c811c2cd 100644 --- a/benchmark/wave2-code/ojbench.md +++ b/benchmark/wave2-code/ojbench.md @@ -2,10 +2,12 @@ ## Decision -OJBench is recorded as unsupported for this OpenBench wave. Unlike Spider and -MEGA, OJBench does publish a full prompt file for LLM generation, but faithful -scoring depends on a DMOJ-based online-judge runtime that has not been validated -inside OpenBench's hardened Docker policy. +OJBench now has alpha `ojbench_python` and `ojbench_cpp` registry IDs backed by +the official DMOJ judge. Each language track contains 232 problems and uses +eight samples per problem, matching the paper's Pass@1/Pass@8 protocol. The +paper uses each model's recommended sampling parameters rather than one global +temperature/top-p configuration, so OpenBench deliberately leaves those model +settings configurable. ## Canonical sources @@ -53,12 +55,35 @@ JSONL file. It does not implement DMOJ setup, code extraction, test execution, partial verdicts, or scoring, so it is not sufficient evidence for a faithful OpenBench integration. -## OpenBench compatibility finding +## OpenBench implementation -OJBench should be integrated only once there is a validated Docker execution -boundary for DMOJ that preserves OpenBench's safety policy: network disabled, -capabilities dropped, no new privileges, bounded process/memory limits, and no -host compiler/runtime escape. That image also needs the large LFS problem zips -or a reproducible cache step. Until that exists, adding a registry ID would risk -either weakening the sandbox or reporting scores from an unvalidated judge. The -candidate is therefore blocked rather than approximated. +- Prompts are checksum-verified and loaded from the pinned Hugging Face + revision. +- Test archives and custom validators are downloaded lazily per problem into a + host cache mounted read-only at `/problems`. A limited run therefore does not + require downloading the complete 7.85 GB source repository. +- The image installs OJBench commit + `5e94480b1e135b98855cf5bc81213c256aff5b17` and DMOJ commit + `f098cd3a49a60186d1fadde5132329ec5f4f2213`, with exact Python dependencies, + C++17 `g++`, and PyPy3. +- The scorer returns only the final verdict, number of executed cases, and + partial pass booleans. Per-case inputs, outputs, and feedback never enter + Inspect logs. + +The container runs without networking, with a read-only root filesystem, +`no-new-privileges`, bounded memory/PIDs, and `cap_drop: ALL`. DMOJ cptbox +requires `SYS_PTRACE` to supervise its own child process, so that single +capability is restored explicitly. Generated programs remain under DMOJ's +seccomp and filesystem policies; `/problems` is not in their readable policy. + +## Validation and remaining gate + +On local arm64 Docker, both CPP17 and PYPY3 executor self-tests passed. Wrong +Python and C++ submissions were judged against 20 real `loj-2083` cases and +returned `WA`. A submission attempting to read the mounted hidden-test +`init.yml` returned DMOJ `IR`, confirming the anti-oracle filesystem boundary. + +The complete 7.85 GB problem corpus has not yet been executed end to end, and +no credentialed model run has been scored. The two registry IDs therefore +remain alpha until broader custom-validator coverage and real Pass@1/Pass@8 +runs are complete. diff --git a/docs/snippets/benchmarks.data.mdx b/docs/snippets/benchmarks.data.mdx index e9f098e0..23b430a8 100644 --- a/docs/snippets/benchmarks.data.mdx +++ b/docs/snippets/benchmarks.data.mdx @@ -6767,6 +6767,34 @@ export const benchmarksData = [ "function_name": "ocrbenchv2", "is_alpha": false }, + { + "name": "OJBench C++", + "description": "Competition-level C++ generation with the official DMOJ judge", + "category": "core", + "tags": [ + "coding", + "cpp", + "execution", + "docker", + "online-judge" + ], + "function_name": "ojbench_cpp", + "is_alpha": true + }, + { + "name": "OJBench Python", + "description": "Competition-level Python generation with the official DMOJ judge", + "category": "core", + "tags": [ + "coding", + "python", + "execution", + "docker", + "online-judge" + ], + "function_name": "ojbench_python", + "is_alpha": true + }, { "name": "OpenAI MRCR (2 Needles)", "description": "Memory-Recall with Contextual Retrieval - long-context evaluation that measures recall of 2 needles across million-token contexts", diff --git a/packages/openbench-core/pyproject.toml b/packages/openbench-core/pyproject.toml index 333fc07d..47e09fc4 100644 --- a/packages/openbench-core/pyproject.toml +++ b/packages/openbench-core/pyproject.toml @@ -57,6 +57,7 @@ include = ["openbench*"] "requirements-arm64.lock", ] "openbench.evals.evalplus" = ["Dockerfile", "compose.yaml"] +"openbench.evals.ojbench" = ["Dockerfile", "compose.yaml"] "openbench.evals.bfcl" = [ "Dockerfile", "Dockerfile.agentic", diff --git a/pyproject.toml b/pyproject.toml index a3171a31..f6ec85ea 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -62,6 +62,7 @@ include = ["openbench*"] "requirements-arm64.lock", ] "openbench.evals.evalplus" = ["Dockerfile", "compose.yaml"] +"openbench.evals.ojbench" = ["Dockerfile", "compose.yaml"] "openbench.evals.bfcl" = [ "Dockerfile", "Dockerfile.agentic", diff --git a/src/openbench/config.py b/src/openbench/config.py index 26f30704..8c461d61 100644 --- a/src/openbench/config.py +++ b/src/openbench/config.py @@ -457,6 +457,24 @@ class EvalGroup: module_path="openbench.evals.livebench", function_name="livebench_coding_2024_11_25", ), + "ojbench_python": BenchmarkMetadata( + name="OJBench Python", + description="Competition-level Python generation with the official DMOJ judge", + category="core", + tags=["coding", "python", "execution", "docker", "online-judge"], + module_path="openbench.evals.ojbench", + function_name="ojbench_python", + is_alpha=True, + ), + "ojbench_cpp": BenchmarkMetadata( + name="OJBench C++", + description="Competition-level C++ generation with the official DMOJ judge", + category="core", + tags=["coding", "cpp", "execution", "docker", "online-judge"], + module_path="openbench.evals.ojbench", + function_name="ojbench_cpp", + is_alpha=True, + ), # Exercism benchmarks "exercism": BenchmarkMetadata( name="Exercism", diff --git a/src/openbench/datasets/ojbench.py b/src/openbench/datasets/ojbench.py new file mode 100644 index 00000000..ffb25a37 --- /dev/null +++ b/src/openbench/datasets/ojbench.py @@ -0,0 +1,134 @@ +"""Pinned loader for the official OJBench prompt release.""" + +from __future__ import annotations + +import hashlib +import json +import os +import re +from pathlib import Path +from typing import Literal + +from huggingface_hub import hf_hub_download, snapshot_download +from inspect_ai.dataset import MemoryDataset, Sample +from platformdirs import user_cache_dir + +DATASET_REPOSITORY = "He-Ren/OJBench_testdata" +DATASET_REVISION = "61cf9986f22c25d08e1657b03742124099c74353" +PROMPT_FILE = "prompts/full.jsonl" +PROMPT_SHA256 = "bcc8c94eb1fefb856355aa8b5a3e20cc0a2112f5436c5d83ab686edb417bce2c" +PROMPT_COUNT = 464 + + +def problem_cache_dir() -> Path: + """Return the host directory mounted read-only into the judge.""" + + configured = os.environ.get("OPENBENCH_OJBENCH_DATA_DIR") + path = ( + Path(configured).expanduser() + if configured + else Path(user_cache_dir("openbench")) / "ojbench" / "problems" + ) + path.mkdir(parents=True, exist_ok=True) + (path / "NOI").mkdir(exist_ok=True) + (path / "ICPC").mkdir(exist_ok=True) + os.environ["OPENBENCH_OJBENCH_DATA_DIR"] = str(path.resolve()) + return path.resolve() + + +def _prompt_path() -> Path: + path = Path( + hf_hub_download( + repo_id=DATASET_REPOSITORY, + filename=PROMPT_FILE, + repo_type="dataset", + revision=DATASET_REVISION, + ) + ) + digest = hashlib.sha256(path.read_bytes()).hexdigest() + if digest != PROMPT_SHA256: + raise ValueError( + f"OJBench prompt checksum mismatch: expected {PROMPT_SHA256}, got {digest}" + ) + return path + + +def canonical_problem_id(dataset: str, record_id: int | str) -> tuple[str, str]: + """Map prompt identifiers to the canonical problem directory.""" + + if dataset == "NOI" and isinstance(record_id, int): + return "NOI", f"loj-{record_id}" + if dataset == "icpc" and isinstance(record_id, str): + if re.fullmatch(r"[A-Za-z0-9_-]+", record_id): + return "ICPC", record_id + raise ValueError(f"Invalid OJBench problem identity: {dataset}/{record_id!r}") + + +def ensure_problem_data(dataset: str, problem_id: str) -> Path: + """Download one immutable problem package into the mounted host cache.""" + + if dataset not in {"NOI", "ICPC"}: + raise ValueError(f"Invalid OJBench dataset: {dataset}") + pattern = r"loj-[0-9]+" if dataset == "NOI" else r"[A-Za-z0-9_-]+" + if re.fullmatch(pattern, problem_id) is None: + raise ValueError(f"Invalid OJBench problem id: {problem_id}") + + root = problem_cache_dir() + snapshot_download( + repo_id=DATASET_REPOSITORY, + repo_type="dataset", + revision=DATASET_REVISION, + allow_patterns=[f"{dataset}/{problem_id}/**"], + local_dir=root, + ) + problem = root / dataset / problem_id + if not (problem / "init.yml").is_file(): + raise FileNotFoundError(f"Missing OJBench init.yml for {dataset}/{problem_id}") + if not any(problem.glob("*.zip")): + raise FileNotFoundError( + f"Missing OJBench test archive for {dataset}/{problem_id}" + ) + return problem + + +def record_to_sample(record: dict[str, object]) -> Sample: + dataset = str(record["dataset"]) + language = str(record["language"]) + if language not in {"python", "cpp"}: + raise ValueError(f"Unsupported OJBench language: {language}") + raw_id = record["id"] + if not isinstance(raw_id, (int, str)): + raise TypeError("OJBench id must be an integer or string") + canonical_dataset, problem_id = canonical_problem_id(dataset, raw_id) + return Sample( + id=f"{problem_id}:{language}", + input=str(record["prompt"]), + target="", + metadata={ + "record_id": raw_id, + "problem_id": problem_id, + "dataset": canonical_dataset, + "language": language, + "difficulty": str(record["difficulty"]), + "dataset_revision": DATASET_REVISION, + }, + ) + + +def get_ojbench_dataset(language: Literal["python", "cpp"]) -> MemoryDataset: + """Load one official 232-problem language track.""" + + if language not in {"python", "cpp"}: + raise ValueError("OJBench language must be 'python' or 'cpp'") + problem_cache_dir() + rows = [json.loads(line) for line in _prompt_path().read_text().splitlines()] + if len(rows) != PROMPT_COUNT: + raise ValueError(f"OJBench expected {PROMPT_COUNT} prompts, got {len(rows)}") + samples = [record_to_sample(row) for row in rows if row["language"] == language] + if len(samples) != PROMPT_COUNT // 2: + raise ValueError(f"OJBench {language} expected 232 prompts, got {len(samples)}") + return MemoryDataset( + samples=samples, + name=f"ojbench_{language}", + location=DATASET_REPOSITORY, + ) diff --git a/src/openbench/evals/ojbench/Dockerfile b/src/openbench/evals/ojbench/Dockerfile new file mode 100644 index 00000000..741a7ed1 --- /dev/null +++ b/src/openbench/evals/ojbench/Dockerfile @@ -0,0 +1,51 @@ +FROM python:3.11-slim@sha256:94c50be2dc994b873b55bc123e95e6dbade08095b3dfd790f51c34de3f08cbb7 AS builder + +ARG DMOJ_REVISION=f098cd3a49a60186d1fadde5132329ec5f4f2213 +ARG OJBENCH_REVISION=5e94480b1e135b98855cf5bc81213c256aff5b17 + +RUN apt-get update \ + && apt-get install --yes --no-install-recommends build-essential git libseccomp-dev \ + && rm -rf /var/lib/apt/lists/* + +RUN python -m pip install --no-cache-dir \ + Cython==3.0.10 setuptools==69.5.1 wheel==0.43.0 \ + && git clone https://github.com/DMOJ/judge-server.git /src/dmoj \ + && git -C /src/dmoj checkout "${DMOJ_REVISION}" \ + && test "$(git -C /src/dmoj rev-parse HEAD)" = "${DMOJ_REVISION}" \ + && git clone https://github.com/He-Ren/OJBench.git /src/ojbench \ + && git -C /src/ojbench checkout "${OJBENCH_REVISION}" \ + && test "$(git -C /src/ojbench rev-parse HEAD)" = "${OJBENCH_REVISION}" \ + && python -m pip wheel --no-build-isolation --no-cache-dir --wheel-dir /wheels \ + filelock==3.18.0 loguru==0.7.2 PyYAML==6.0.2 setuptools==69.5.1 \ + tqdm==4.66.4 watchdog==4.0.1 termcolor==2.4.0 Pygments==2.18.0 \ + setproctitle==1.3.3 pylru==1.2.1 /src/dmoj \ + && python -m pip wheel --no-build-isolation --no-cache-dir --no-deps \ + --wheel-dir /wheels /src/ojbench + +FROM python:3.11-slim@sha256:94c50be2dc994b873b55bc123e95e6dbade08095b3dfd790f51c34de3f08cbb7 + +RUN apt-get update \ + && apt-get install --yes --no-install-recommends g++ libseccomp2 pypy3 \ + && rm -rf /var/lib/apt/lists/* \ + && useradd --create-home --uid 1000 judge \ + && mkdir --parents /workspace /problems /opt/ojbench \ + && chown judge:judge /workspace + +COPY --from=builder /wheels /wheels +COPY --from=builder /src/ojbench/ojbench /opt/ojbench/package/ojbench +COPY --from=builder /src/ojbench/ojbench/config.yaml /opt/ojbench/config.yaml +COPY --from=builder /src/ojbench/ojbench/runtime.yaml /opt/ojbench/runtime.yaml +COPY --from=builder /src/ojbench/LICENSE /usr/share/doc/ojbench/LICENSE + +RUN python -m pip install --no-cache-dir /wheels/*.whl \ + && rm -rf /wheels + +ENV OJBENCH_CONFIG_PATH=/opt/ojbench/config.yaml \ + OJBENCH_RUNTIME_PATH=/opt/ojbench/runtime.yaml \ + HOME=/home/judge \ + PYTHONPATH=/opt/ojbench/package \ + PYTHONUNBUFFERED=1 + +USER judge +WORKDIR /workspace +CMD ["tail", "-f", "/dev/null"] diff --git a/src/openbench/evals/ojbench/__init__.py b/src/openbench/evals/ojbench/__init__.py new file mode 100644 index 00000000..10867ff2 --- /dev/null +++ b/src/openbench/evals/ojbench/__init__.py @@ -0,0 +1,5 @@ +"""OJBench tasks.""" + +from openbench.evals.ojbench.ojbench import ojbench_cpp, ojbench_python + +__all__ = ["ojbench_cpp", "ojbench_python"] diff --git a/src/openbench/evals/ojbench/compose.yaml b/src/openbench/evals/ojbench/compose.yaml new file mode 100644 index 00000000..c7361c2d --- /dev/null +++ b/src/openbench/evals/ojbench/compose.yaml @@ -0,0 +1,22 @@ +services: + default: + build: + context: . + init: true + command: tail -f /dev/null + user: "1000:1000" + network_mode: none + read_only: true + volumes: + - ${OPENBENCH_OJBENCH_DATA_DIR:?OJBench data cache is not configured}:/problems:ro + tmpfs: + - /workspace:rw,exec,nosuid,nodev,uid=1000,gid=1000,mode=0700,size=1073741824 + - /tmp:rw,exec,nosuid,nodev,uid=1000,gid=1000,mode=0700,size=4294967296 + cap_drop: + - ALL + cap_add: + - SYS_PTRACE + security_opt: + - no-new-privileges:true + pids_limit: 128 + mem_limit: 3g diff --git a/src/openbench/evals/ojbench/ojbench.py b/src/openbench/evals/ojbench/ojbench.py new file mode 100644 index 00000000..741fc8af --- /dev/null +++ b/src/openbench/evals/ojbench/ojbench.py @@ -0,0 +1,37 @@ +"""Official OJBench language tracks with DMOJ scoring.""" + +from pathlib import Path +from typing import Literal + +from inspect_ai import Epochs, Task, task +from inspect_ai.solver import generate + +from openbench.datasets.ojbench import get_ojbench_dataset +from openbench.scorers.ojbench import ojbench_scorer + +COMPOSE_PATH = (Path(__file__).parent / "compose.yaml").resolve() + + +def _ojbench_task(language: Literal["python", "cpp"], total_timeout: int) -> Task: + return Task( + name=f"ojbench_{language}", + dataset=get_ojbench_dataset(language), + solver=generate(), + scorer=ojbench_scorer(total_timeout=total_timeout), + sandbox=("docker", str(COMPOSE_PATH)), + epochs=Epochs(8, reducer=["mean", "pass_at_1", "pass_at_8"]), + ) + + +@task +def ojbench_python(total_timeout: int = 3600) -> Task: + """Evaluate the 232 official Python prompts with eight samples each.""" + + return _ojbench_task("python", total_timeout) + + +@task +def ojbench_cpp(total_timeout: int = 3600) -> Task: + """Evaluate the 232 official C++ prompts with eight samples each.""" + + return _ojbench_task("cpp", total_timeout) diff --git a/src/openbench/scorers/ojbench.py b/src/openbench/scorers/ojbench.py new file mode 100644 index 00000000..41797e98 --- /dev/null +++ b/src/openbench/scorers/ojbench.py @@ -0,0 +1,86 @@ +"""OJBench scorer backed by the pinned DMOJ sandbox.""" + +from __future__ import annotations + +import asyncio +import json +import threading +from pathlib import Path + +from inspect_ai.scorer import ( + CORRECT, + INCORRECT, + Score, + Scorer, + Target, + accuracy, + scorer, + stderr, +) +from inspect_ai.solver import TaskState +from inspect_ai.util import sandbox + +from openbench.datasets.ojbench import ensure_problem_data + +_DOWNLOAD_LOCK = threading.Lock() + + +def _prepare_problem(dataset: str, problem_id: str) -> None: + with _DOWNLOAD_LOCK: + ensure_problem_data(dataset, problem_id) + + +@scorer(metrics=[accuracy(), stderr()]) +def ojbench_scorer(total_timeout: int = 3600) -> Scorer: + """Judge one completion without exposing problem archives to Inspect logs.""" + + if total_timeout <= 0: + raise ValueError("total_timeout must be positive") + + async def score(state: TaskState, target: Target) -> Score: + del target + dataset = str(state.metadata["dataset"]) + problem_id = str(state.metadata["problem_id"]) + await asyncio.to_thread(_prepare_problem, dataset, problem_id) + + payload = { + "record_id": state.metadata["record_id"], + "problem_id": problem_id, + "language": state.metadata["language"], + "completion": state.output.completion, + } + environment = sandbox() + payload_path = ".openbench_ojbench_payload.json" + runner_path = ".openbench_ojbench_runner.py" + await environment.write_file(payload_path, json.dumps(payload)) + await environment.write_file( + runner_path, + Path(__file__).with_name("ojbench_runner.py").read_text(), + ) + try: + result = await environment.exec( + ["python", runner_path, payload_path], + timeout=total_timeout, + timeout_retry=False, + ) + except TimeoutError as exc: + raise RuntimeError("OJBench judge exceeded its total timeout") from exc + if not result.success: + raise RuntimeError("OJBench judge failed inside the DMOJ sandbox") + try: + evaluation = json.loads(result.stdout.strip().splitlines()[-1]) + except (IndexError, json.JSONDecodeError) as exc: + raise RuntimeError("OJBench judge returned an invalid result") from exc + if not isinstance(evaluation.get("passed"), bool): + raise RuntimeError("OJBench judge omitted its pass verdict") + return Score( + value=CORRECT if evaluation["passed"] else INCORRECT, + answer=state.output.completion, + explanation=( + f"OJBench verdict={evaluation['verdict']}; " + f"tests={evaluation['tests_run']}" + ), + metadata={"partial_passed": evaluation["partial_passed"]}, + ) + + return score diff --git a/src/openbench/scorers/ojbench_runner.py b/src/openbench/scorers/ojbench_runner.py new file mode 100644 index 00000000..ddda5d35 --- /dev/null +++ b/src/openbench/scorers/ojbench_runner.py @@ -0,0 +1,69 @@ +"""Container-side adapter around the pinned OJBench/DMOJ judge.""" + +from __future__ import annotations + +import json +import os +import sys +from pathlib import Path +from typing import Any + + +def _partial_verdict( + full_verdict: str, results: list[dict[str, Any]], ratio: float +) -> str: + if not results: + return full_verdict + first_not_ac = next( + (i for i, result in enumerate(results) if result["readable_main_code"] != "AC"), + len(results), + ) + return full_verdict if first_not_ac < round(len(results) * ratio) else "AC" + + +def evaluate(payload: dict[str, Any]) -> dict[str, Any]: + import ojbench # type: ignore[import-not-found] + + ojbench.init( + problem_dirs=[Path("/problems/NOI"), Path("/problems/ICPC")], + config_path=Path(os.environ["OJBENCH_CONFIG_PATH"]), + runtime_path=Path(os.environ["OJBENCH_RUNTIME_PATH"]), + compile_lock_path=Path("/workspace/compile.lock"), + ) + verdict, results = ojbench.judge_entry( + { + "id": payload["problem_id"], + "language": payload["language"], + "content": payload["completion"], + }, + use_tqdm=False, + ) + if verdict in {"IE", "Skip"}: + raise RuntimeError(f"OJBench infrastructure verdict: {verdict}") + partial = { + label: _partial_verdict(verdict, results, ratio) == "AC" + for label, ratio in (("1/8", 1 / 8), ("1/4", 1 / 4), ("1/2", 1 / 2)) + } + return { + "passed": verdict == "AC", + "verdict": verdict, + "tests_run": len(results), + "partial_passed": partial, + } + + +def main() -> int: + payload_path = Path(sys.argv[1]) + try: + payload = json.loads(payload_path.read_text()) + payload_path.unlink(missing_ok=True) + result = evaluate(payload) + except Exception as exc: + print(json.dumps({"error": "infrastructure_error", "type": type(exc).__name__})) + return 2 + print(json.dumps(result)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_ojbench.py b/tests/test_ojbench.py new file mode 100644 index 00000000..ed7ccd3a --- /dev/null +++ b/tests/test_ojbench.py @@ -0,0 +1,157 @@ +"""Tests for the pinned OJBench adapter.""" + +import hashlib +import json +import os +import sys +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import patch + +from inspect_ai.dataset import MemoryDataset, Sample +import yaml + +from openbench.config import BENCHMARKS +from openbench.datasets.ojbench import ( + PROMPT_SHA256, + canonical_problem_id, + ensure_problem_data, + get_ojbench_dataset, + record_to_sample, +) +from openbench.evals.ojbench import ojbench_cpp, ojbench_python +from openbench.scorers.ojbench_runner import _partial_verdict, evaluate + + +def _record(language: str = "python") -> dict[str, object]: + return { + "id": 2083, + "prompt": "private prompt", + "dataset": "NOI", + "language": language, + "difficulty": "hard", + } + + +def test_record_to_sample_preserves_official_identity(): + sample = record_to_sample(_record()) + assert sample.id == "loj-2083:python" + assert sample.input == "private prompt" + assert sample.metadata["problem_id"] == "loj-2083" + assert sample.metadata["dataset"] == "NOI" + + +def test_canonical_problem_id_rejects_path_traversal(): + assert canonical_problem_id("icpc", "nwerc2022_A") == ("ICPC", "nwerc2022_A") + try: + canonical_problem_id("icpc", "../secret") + except ValueError: + pass + else: + raise AssertionError("path traversal must be rejected") + + +def test_dataset_filters_language_and_checks_release(tmp_path: Path): + rows = [_record("python"), _record("cpp")] + prompt_file = tmp_path / "full.jsonl" + prompt_file.write_text("\n".join(json.dumps(row) for row in rows)) + digest = hashlib.sha256(prompt_file.read_bytes()).hexdigest() + with ( + patch("openbench.datasets.ojbench._prompt_path", return_value=prompt_file), + patch("openbench.datasets.ojbench.PROMPT_COUNT", 2), + patch("openbench.datasets.ojbench.PROMPT_SHA256", digest), + patch.dict(os.environ, {"OPENBENCH_OJBENCH_DATA_DIR": str(tmp_path / "data")}), + ): + dataset = get_ojbench_dataset("python") + assert len(dataset) == 1 + + +def test_problem_download_uses_pinned_allow_pattern(tmp_path: Path): + problem = tmp_path / "NOI" / "loj-2083" + + def fake_download(**kwargs): + assert kwargs["allow_patterns"] == ["NOI/loj-2083/**"] + problem.mkdir(parents=True) + (problem / "init.yml").write_text("archive: tests.zip") + (problem / "tests.zip").write_bytes(b"zip") + + with ( + patch.dict(os.environ, {"OPENBENCH_OJBENCH_DATA_DIR": str(tmp_path)}), + patch( + "openbench.datasets.ojbench.snapshot_download", side_effect=fake_download + ), + ): + assert ensure_problem_data("NOI", "loj-2083") == problem + + +def test_partial_verdict_matches_official_rounding(): + results = [{"readable_main_code": "AC"}] * 7 + [{"readable_main_code": "WA"}] + assert _partial_verdict("WA", results, 1 / 8) == "AC" + assert _partial_verdict("WA", results, 1) == "WA" + + +def test_runner_uses_canonical_problem_id_without_details(): + captured: dict[str, object] = {} + + def fake_judge_entry(entry, use_tqdm): + captured.update(entry) + assert use_tqdm is False + return "WA", [{"readable_main_code": "WA"}] + + fake_ojbench = SimpleNamespace( + init=lambda **kwargs: None, judge_entry=fake_judge_entry + ) + with ( + patch.dict(sys.modules, {"ojbench": fake_ojbench}), + patch.dict( + os.environ, + { + "OJBENCH_CONFIG_PATH": "/config.yaml", + "OJBENCH_RUNTIME_PATH": "/runtime.yaml", + }, + ), + ): + result = evaluate( + { + "record_id": 2083, + "problem_id": "loj-2083", + "language": "python", + "completion": "print(0)", + } + ) + assert captured["id"] == "loj-2083" + assert result["verdict"] == "WA" + assert "results" not in result + + +def test_tasks_use_eight_samples_and_docker(): + dataset = MemoryDataset([Sample(input="x", target="")]) + with patch( + "openbench.evals.ojbench.ojbench.get_ojbench_dataset", return_value=dataset + ): + python_task = ojbench_python.__wrapped__() + cpp_task = ojbench_cpp.__wrapped__() + assert python_task.epochs == 8 + assert cpp_task.epochs == 8 + assert python_task.sandbox.type == "docker" + + +def test_compose_declares_required_dmoj_boundary(): + compose = Path(__file__).parents[1] / "src/openbench/evals/ojbench/compose.yaml" + service = yaml.safe_load(compose.read_text())["services"]["default"] + assert service["network_mode"] == "none" + assert service["read_only"] is True + assert service["cap_drop"] == ["ALL"] + assert service["cap_add"] == ["SYS_PTRACE"] + assert service["security_opt"] == ["no-new-privileges:true"] + + +def test_registry_entries(): + assert BENCHMARKS["ojbench_python"].function_name == "ojbench_python" + assert BENCHMARKS["ojbench_cpp"].function_name == "ojbench_cpp" + assert BENCHMARKS["ojbench_python"].is_alpha is True + assert BENCHMARKS["ojbench_cpp"].is_alpha is True + + +def test_prompt_checksum_literal_is_sha256(): + assert len(PROMPT_SHA256) == 64