fix(lcb-service): use the fork start method explicitly - #433
Conversation
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #433 +/- ##
=======================================
Coverage ? 79.73%
=======================================
Files ? 151
Lines ? 20592
Branches ? 0
=======================================
Hits ? 16419
Misses ? 4173
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
e9aa1db to
4ff0df4
Compare
|
Verified on the Python 3.14 lcb-service image (forkserver default): grading works with the fork pin; dead grading children classify as -6, and the all-infra-errors check raises, so the original failure is still caught; an all-timeout batch scores 0 without raising — that case would have raised on commit one fix. |
47115a5 to
9c6739e
Compare
9c6739e to
3dfa3ca
Compare
83a5f2f to
3dfa3ca
Compare
|
|
||
| with ProcessPoolExecutor(max_workers=self.n_lcb_workers) as executor: | ||
| with ProcessPoolExecutor( | ||
| max_workers=self.n_lcb_workers, mp_context=_MP_CTX |
There was a problem hiding this comment.
High — concurrency: LCBServe.evaluate() is invoked through event_loop.run_in_executor() in _server.py, so this outer ProcessPoolExecutor is created from a worker thread in the already multithreaded uvicorn process. Forcing it to use fork can inherit locks held by threads that disappear in the child, leaving grading workers deadlocked and the evaluation hung. The reported compatibility problem establishes that the inner grading child needs fork semantics, but not that the outer pool does. Please use separate contexts—for example, forkserver for this outer pool and fork only for the Manager/Process inside run_code_subprocess, where the pool worker is single-threaded.
There was a problem hiding this comment.
High — concurrency:
LCBServe.evaluate()is invoked throughevent_loop.run_in_executor()in_server.py, so this outerProcessPoolExecutoris created from a worker thread in the already multithreaded uvicorn process. Forcing it to useforkcan inherit locks held by threads that disappear in the child, leaving grading workers deadlocked and the evaluation hung. The reported compatibility problem establishes that the inner grading child needs fork semantics, but not that the outer pool does. Please use separate contexts—for example,forkserverfor this outer pool andforkonly for theManager/Processinsiderun_code_subprocess, where the pool worker is single-threaded.
Good catch — tried forkserver as suggested, it passed locally but hung at pool shutdown in the lcb-service container (Python 3.14.5, unreaped zombie workers), so I switched the outer pool to spawn instead: worker startup is a bit slower but it's thread-safe for the same reason and all local + container tests pass; the grading child keeps fork. Let me know if you see any other issues.
| # submitted code. | ||
| res = [-1] * len(suite["inputs"]) | ||
| metadata = { | ||
| "error": "Grading subprocess died before reporting a result", |
There was a problem hiding this comment.
Medium — correctness: execute_code_single_suppressed_errors() catches Exception, but SystemExit derives from BaseException. A submitted solution that calls sys.exit() therefore terminates the grading child without filling resp_buffer, and this branch classifies it as GradingChildDied (-6). For a one-sample batch—or when every submission exits—the all-infrastructure-error guard raises RuntimeError instead of recording ordinary failed submissions. This was reproduced through the actual LiveCodeBench path. Please catch SystemExit as a submission runtime failure and account for os._exit, which can bypass exception handling entirely.
There was a problem hiding this comment.
Good catch — I did see SystemExit getting mixed in with GradingChildDied. Turns out it's specifically the call-based (fn_name) grading path, since the stdio path already guards against it internally. Added a SystemExit except guard as suggested (bc502f1), scoring it as its own -7 SubmissionExit instead of -6, so it doesn't get miscounted as an infra failure anymore.
Let me know if you still see any issues.
nv-alicheng
left a comment
nv-alicheng
left a comment
There was a problem hiding this comment.
Review Council — Multi-AI Code Review
Reviewed by: Claude + Code-Quality (×2: diff + import-neighborhood, per request) | Depth: quick + forced code-quality
codex was unavailable in this environment. See the summary comment for neighborhood findings on _server.py/run_lcb_tests.py and untouched-line items that can't be posted inline.
| metadata = { | ||
| "error": "Grading subprocess died before reporting a result", | ||
| "error_code": -6, | ||
| "error_message": f"GradingChildDied (exitcode={p.exitcode})", |
There was a problem hiding this comment.
[Claude] high (data-integrity): The -6 GradingChildDied branch conflates a broken judge with a submission that kills its own grading process, and -6 is in _LCB_INFRA_ERROR_CODES (line 64), so an all-crash batch trips the infra_errors == len(futures) RuntimeError (line 361) and refuses to report a legitimately-0 score — the exact opposite of this PR's stated goal. Untrusted submitted code runs inside the child (_MP_CTX.Process(target=execute_code_single_suppressed_errors) → run_test(..., test=code)), and the child only records a result at the very end (resp_buffer.append(...), line 122). Any path terminating the interpreter before that append — and before the SystemExit/except handlers run — leaves resp_buffer empty with p.is_alive()==False, landing here as -6. Submission-controlled ways to hit it (all bypass Python exception handling, so the new SystemExit→-7 fix at :106 does NOT cover them): os._exit(0); a native segfault (malicious/broken numpy, ctypes, C-extension); an OOM SIGKILL from a huge allocation. So -6 is reachable by ordinary adversarial/bad model output, not just infra. Consequence: a batch where every submission crashes its interpreter aborts with RuntimeError instead of reporting the correct pass@1=0 — discarding a true 0 score.
Root cause / fix: the infra-vs-submission distinction is asserted in a comment + a set literal, not derived from how the child died. Reserve -6 for judge-startup failures (the fork/forkserver problem this PR targets) and classify submission-self-terminated deaths as a submission fault alongside -7 — e.g. treat a clean exitcode==0 empty-buffer as submission fault, and exclude negative/signal exit codes a submission can self-induce from _LCB_INFRA_ERROR_CODES. (Distinct from the already-fixed SystemExit miscount at :176.)
There was a problem hiding this comment.
Good catch — os._exit/segfault/OOM do bypass the -7 guard entirely. Went with a shared started-flag instead of exit codes (a submission can fake any exit code via os._exit): the child flips it right before grading starts, so pre-flag deaths stay -6 infra and post-flag deaths become -8 SubmissionKilledChild, scored as a normal fail. Verified in the container that an all-os._exit batch now reports a true 0 instead of tripping the guard (83db6ed).
| ) + flat_timeout_extension | ||
|
|
||
| manager = mp.Manager() | ||
| manager = _MP_CTX.Manager() |
There was a problem hiding this comment.
[Quality] medium (code-quality): manager = _MP_CTX.Manager() creates a full multiprocessing Manager (a separate server process) for every single code sample and never shuts it down — no manager.shutdown(), no with — so it's only reclaimed by GC finalizer and manager processes leak under load. This compounds with the fork change: run_code_subprocess already runs inside a forked ProcessPoolExecutor worker, so each grade is pool-worker-fork → Manager-server-fork → grading-Process-fork, nested fork-from-fork each carrying inherited parent state. Wrap in with _MP_CTX.Manager() as manager: (or reuse one manager per _LCBWorker batch) to bound process count deterministically.
There was a problem hiding this comment.
Fair point — this actually predates this PR, but worth tightening while we're in here: wrapped it in a with-block and added a join() after kill so a killed grading child gets reaped instead of lingering as a zombie (1918fd7).
| _LCB_INFRA_ERROR_CODES = {-5, -6} | ||
|
|
||
|
|
||
| def execute_code_single(test_suite_json: str, code: str, timeout_sec: int = 60): |
There was a problem hiding this comment.
[Quality] low (code-quality): Missing return-type annotations on the three grading-path functions that cross the process boundary: execute_code_single (67), execute_code_single_suppressed_errors (91), run_code_subprocess (126) — all return tuple[list, dict] (results + metadata) but none declare it, so the res, metadata = ... unpack the whole error-code protocol depends on is unchecked. Relatedly, the fork target at line 91 is def execute_code_single_suppressed_errors(*args, resp_buffer=None, **kwargs) — fully untyped variadics exactly where _MP_CTX.Process(target=..., args=..., kwargs=...) wires args, so a wrong positional/keyword fails only at runtime inside the child (surfacing as a spurious -6). Give both concrete signatures / a named result-tuple alias.
There was a problem hiding this comment.
Done — gave the fork target named parameters and declared tuple[list, dict] on all three grading helpers (7f4d040); the variadics were from previous prs but agreed they hid wiring mistakes until runtime.
| if timed_out: | ||
| p.kill() | ||
|
|
||
| if len(resp_buffer) == 0: |
There was a problem hiding this comment.
[Both] low (code-quality): The two no-result branches duplicate res = [-1] * len(suite["inputs"]) plus a same-shape {error, error_code, error_message} dict (timeout arm 164-169, child-died arm 174-179), and use an else-after-return. A guard clause flattens it: if resp_buffer: return resp_buffer[0] up front, then handle the empty case with no nested else; hoist the shared res assignment and set only the differing metadata per branch.
There was a problem hiding this comment.
Done — early return for the reported-result case and hoisted the shared res (35feb8e); the duplication was honestly my own doing from stacking the attribution branches one review round at a time.
Review Council — Multi-AI Code ReviewReviewed by: Claude + Code-Quality (run twice — diff scope + import-neighborhood, per request) | Depth: quick with code-quality forced on (normally skipped at quick depth) Tight, well-reasoned fix. One high-severity correctness gap survives it, plus code-quality/neighborhood items you asked for. Existing bot/human threads corroborated but not duplicated (see bottom). 🔴 Must Fix (high)
🟡 Should Fix (medium)
🔵 Consider (low)
Existing threads (corroborated, not duplicated)
|
|
One thing to note - I think this is a bug on my part: I'd used py3.14 as the base container for the lcb_runner, but maybe this also should be studied: From the lcb_runner repo, they use py3.11 (https://github.com/LiveCodeBench/LiveCodeBench) I have personally not checked but there might be some variance between python versions (i.e. version specific language features like walrus operator, same-line multi-context managers, etc.), and the code generated by the worker should be either version-agnostic or catered to a specific version. This can get solved a little more easily if we just pin to a python 3.11 container for lcb-service. |
3.14's forkserver default is what exposed this, but rolling back to 3.11 would just hide it — fork-from-thread, the -6/-8 attribution, and the SystemExit handling are all version-independent bugs that'd still exist, so I don't see much upside to reverting now. |
Python 3.14 changed the default multiprocessing start method on Linux from fork to forkserver. The grading pipeline (pool workers forking a per-problem mp.Process + mp.Manager) only works with fork: under forkserver the grading children die at startup, every result comes back as an error, and execute_code_single_suppressed_errors turns that into all-failed tests, so the service sits at 0/N forever. Pin the fork context for the executor, the per-problem Process and its Manager. Also raise if every subprocess reported an execution error -- that means the judge is broken, not that all samples failed -- and log those errors at error level instead of warning. Seen on a python 3.14 lcb-service image: 0/349 after 3.5h, one defunct child per pool worker. Same inputs with fork forced: done in 6 min. The repo pins 3.12 so CI won't hit this, but shipped images have.
Timeouts were counted as execution errors, so a small batch where every submission loops forever would trip the guard and raise instead of scoring 0. Split the empty-buffer case in run_code_subprocess: child still alive at the deadline -> timeout (-1, submission's fault), child exited without reporting -> new GradingChildDied (-6, judge's fault). The guard now only counts -5/-6, so the forkserver startup deaths still raise and all-timeout batches score normally. Also log the multiprocessing start method at service init; that would have made the original 0/N a one-line diagnosis.
…failure sys.exit() is a BaseException, not caught by the existing `except Exception`. grade_call_based's method invocation has no SystemExit guard (unlike the stdio path's call_method, which already does), so a call-based submission calling sys.exit() killed the grading child before it filled resp_buffer and got misclassified as -6 GradingChildDied - an infra error that can trip the all-errors guard even for a single-sample batch. Give it its own code (-7) instead, kept out of _LCB_INFRA_ERROR_CODES.
evaluate() runs on an executor thread (the server dispatches it via run_in_executor), so the per-request ProcessPoolExecutor was forking an already-multithreaded process - a known deadlock risk: only the forking thread survives in the child, locks held by other threads stay locked forever. Switch the pool to spawn: fork+exec inherits no locks, so it is safe to start from a thread, and everything submitted to the pool is picklable, so it is a drop-in. Tried forkserver first, but its helper hangs at pool shutdown in the lcb-service container (Python 3.14.5) and leaks semaphores. Probed all three start methods in the deployment image: fork and spawn tear down cleanly, forkserver hangs indefinitely. The inner grading child keeps fork: grading relies on fork semantics, and forking from a freshly exec'd single-threaded pool worker is fine. The startup log now prints both start methods.
A submission can kill its own grading child in ways no except block sees (os._exit(), a native segfault, an OOM kill). That landed in the -6 GradingChildDied bucket and counted as an infrastructure error, so a batch where every submission crashed its interpreter tripped the all-infra-errors guard and aborted instead of reporting a legitimate 0 score. The child now sets a shared started flag right before grading begins, so an empty resp_buffer can be attributed: died before the flag means a judge startup failure - still -6, still counted by the guard; died after means the submission killed the interpreter - new -8 SubmissionKilledChild, scored as a normal failed sample. Exit codes cannot make this distinction because os._exit() lets the submission pick any code.
…process Return the reported result early, hoist the shared all-failed res out of the attribution branches, and keep only the metadata construction per branch. No behavior change.
Each graded sample created a Manager (its own server process) and relied on the GC finalizer to shut it down. Make the lifecycle explicit with a with-block so the process count under load is bounded deterministically, capture the child's started/exitcode state before the scope closes, and reap a killed grading child with join() instead of leaving a zombie in the pool worker.
execute_code_single_suppressed_errors is the fork target, but took fully untyped variadics, so a miswired argument only failed at runtime inside the grading child (surfacing as a spurious child-death error). Give it named parameters, and declare the tuple[list, dict] return type on all three grading helpers so the res/metadata unpacking is checked.
7f4d040 to
750f3aa
Compare
|
Kindly ping. |
Lets also revert to 3.11 to keep it consistent with LCB specification. We can do that separately. |
arekay-nv
left a comment
arekay-nv
left a comment
There was a problem hiding this comment.
Review Council — Multi-AI Code Review
Reviewers: Codex (gpt-5.6-sol) + Claude (concurrency + accuracy lenses) + Grok 4.5 (Cursor). Event: comment only (no approve/reject). Findings verified across models; a couple of single-model claims were dropped after checking (e.g. an "open Manager() makes the fork worker multi-threaded" claim — empirically the worker stays single-threaded, so the spawn-pool / fork-child split is sound).
This PR has iterated well: the earlier SystemExit→-7, outer-pool fork→spawn, Manager-leak→with+join, return-types, and DRY threads are all resolved — not re-raised here.
Posted inline (still open)
| Line | Sev | Finding |
|---|---|---|
| 117 | high | started_flag flips before judge-side setup → a judge-setup crash is mislabeled -8 and escapes the all-infra guard → silent pass@1=0 (mirror of the 83db6ed fix; boundary one frame too high). |
| 213 | high | The regression test the PR body claims does not exist anywhere in tests/. |
| 400 | med | All-infra guard is narrow: -8/-1 excluded, a single non-infra result disables it, and LCB's -4 "Error during testing" (run_lcb_tests.py:521/541) carries no "error" key so it's never classified/logged. |
| 371 | med | warning→error floods ERROR with routine timeouts/submission failures, drowning genuine -5/-6. |
Lower priority (not posted inline)
- Lazy imports in
execute_code_single(numpy,run_lcb_tests) — hoisting to module scope also shrinks the line-117 window (two birds). - PR description drift: the body says fork is used for the "outer process pool," but the code uses
spawnthere (fork only for the grading child) — please update. - Document the invariant that the
forkchild is safe only while the pool worker is single-threaded, so a future top-level import that spawns a thread doesn't silently reintroduce a fork+lock deadlock. started_flag'sValuecan belock=False(parent reads only post-join); the per-sampleManagercould be aPipe/SimpleQueue— hardening only.
Filed separately
- A pre-existing accuracy bug found during this review — an empty test suite scores every submission as PASS (
all([]) == True) — is tracked in #443 (out of scope for this PR's diff).
Orthogonal (existing thread)
The Python-3.11 discussion: pinning would only mask the (real, version-independent) MP bugs, so keep these fixes — but grading-interpreter parity with upstream LCB is a separate result-validity concern worth its own tracking, not a substitute.
| # submission) is about to execute; if the interpreter dies now it is | ||
| # the submission's doing. Deaths before this point are judge startup | ||
| # failures. | ||
| started_flag.value = True |
There was a problem hiding this comment.
high (correctness): started_flag is flipped True at wrapper entry — before the judge's own setup runs (import numpy / from .run_lcb_tests import run_test in execute_code_single, reliability_guard(), and the trusted-input json.loads/parse in grade_*). A hard interpreter death in that window (segfault on a bad numpy/arch, OOM SIGKILL, os._exit) is therefore attributed to the submission as -8 SubmissionKilledChild. Because -8 ∉ _LCB_INFRA_ERROR_CODES, infra_errors stays below len(futures), the all-infra RuntimeError guard never fires, and a systemically broken judge reports a silent pass@1 = 0 — the exact failure this PR exists to prevent.
This is the mirror/residual of the os._exit→-6 fix (83db6ed): that fix moved the attribution boundary one frame too high. The flag should bracket only the untrusted submission's execution.
Fix: set started_flag.value = True immediately before the submission actually executes — push it into run_test/grade_*, after imports + reliability_guard() + suite parse. Hoisting the two lazy imports in execute_code_single to module scope (so they run in the parent before the fork) shrinks the window further and also satisfies the no-lazy-imports rule. The line-113 comment ("submission … is about to execute") is itself inaccurate, since trusted judge setup still runs first.
Independently flagged high by the concurrency reviewer and critical by Grok 4.5.
There was a problem hiding this comment.
Fix as suggested, moved the flag into run_test itself, set right after reliability_guard()/suite parse and before dispatching into grade_call_based/grade_stdio, so it only brackets the submission's own execution; hoisted the two lazy imports to module scope like you suggested (f866eb7).
| ) | ||
| if metadata.get("error_code") in _LCB_INFRA_ERROR_CODES: | ||
| infra_errors += 1 | ||
| logger.error(f"Test execution error for question {qid}: {metadata}") |
There was a problem hiding this comment.
medium (error-handling): warning → error here fires for every sample whose metadata has an "error" key — including routine outcomes: -1 timeouts, -7 SubmissionExit, -8 SubmissionKilledChild, and ordinary submission runtime errors. On a normal accuracy batch (many slow/buggy submissions) this floods ERROR and makes the genuine -5/-6 judge-broken signals indistinguishable in ops. The infra-vs-submission split already exists (_LCB_INFRA_ERROR_CODES) — gate the level on it: ERROR for infra codes, WARNING/INFO for submission-attributed ones.
There was a problem hiding this comment.
Done — split it exactly that way: ERROR for infra codes, WARNING for submission-attributed ones (225598a).
| # broken, not that every code sample failed its tests. Timeouts are | ||
| # excluded: a batch where every submission loops forever is a valid | ||
| # 0 score, not a broken judge. | ||
| if futures and infra_errors == len(futures): |
There was a problem hiding this comment.
medium (error-handling / accuracy): The all-infra guard is narrower than its intent:
- It only fires when every future is
-5/-6. A single legitimate non-infra result among N (a real-2/-7/timeout) dropsinfra_errorsbelowlen(futures), so a judge broken for a subset of samples is silently folded into the score. -8(a post-started_flagcrash — see the line-117 thread) and-1(timeout) are excluded, so a wholesale judge crash during setup escapes entirely.- The classification gate keys on
"error" in metadata(line 368), but LCB's own outer handler returns[-4]with onlyerror_code/error_messageand no"error"key (run_lcb_tests.py:521, 541). A systemic exception insidegrade_*→ every sample-4→ not logged, not counted infra → silentpass@1 = 0.
Consider deriving the infra set structurally (an IntEnum) and treating "no result / judge-side death / grade-time exception" as infra regardless of the "error" key.
There was a problem hiding this comment.
Added the missing error key so -4 at least logs (225598a), but kept it out of _LCB_INFRA_ERROR_CODES on purpose — that outer catch is reached almost entirely by submissions that fail to compile, which is squarely the submission's fault, not the judge's; flagging it infra would abort ordinary "model wrote broken code" batches. The all-or-nothing threshold and -8/-1 exclusion are real gaps but a false-positive/negative tradeoff on the guard, so I'd rather get your take before picking a number than just change it.
There was a problem hiding this comment.
IntEnum's the same ask as before with @nv-alicheng
Oop - this is an artifact of a direct copy/paste port from the official LCB runner repo. I can take the task to clean it up to fit code standards.
05f5608 to
0066d11
Compare
Should've gone out with the earlier attribution-fix commits -- had this written already, just missed staging it at the time. Covers timeout (-1), sys.exit (-7), os._exit (-8), and judge-side deaths (-6, both before run_test and during its own setup), plus the all-infra guard: an os._exit()-only batch scores 0 without tripping it, a judge-startup-death batch does.
started_flag flipped True at wrapper entry, before run_test's own setup (reliability_guard, suite parse) ran -- a death in that window got misattributed as -8 SubmissionKilledChild instead of -6 GradingChildDied, so a real judge bug could sneak past the all-infra guard as a silent 0. Moved the flag into run_test itself, set right before dispatch to grade_call_based/grade_stdio -- the actual first line of the submission's own code. Hoisted the numpy/run_lcb_tests imports to module scope while in there too (same window, and it'd been flagged as a lazy import anyway); costs the grading child nothing since it forks from an already-warm pool worker.
89dbccb to
f866eb7
Compare
…e logging Two gaps in the all-infra guard/logging: the outer except in grade_call_based/grade_stdio's callers returns -4 with no "error" key, so a bad submission that fails to compile or define the expected function never gets logged (the classification gate keys on "error" in metadata) -- give it one. Left -4 out of _LCB_INFRA_ERROR_CODES on purpose: it's reached whenever the submission's own code fails to compile, which is the common case and is plainly not the judge's fault, not some rare harness bug worth aborting a whole run over. Also split logger.error into error (infra codes) vs warning (everything else) -- it was firing for every timeout/sys.exit/os._exit/bad-code sample, which drowns the real -5/-6 signal in ops on any batch with a few slow or broken submissions.
|
@liayan can you add more details on how to reproduce the failures. I have tried to run LCB with py3.11 and py3.14 base images on x86/linux and unable to see any failures without the PR. Can you share the steps to reproduce the |
Good catch — I originally hit this on vera-rubin with a new lcb-service image on 3.14, but also confirmed it reproduces on GB200/GB300. Just haven't checked on x86 yet. Just reproduced it again Steps:
Grading children dead as zombies under a nested tree of forkserver helper processes — same shape as the original 0/349-for-3.5h, one defunct child per pool worker. |
|
Interesting — same exact steps on x86_64 (same Python 3.14, same code): finishes in ~1s, every time, at every scale I tried (up to 349 samples / 176 workers). So this looks architecture-specific rather than Python-only-specific. |
6 participants
What
Pin the
forkmultiprocessing context for the outer process pool, per-problem process, and manager used by the LiveCodeBench service. Python 3.14 changed the Linux default toforkserver; in the shipped Python 3.14 service image that caused grading children to die at startup and left evaluation at 0/N.The patch intentionally preserves the existing x86/Python 3.12 grading contract. An empty child response remains an ordinary failed submission, with no new error codes or service-level failure conditions.
Type of change
Testing
pre-commit run --all-filespytest -m unit— 1,514 passed, 5 skippedforkcompleted grading successfullyChecklist