fix(requirements): add missing runtime deps; pin antlr4 to 4.7.2 - #2
Open
ffrappo wants to merge 1 commit into
Open
fix(requirements): add missing runtime deps; pin antlr4 to 4.7.2#2ffrappo wants to merge 1 commit into
ffrappo wants to merge 1 commit into
Conversation
Out of the box, == [1/3] main env: pacebench + 18 benchmarks == +
=== infobench ===
evaluations/verify_benchmarks.sh: line 32: python: command not found
=== aime25 ===
evaluations/verify_benchmarks.sh: line 32: python: command not found
=== gpqa ===
evaluations/verify_benchmarks.sh: line 32: python: command not found
=== ifeval ===
evaluations/verify_benchmarks.sh: line 32: python: command not found
=== logiqa ===
evaluations/verify_benchmarks.sh: line 32: python: command not found
=== humaneval_chat ===
evaluations/verify_benchmarks.sh: line 32: python: command not found
=== mbpp_chat ===
evaluations/verify_benchmarks.sh: line 32: python: command not found
=== mmlu_cot ===
evaluations/verify_benchmarks.sh: line 32: python: command not found
=== acp_gen/acp_app_gen ===
evaluations/verify_benchmarks.sh: line 29: python: command not found
=== livecodebench/codegeneration ===
evaluations/verify_benchmarks.sh: line 29: python: command not found
=== repobench/repobench_xff_python ===
evaluations/verify_benchmarks.sh: line 29: python: command not found
=== debugbench ===
evaluations/verify_benchmarks.sh: line 32: python: command not found
=== lifbench/onedoc-qa ===
evaluations/verify_benchmarks.sh: line 29: python: command not found
=== planbench/task_3_plan_verification ===
evaluations/verify_benchmarks.sh: line 29: python: command not found
=== mmmu ===
evaluations/verify_benchmarks.sh: line 32: python: command not found
=== visualpuzzles/cot ===
evaluations/verify_benchmarks.sh: line 29: python: command not found
=== visualwebbench/web_caption ===
evaluations/verify_benchmarks.sh: line 29: python: command not found
=== beir_nfcorpus ===
evaluations/verify_benchmarks.sh: line 32: python: command not found
===================== SUMMARY =====================
FAIL infobench FAIL aime25 FAIL gpqa FAIL ifeval FAIL logiqa FAIL humaneval_chat FAIL mbpp_chat FAIL mmlu_cot FAIL acp_gen/acp_app_gen FAIL livecodebench/codegeneration FAIL repobench/repobench_xff_python FAIL debugbench FAIL lifbench/onedoc-qa FAIL planbench/task_3_plan_verification FAIL mmmu FAIL visualpuzzles/cot FAIL visualwebbench/web_caption FAIL beir_nfcorpus SKIP bfcl (evaluations/.bfcl-venv not found — run install.sh to create it)
0 passed, 18 failed
fails on 11 of 19 benchmarks because several runtime dependencies are missing
from requirements-main.txt. The vendored packages (lm_eval, livecodebench,
lmms_eval, visualwebbench) are installed with --no-deps, so their own extras
are never pulled in.
This adds the four missing deps and pins antlr4 to resolve a silent three-way
conflict:
tenacity lm_eval/models/api_models.py imports it for API-model
retry. Without it, all 8 lm_eval API-model benchmarks
(aime25, gpqa, ifeval, logiqa, humaneval_chat, mbpp_chat,
mmlu_cot, acp_gen) raise ModuleNotFoundError.
anthropic livecodebench/lcb_runner/runner/claude_runner.py imports
Anthropic at module load even when using the OpenAI
endpoint. Without it, livecodebench fails on import.
loguru lmms_eval/tasks/__init__.py imports loguru unconditionally
as eval_logger. Without it, mmmu and any lmms_eval task
fail on import.
google-generativeai visualwebbench/run.py + gemini_adapter.py import it at
module load. Without it, visualwebbench fails on import.
antlr4-python3-runtime==4.7.2
tarski 0.8.2 (planbench PDDL parsing) requires ATN v3
(antlr4 4.7.x). latex2sympy2 (an optional lmms_eval math
util) pulls antlr4 4.13.x (ATN v4), which silently breaks
planbench plan tasks with:
'Could not deserialize ATN with version 3 (expected 4)'.
Pin 4.7.2 to keep planbench working.
Verified: 19/19 verify_benchmarks.sh PASS after these changes
(16 text benchmarks via a text model + 3 multimodal via a vision model).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Out of the box,
bash evaluations/install.sh+bash evaluations/verify_benchmarks.shfails on 11 of 19 benchmarks because several runtime dependencies are missing fromevaluations/requirements-main.txt. The vendored packages (lm_eval,livecodebench,lmms_eval,visualwebbench) are installed with--no-deps, so their own extras are never pulled in.Fix
Adds four missing deps and pins antlr4 to resolve a silent three-way conflict:
tenacitylm_eval/models/api_models.pyimports it for API-model retry. Without it, all 8 lm_eval API-model benchmarks (aime25, gpqa, ifeval, logiqa, humaneval_chat, mbpp_chat, mmlu_cot, acp_gen) raiseModuleNotFoundError: Attempted to use an API model, but the required packages ['tenacity'] are not installed.anthropiclivecodebench/lcb_runner/runner/claude_runner.pyimportsAnthropicat module load even when using the OpenAI endpoint. Without it, livecodebench fails on import.logurulmms_eval/tasks/__init__.pyimportsloguruunconditionally aseval_logger. Without it, mmmu and any lmms_eval task fail on import.google-generativeaivisualwebbench/run.py+gemini_adapter.pyimport it at module load. Without it, visualwebbench fails on import.antlr4-python3-runtime==4.7.2tarski==0.8.2(planbench PDDL parsing) requires ATN v3 (antlr4 4.7.x).latex2sympy2(an optional lmms_eval math util) pulls antlr4 4.13.x (ATN v4), which silently breaks planbench plan tasks with:Could not deserialize ATN with version 3 (expected 4). Pin 4.7.2 to keep planbench working.Verification
Fresh clone +
install.sh+verify_benchmarks.sh:Before: 8 passed, 11 failed.
After: 19/19 PASS (16 text benchmarks via a text endpoint + 3 multimodal via a vision endpoint).
Tested on Ubuntu 24.04, Python 3.11, with an OpenAI-compatible endpoint.