fix(eval): add missing runtime deps; pin antlr4 4.7.2 for planbench - #3
Open
ffrappo wants to merge 1 commit into
Open
fix(eval): add missing runtime deps; pin antlr4 4.7.2 for planbench#3ffrappo wants to merge 1 commit into
ffrappo wants to merge 1 commit into
Conversation
requirements-main.txt was missing four runtime dependencies, so a fresh
`bash evaluations/install.sh` + `bash evaluations/verify_benchmarks.sh`
only passed 8 of 19 benchmarks. The vendored packages (lm_eval,
livecodebench, lmms_eval, visualwebbench) are installed with --no-deps,
so their transitive imports are never pulled in and must be listed here.
Added (each with a comment naming the benchmark it unblocks):
- tenacity>=8.5.0 lm_eval API models (api_models.py); unblocks
aime25, gpqa, ifeval, logiqa, humaneval_chat,
mbpp_chat, mmlu_cot, acp_gen (8 benchmarks)
- anthropic>=0.75.0 livecodebench (lcb_runner claude_runner.py is
imported at module load even on the OpenAI
endpoint)
- loguru>=0.7.0 lmms_eval (tasks/__init__.py imports it as
eval_logger); unblocks mmmu, visualpuzzles
- google-generativeai visualwebbench (run.py + gemini_adapter.py
import it at module load)
antlr4 three-way conflict (planbench): tarski 0.8.2 (planbench PDDL
parsing) needs antlr4-python3-runtime 4.7.x (ATN v3). lm_eval's math
extra pulls latex2sympy2, which depends on antlr4 4.13.x (ATN v4) and
silently breaks planbench plan tasks (task_1/2/4/5/6/8) with:
"Could not deserialize ATN with version 3 (expected 4)."
Pin antlr4-python3-runtime==4.7.2 and intentionally do NOT add
latex2sympy2 (mmmu-MCQ and visualpuzzles grading do not need it).
Verified end-to-end: verify_benchmarks.sh goes 8/19 PASS -> 19/19 PASS.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
evaluations/requirements-main.txtwas missing four runtime dependencies, so a freshbash evaluations/install.shfollowed bybash evaluations/verify_benchmarks.shonly passed 8 of 19 benchmarks. The vendored packages (lm_eval,livecodebench,lmms_eval,visualwebbench) are installed with--no-deps, so their transitive imports are never pulled in and must be listed explicitly.This PR adds the four missing deps and pins
antlr4to resolve a silent three-way conflict.Missing runtime deps added
tenacity>=8.5.0lm_eval/models/api_models.pyimports it for API-model retryaime25,gpqa,ifeval,logiqa,humaneval_chat,mbpp_chat,mmlu_cot,acp_gen(8 benchmarks)anthropic>=0.75.0livecodebenchlcb_runner/.../claude_runner.pyimportsAnthropicat module load, even when using the OpenAI endpointlivecodebenchloguru>=0.7.0lmms_eval/tasks/__init__.pyimportsloguruunconditionally aseval_loggermmmu,visualpuzzles(anylmms_evaltask)google-generativeaivisualwebbenchrun.py+gemini_adapter.pyimport it at module loadvisualwebbenchEach line has an inline comment naming the benchmark it unblocks.
antlr4 three-way conflict (planbench)
tarski==0.8.2(planbench PDDL parsing) requiresantlr4-python3-runtime4.7.x (ATN v3).lm_eval's math extra pullslatex2sympy2, which depends onantlr44.13.x (ATN v4) and silently breaks planbench plan tasks (task_1/2/4/5/6/8) at parse time:Resolution: pin
antlr4-python3-runtime==4.7.2and intentionally do not addlatex2sympy2(mmmu-MCQ and visualpuzzles grading do not need it). The pin is placed next to thetarski/clingoblock with an explanatory comment.Verification
Verified end-to-end on a fresh install (Python 3.11) with
bash evaluations/verify_benchmarks.sh:verify_benchmarks.shChanges
evaluations/requirements-main.txt: +12 lines (4 deps +antlr4pin + comments). No other files touched.