Skip to content

fix(requirements): add missing runtime deps; pin antlr4 to 4.7.2 - #2

Open
ffrappo wants to merge 1 commit into
neulab:mainfrom
ffrappo:fix/requirements-missing-deps
Open

fix(requirements): add missing runtime deps; pin antlr4 to 4.7.2#2
ffrappo wants to merge 1 commit into
neulab:mainfrom
ffrappo:fix/requirements-missing-deps

Conversation

@ffrappo

@ffrappo ffrappo commented Aug 6, 2026

Copy link
Copy Markdown

Problem

Out of the box, bash evaluations/install.sh + bash evaluations/verify_benchmarks.sh fails on 11 of 19 benchmarks because several runtime dependencies are missing from evaluations/requirements-main.txt. The vendored packages (lm_eval, livecodebench, lmms_eval, visualwebbench) are installed with --no-deps, so their own extras are never pulled in.

Fix

Adds four missing deps and pins antlr4 to resolve a silent three-way conflict:

dep why
tenacity lm_eval/models/api_models.py imports it for API-model retry. Without it, all 8 lm_eval API-model benchmarks (aime25, gpqa, ifeval, logiqa, humaneval_chat, mbpp_chat, mmlu_cot, acp_gen) raise ModuleNotFoundError: Attempted to use an API model, but the required packages ['tenacity'] are not installed.
anthropic livecodebench/lcb_runner/runner/claude_runner.py imports Anthropic at module load even when using the OpenAI endpoint. Without it, livecodebench fails on import.
loguru lmms_eval/tasks/__init__.py imports loguru unconditionally as eval_logger. Without it, mmmu and any lmms_eval task fail on import.
google-generativeai visualwebbench/run.py + gemini_adapter.py import it at module load. Without it, visualwebbench fails on import.
antlr4-python3-runtime==4.7.2 tarski==0.8.2 (planbench PDDL parsing) requires ATN v3 (antlr4 4.7.x). latex2sympy2 (an optional lmms_eval math util) pulls antlr4 4.13.x (ATN v4), which silently breaks planbench plan tasks with: Could not deserialize ATN with version 3 (expected 4). Pin 4.7.2 to keep planbench working.

Verification

Fresh clone + install.sh + verify_benchmarks.sh:

Before: 8 passed, 11 failed.

After: 19/19 PASS (16 text benchmarks via a text endpoint + 3 multimodal via a vision endpoint).

===================== SUMMARY =====================
  PASS  infobench  PASS  aime25  PASS  gpqa  PASS  ifeval  PASS  logiqa
  PASS  humaneval_chat  PASS  mbpp_chat  PASS  mmlu_cot  PASS  acp_gen/acp_app_gen
  PASS  livecodebench/codegeneration  PASS  repobench/repobench_xff_python
  PASS  debugbench  PASS  lifbench/onedoc-qa  PASS  planbench/task_3_plan_verification
  PASS  mmmu  PASS  visualpuzzles/cot  PASS  visualwebbench/web_caption
  PASS  beir_nfcorpus  PASS  bfcl/non_live_parallel_multiple
  19 passed, 0 failed

Tested on Ubuntu 24.04, Python 3.11, with an OpenAI-compatible endpoint.

Out of the box, == [1/3] main env: pacebench + 18 benchmarks == +
=== infobench ===
    evaluations/verify_benchmarks.sh: line 32: python: command not found

=== aime25 ===
    evaluations/verify_benchmarks.sh: line 32: python: command not found

=== gpqa ===
    evaluations/verify_benchmarks.sh: line 32: python: command not found

=== ifeval ===
    evaluations/verify_benchmarks.sh: line 32: python: command not found

=== logiqa ===
    evaluations/verify_benchmarks.sh: line 32: python: command not found

=== humaneval_chat ===
    evaluations/verify_benchmarks.sh: line 32: python: command not found

=== mbpp_chat ===
    evaluations/verify_benchmarks.sh: line 32: python: command not found

=== mmlu_cot ===
    evaluations/verify_benchmarks.sh: line 32: python: command not found

=== acp_gen/acp_app_gen ===
    evaluations/verify_benchmarks.sh: line 29: python: command not found

=== livecodebench/codegeneration ===
    evaluations/verify_benchmarks.sh: line 29: python: command not found

=== repobench/repobench_xff_python ===
    evaluations/verify_benchmarks.sh: line 29: python: command not found

=== debugbench ===
    evaluations/verify_benchmarks.sh: line 32: python: command not found

=== lifbench/onedoc-qa ===
    evaluations/verify_benchmarks.sh: line 29: python: command not found

=== planbench/task_3_plan_verification ===
    evaluations/verify_benchmarks.sh: line 29: python: command not found

=== mmmu ===
    evaluations/verify_benchmarks.sh: line 32: python: command not found

=== visualpuzzles/cot ===
    evaluations/verify_benchmarks.sh: line 29: python: command not found

=== visualwebbench/web_caption ===
    evaluations/verify_benchmarks.sh: line 29: python: command not found

=== beir_nfcorpus ===
    evaluations/verify_benchmarks.sh: line 32: python: command not found

===================== SUMMARY =====================
  FAIL  infobench  FAIL  aime25  FAIL  gpqa  FAIL  ifeval  FAIL  logiqa  FAIL  humaneval_chat  FAIL  mbpp_chat  FAIL  mmlu_cot  FAIL  acp_gen/acp_app_gen  FAIL  livecodebench/codegeneration  FAIL  repobench/repobench_xff_python  FAIL  debugbench  FAIL  lifbench/onedoc-qa  FAIL  planbench/task_3_plan_verification  FAIL  mmmu  FAIL  visualpuzzles/cot  FAIL  visualwebbench/web_caption  FAIL  beir_nfcorpus  SKIP  bfcl (evaluations/.bfcl-venv not found — run install.sh to create it)
  0 passed, 18 failed
fails on 11 of 19 benchmarks because several runtime dependencies are missing
from requirements-main.txt. The vendored packages (lm_eval, livecodebench,
lmms_eval, visualwebbench) are installed with --no-deps, so their own extras
are never pulled in.

This adds the four missing deps and pins antlr4 to resolve a silent three-way
conflict:

  tenacity           lm_eval/models/api_models.py imports it for API-model
                     retry. Without it, all 8 lm_eval API-model benchmarks
                     (aime25, gpqa, ifeval, logiqa, humaneval_chat, mbpp_chat,
                     mmlu_cot, acp_gen) raise ModuleNotFoundError.
  anthropic          livecodebench/lcb_runner/runner/claude_runner.py imports
                     Anthropic at module load even when using the OpenAI
                     endpoint. Without it, livecodebench fails on import.
  loguru             lmms_eval/tasks/__init__.py imports loguru unconditionally
                     as eval_logger. Without it, mmmu and any lmms_eval task
                     fail on import.
  google-generativeai  visualwebbench/run.py + gemini_adapter.py import it at
                     module load. Without it, visualwebbench fails on import.
  antlr4-python3-runtime==4.7.2
                     tarski 0.8.2 (planbench PDDL parsing) requires ATN v3
                     (antlr4 4.7.x). latex2sympy2 (an optional lmms_eval math
                     util) pulls antlr4 4.13.x (ATN v4), which silently breaks
                     planbench plan tasks with:
                       'Could not deserialize ATN with version 3 (expected 4)'.
                     Pin 4.7.2 to keep planbench working.

Verified: 19/19 verify_benchmarks.sh PASS after these changes
(16 text benchmarks via a text model + 3 multimodal via a vision model).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant