Skip to content

feat: expand benchmark coverage and add BFCL v4 - #1

Merged
aabbdev merged 2 commits into
mainfrom
feat/wave1-bfcl-v4
Aug 6, 2026
Merged

feat: expand benchmark coverage and add BFCL v4#1
aabbdev merged 2 commits into
mainfrom
feat/wave1-bfcl-v4

Conversation

@aabbdev

@aabbdev aabbdev commented Aug 6, 2026

Copy link
Copy Markdown
Owner

Summary

  • complete Wave 1 coverage with C-Eval, full BBH, OCRBench v1, Global PIQA, current MathArena tasks, GSM-Hard, EvalPlus, and LiveCodeBench v6
  • add hardened Docker execution and mandatory CI smoke tests for LiveCodeBench, EvalPlus, and BFCL
  • introduce provider-neutral function-calling primitives for schema conversion, parsing, typed matching, parallel calls, and stateful execution
  • add BFCL v4 single-turn (3,641 samples), official multi-turn checking (800 samples), frozen agentic memory/web tasks (665 samples), and a 5,106-sample offline weighted aggregate
  • pin upstream revisions, release assets, checksums, and Docker base images; document protocol adaptations and blocked historical-score equivalence

BFCL naming

bfcl_v4_multi_turn uses the pinned official BFCL state/response checker. The agentic environment is deliberately exposed as bfcl_v4_agentic_offline, and the weighted aggregate as bfcl_v4_offline, because frozen evidence is reproducible but not numerically identical to the live SERPAPI leaderboard environment. No ambiguous bfcl_v4 alias is introduced.

Validation

  • pytest -m "not integration and not docker" — 444 passed, 2 skipped
  • pytest -m docker -v tests/integration/test_docker_sandboxes.py — 3 passed
  • mypy . — 466 source files clean
  • pre-commit run --all-files — all hooks passed
  • uv build --quiet — wheel and sdist built
  • one-sample end-to-end mock runs completed for BFCL multi-turn and agentic tasks

Limitations

Real-model transport checks across two providers could not be run locally because no provider credentials or local model runtime were available. The provider-neutral paths and mock transport are covered, and the tasks are ready for credentialed CI/manual evaluation.

@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown

✅ Benchmark documentation has been automatically updated.

@aabbdev
aabbdev merged commit a721c7a into main Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant