Skip to content

feat(bfcl): add agentic live workflows - #2

Merged
aabbdev merged 2 commits into
mainfrom
feat/bfcl-v4-live-parity
Aug 6, 2026
Merged

feat(bfcl): add agentic live workflows#2
aabbdev merged 2 commits into
mainfrom
feat/bfcl-v4-live-parity

Conversation

@aabbdev

@aabbdev aabbdev commented Aug 6, 2026

Copy link
Copy Markdown
Owner

Summary

  • add bfcl_v4_agentic_live for the three official memory backends and both live web-search modes
  • execute all 37 model-generated memory prerequisites once per backend/scenario, then isolate each of the 465 target conversations from the resulting state
  • run BFCL's pinned KV, vector, recursive-summary, SerpAPI, URL-fetch, and answer-checking implementations in a dedicated persistent Docker runner
  • pin the BFCL commit, complete Linux dependency graph, CPU PyTorch build, and all-MiniLM-L6-v2 revision
  • support content-exact web trace recording and fail-closed replay in a network-disabled sandbox
  • block private-network URL fetches while preserving public BFCL web behavior

Alias decision

This PR deliberately does not add an unqualified bfcl_v4 alias. The complete live transport now exists, but no provider credentials were available to run the required two-provider comparison against the upstream harness. bfcl_v4_offline remains the reproducible aggregate; bfcl_v4_agentic_live is the credentialed official-protocol component.

Validation

  • pytest -m "not integration and not docker" — 446 passed, 2 skipped
  • pytest -m docker -v tests/integration/test_docker_sandboxes.py — 4 passed
  • mypy . — 466 source files clean
  • pre-commit run --all-files — all hooks passed
  • uv build --quiet — wheel and sdist built
  • all three official memory backends loaded and derived prompts from persisted state in Docker
  • end-to-end mock prerequisite write/target retrieval scored 1.0
  • exact web snapshot replay scored 1.0 with networking disabled
  • agentic answer verdicts matched the upstream checker on differential cases

Credentials still required

A full live run requires SERPAPI_API_KEY plus two configured model providers. No real-model score is claimed by this PR.

@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown

✅ Benchmark documentation has been automatically updated.

@aabbdev
aabbdev merged commit 7f11867 into main Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant