feat(bfcl): add agentic live workflows

benchmark/bfcl-v4/README.md

@@ -1,12 +1,14 @@
 # BFCL v4 integration contract
 
-OpenBench exposes three independently named BFCL v4 sections and one offline
+OpenBench exposes four independently named BFCL v4 sections and one offline
 aggregate:
 
 - `bfcl_v4_single_turn`: all 13 official single-turn categories;
 - `bfcl_v4_multi_turn`: all four official stateful categories;
 - `bfcl_v4_agentic_offline`: three memory backends and two frozen web-search
   configurations;
+- `bfcl_v4_agentic_live`: BFCL's model-generated memory prerequisites and live
+  SERPAPI web backends;
 - `bfcl_v4_offline`: all 5,106 samples with BFCL's 10/10/10/30/40 weights.
 
 ## Provenance
@@ -44,17 +46,39 @@ The multi-turn task runs BFCL's pinned official state and response checker in a
 network-disabled, read-only Docker sandbox. It covers base, missing-function,
 missing-parameter, and long-context categories.
 
-The agentic task is intentionally an offline adaptation. Memory retrieval is
+The offline agentic task is intentionally an adaptation. Memory retrieval is
 initialized from the public BFCL source record, while web search uses a frozen
 corpus derived from BFCL's cited evidence. The no-snippet mode exposes URLs only
 until the model calls `fetch_url_content`. This removes SERPAPI credentials and
 web drift from CI, but it is not numerically interchangeable with the live
 leaderboard environment.
 
+The separate `bfcl_v4_agentic_live` task preserves the official agentic
+protocol:
+
+- each model generates all 37 prerequisite conversations once for each of the
+  three memory backends;
+- the resulting KV, vector, or recursive-summary state is reused for all 155
+  questions without leaking one target conversation into another;
+- `all-MiniLM-L6-v2` is pinned at revision
+  `1110a243fdf4706b3f48f1d95db1a4f5529b4d41` and all Python dependencies are
+  locked;
+- the 100 web questions run in both snippet and no-snippet modes against
+  BFCL's DuckDuckGo SerpAPI backend.
+
+The live task contains five workflow samples but reports 665 logical case
+outcomes. It requires `SERPAPI_API_KEY` for web categories. A run can record
+content-addressable, model-trace-specific web evidence with
+`-T record_web_snapshot_dir=PATH`, then replay that exact trace without network
+using `-T web_snapshot_dir=PATH`. Replay fails closed if a model changes any
+tool call or step count; a recorded trace is therefore an audit artifact, not a
+universal replacement for live search.
+
 Consequently, `bfcl_v4_offline` applies the official section weights but does
-not claim the official leaderboard score. The unqualified `bfcl_v4` alias stays
-reserved until a versioned live-search snapshot and the exact official memory
-prerequisite pipeline can run reproducibly.
+not claim the official leaderboard score. The memory prerequisite pipeline and
+live-search transport now exist, but the unqualified `bfcl_v4` alias remains
+reserved until two credentialed provider runs and differential score comparison
+against the upstream harness demonstrate parity.
 
 The older “Berkeley Function Calling” label in the Phi model card is not aliased
 to this task: without a harness/version citation it may refer to BFCL v1 rather

benchmark/bfcl-v4/live_parity.md

@@ -0,0 +1,36 @@
+# BFCL v4 live parity decision
+
+## Implemented
+
+- Upstream code and datasets are pinned to
+  `6ea57973c7a6097fd7c5915698c54c17c5b1b6c8`.
+- All five memory prerequisite conversations are checksum-verified.
+- Prerequisite calls are generated once per model, scenario, and backend.
+- Target questions start from an isolated replay of the prerequisite state.
+- The official KV, vector, recursive-summary, SerpAPI, URL-fetch, and answer
+  checker implementations run inside a dedicated Docker boundary.
+- The vector encoder and the complete Linux dependency graph are pinned.
+- Live web traces can be recorded and replayed content-exactly with networking
+  disabled.
+
+## Evidence
+
+- 155 logical cases load for each memory backend and 100 for each web mode.
+- KV, vector, and recursive-summary backends were executed in the live image;
+  each produced the official state-derived system prompt.
+- A persistent KV prerequisite followed by a target retrieval scored 1.0 in an
+  end-to-end Inspect run.
+- A recorded web-search trace replayed in the network-disabled image and scored
+  1.0; mismatched calls and unused steps fail closed.
+
+## Alias decision
+
+Do not expose `bfcl_v4` yet. No real provider credentials were available in the
+execution environment, so the required two-provider transport run and
+model-output differential against the upstream harness could not be performed.
+In addition, live SerpAPI results and fetched pages are intrinsically mutable;
+recorded snapshots prove a particular run but are not a provider-independent
+official environment.
+
+The supported live component is named `bfcl_v4_agentic_live`. The existing
+`bfcl_v4_offline` remains the reproducible full-coverage diagnostic aggregate.

benchmark/bfcl-v4/matrix.tsv

@@ -10,3 +10,9 @@ bfcl_v4_agentic_offline	pinned_dataset	1	665	5	done	Three memory backends and tw
 bfcl_v4_offline	weighted_aggregate	1	5106	5	done	Official 10/10/10/30/40 section weights applied; explicitly not labeled as the live official score
 bfcl_v4_multi_turn	mock_transport	1	1	4.41	done	End-to-end Inspect tool transport, Docker startup, solver, and scorer completed
 bfcl_v4_agentic_offline	mock_transport	1	1	0.02	done	End-to-end frozen agentic solver and scorer completed
+bfcl_v4_agentic_live	pinned_workflows	1	665	5	done	37 model-specific memory prerequisites, 465 memory targets, and 200 live web targets grouped into five race-free workflows
+bfcl_v4_agentic_live	memory_backend_smoke	1	3	1	done	KV, all-MiniLM-L6-v2 vector, and recursive-summary backends produced official state-derived prompts in Docker
+bfcl_v4_agentic_live	persistent_memory_transport	1	1	1	done	Inspect mock model wrote prerequisite state, retrieved it in an isolated target conversation, and scored 1.0
+bfcl_v4_agentic_live	web_snapshot_replay	1	1	1	done	Exact recorded tool trace replayed in a network-disabled image and scored 1.0
+bfcl_v4_agentic_live	two_provider_transport	1	0	0	blocked	No provider credentials or configured local model were available; no real-model score is claimed
+bfcl_v4	official_alias_gate	1	0	0	blocked	Alias withheld pending two-provider runs and differential output scoring against the upstream harness

docs/snippets/benchmarks.data.mdx

@@ -1571,6 +1571,21 @@ export const benchmarksData = [
     "function_name": "bbq_ses",
     "is_alpha": false
   },
+  {
+    "name": "BFCL v4 Agentic Live",
+    "description": "Official model-specific BFCL memory pipelines and live SERPAPI web search",
+    "category": "core",
+    "tags": [
+      "function-calling",
+      "tools",
+      "bfcl",
+      "agentic",
+      "live",
+      "docker"
+    ],
+    "function_name": "bfcl_v4_agentic_live",
+    "is_alpha": false
+  },
   {
     "name": "BFCL v4 Agentic Offline",
     "description": "Reproducible offline BFCL v4 memory and frozen web-search adaptation",

packages/openbench-core/pyproject.toml

@@ -51,7 +51,16 @@ include = ["openbench*"]
 [tool.setuptools.package-data]
 "openbench.evals.livecodebench" = ["Dockerfile", "compose.yaml"]
 "openbench.evals.evalplus" = ["Dockerfile", "compose.yaml"]
-"openbench.evals.bfcl" = ["Dockerfile", "compose.yaml", "runner.py"]
+"openbench.evals.bfcl" = [
+  "Dockerfile",
+  "Dockerfile.agentic",
+  "compose.yaml",
+  "compose.live.yaml",
+  "compose.replay.yaml",
+  "requirements-agentic.in",
+  "requirements-agentic.txt",
+  "runner.py",
+]
 
 [tool.setuptools.exclude-package-data]
 "*" = ["__pycache__/*", "*.pyc"]

pyproject.toml

@@ -56,7 +56,16 @@ include = ["openbench*"]
 [tool.setuptools.package-data]
 "openbench.evals.livecodebench" = ["Dockerfile", "compose.yaml"]
 "openbench.evals.evalplus" = ["Dockerfile", "compose.yaml"]
-"openbench.evals.bfcl" = ["Dockerfile", "compose.yaml", "runner.py"]
+"openbench.evals.bfcl" = [
+  "Dockerfile",
+  "Dockerfile.agentic",
+  "compose.yaml",
+  "compose.live.yaml",
+  "compose.replay.yaml",
+  "requirements-agentic.in",
+  "requirements-agentic.txt",
+  "runner.py",
+]
 
 [tool.setuptools.exclude-package-data]
 "*" = ["__pycache__/*", "*.pyc"]

src/openbench/config.py

@@ -124,6 +124,15 @@ class EvalGroup:
         function_name="bfcl_v4_agentic_offline",
         is_alpha=False,
     ),
+    "bfcl_v4_agentic_live": BenchmarkMetadata(
+        name="BFCL v4 Agentic Live",
+        description="Official model-specific BFCL memory pipelines and live SERPAPI web search",
+        category="core",
+        tags=["function-calling", "tools", "bfcl", "agentic", "live", "docker"],
+        module_path="openbench.evals.bfcl",
+        function_name="bfcl_v4_agentic_live",
+        is_alpha=False,
+    ),
     "bfcl_v4_offline": BenchmarkMetadata(
         name="BFCL v4 Offline Aggregate",
         description="All BFCL v4 sections with official weights and frozen agentic evidence",

src/openbench/datasets/bfcl.py

@@ -50,6 +50,8 @@
     "web_search_no_snippet",
 )
 
+MEMORY_SCENARIOS = ("customer", "finance", "healthcare", "notetaker", "student")
+
 _MULTI_TURN_COUNT = 200
 
 _FUNCTION_DOCS = {
@@ -115,6 +117,11 @@
     "multi_turn_func_doc/travel_booking.json": "f17b950c13adddf41d0848077df58788252e4c2e7cad5cfa71c8c4bf04f57b26",
     "multi_turn_func_doc/vehicle_control.json": "0c8a66292844874ef7b168f343bc394d8615d2d9e1f4387999a9ee23011eac78",
     "multi_turn_func_doc/web_search.json": "61fcee411e35f7ff67415e18cd67276615cf06e1c8841a683d2d997dbb46eac5",
+    "memory_prereq_conversation/memory_customer.json": "806ac91e558e3d933e526ecc0c286e417657f76a3c9b9073eef830a532eb4eb2",
+    "memory_prereq_conversation/memory_finance.json": "3cef1f235667f479b8f45c7082247454b2129ee88fe4fbcdbc53fd2cdf495b75",
+    "memory_prereq_conversation/memory_healthcare.json": "56e7bd7b2e7c01efe81f1e0be85149ea3886bb2d9cb59f90e9d9e27b91f4604d",
+    "memory_prereq_conversation/memory_notetaker.json": "07f3239ef2e9e1421901f05d3c8192469fa129324e124a47bcb35ff9c7785c91",
+    "memory_prereq_conversation/memory_student.json": "e2d6c32e9e2671c61f682e845dcdafd9eee7c9f3e2d5da1615d3afd2aeecf2f0",
 }
 
 _COUNTS = {
@@ -347,3 +354,99 @@ def get_bfcl_v4_agentic_dataset(
                 )
             )
     return MemoryDataset(samples=samples, name="bfcl_v4_agentic_offline")
+
+
+def get_bfcl_v4_agentic_live_dataset(
+    categories: list[str] | tuple[str, ...] | None = None,
+) -> MemoryDataset:
+    """Load BFCL's live agentic workflows, grouped by backend.
+
+    Each dataset sample represents one complete backend evaluation. Grouping keeps
+    the official memory prerequisite chain model-specific and prevents Inspect's
+    sample parallelism from racing the shared scenario snapshots.
+    """
+
+    selected = tuple(categories or AGENTIC_CATEGORIES)
+    unknown = set(selected) - set(AGENTIC_CATEGORIES)
+    if unknown:
+        raise ValueError(f"Unsupported BFCL live agentic categories: {sorted(unknown)}")
+
+    memory_questions = _load_jsonl("BFCL_v4_memory.json")
+    memory_answers = {
+        str(row["id"]): list(row["ground_truth"])
+        for row in _load_jsonl("possible_answer/BFCL_v4_memory.json")
+    }
+    web_questions = _load_jsonl("BFCL_v4_web_search.json")
+    web_answers = {
+        str(row["id"]): list(row["ground_truth"])
+        for row in _load_jsonl("possible_answer/BFCL_v4_web_search.json")
+    }
+    prerequisites = {
+        scenario: _load_jsonl(f"memory_prereq_conversation/memory_{scenario}.json")
+        for scenario in MEMORY_SCENARIOS
+    }
+
+    samples: list[Sample] = []
+    for category in selected:
+        is_memory = category.startswith("memory_")
+        if is_memory:
+            backend = category.removeprefix("memory_")
+            class_name = f"MemoryAPI_{backend}"
+            scenarios = []
+            for scenario in MEMORY_SCENARIOS:
+                cases = [
+                    {
+                        "id": str(question["id"]).replace("memory", category),
+                        "question": question["question"],
+                        "expected_answers": memory_answers[str(question["id"])],
+                    }
+                    for question in memory_questions
+                    if question["scenario"] == scenario
+                ]
+                scenarios.append(
+                    {
+                        "name": scenario,
+                        "prerequisites": prerequisites[scenario],
+                        "cases": cases,
+                    }
+                )
+            workflow = {"scenarios": scenarios}
+            case_count = len(memory_questions)
+        else:
+            class_name = "WebSearchAPI"
+            workflow = {
+                "cases": [
+                    {
+                        "id": str(question["id"]).replace("web_search", category),
+                        "question": question["question"],
+                        "expected_answers": web_answers[str(question["id"])],
+                    }
+                    for question in web_questions
+                ]
+            }
+            case_count = len(web_questions)
+
+        samples.append(
+            Sample(
+                id=f"{category}_workflow",
+                input=[
+                    ChatMessageUser(
+                        content=f"Run the pinned BFCL v4 {category} workflow."
+                    )
+                ],
+                target="",
+                metadata={
+                    "category": category,
+                    "functions": _load_function_docs([class_name]),
+                    "involved_classes": [class_name],
+                    "workflow": workflow,
+                    "case_count": case_count,
+                    "show_snippet": category != "web_search_no_snippet",
+                    "bfcl_revision": BFCL_REVISION,
+                    "license": BFCL_LICENSE,
+                    "live_agentic": True,
+                },
+            )
+        )
+
+    return MemoryDataset(samples=samples, name="bfcl_v4_agentic_live")

src/openbench/evals/bfcl/Dockerfile

@@ -2,9 +2,13 @@ FROM python:3.11-slim@sha256:94c50be2dc994b873b55bc123e95e6dbade08095b3dfd790f51
 
 RUN apt-get update \
     && apt-get install --yes --no-install-recommends git ca-certificates \
-    && pip install --no-cache-dir --no-deps \
+    && pip install --no-cache-dir \
+      build==1.5.0 setuptools==83.0.0 setuptools-scm==10.2.1 wheel==0.47.0 \
+    && pip install --no-cache-dir --no-build-isolation --no-deps \
       "bfcl_eval @ git+https://github.com/ShishirPatil/gorilla.git@6ea57973c7a6097fd7c5915698c54c17c5b1b6c8#subdirectory=berkeley-function-call-leaderboard" \
-    && pip install --no-cache-dir mpmath==1.3.0 \
+    && pip install --no-cache-dir \
+      filelock==3.20.0 mpmath==1.3.0 numpy==1.26.4 overrides==7.7.0 \
+      rank-bm25==0.2.2 \
     && apt-get purge --yes --auto-remove git \
     && rm -rf /var/lib/apt/lists/*
 

src/openbench/evals/bfcl/Dockerfile.agentic

@@ -0,0 +1,37 @@
+FROM python:3.11-slim@sha256:94c50be2dc994b873b55bc123e95e6dbade08095b3dfd790f51c34de3f08cbb7
+
+ARG BFCL_REVISION=6ea57973c7a6097fd7c5915698c54c17c5b1b6c8
+ARG EMBEDDING_REVISION=1110a243fdf4706b3f48f1d95db1a4f5529b4d41
+
+ENV HF_HOME=/opt/huggingface \
+    TOKENIZERS_PARALLELISM=false \
+    OMP_NUM_THREADS=1 \
+    MKL_NUM_THREADS=1
+
+COPY requirements-agentic.txt /tmp/requirements-agentic.txt
+
+RUN apt-get update \
+    && apt-get install --yes --no-install-recommends git ca-certificates \
+    && pip install --no-cache-dir \
+      --extra-index-url https://download.pytorch.org/whl/cpu \
+      -r /tmp/requirements-agentic.txt \
+    && pip install --no-cache-dir --no-build-isolation --no-deps \
+      "bfcl_eval @ git+https://github.com/ShishirPatil/gorilla.git@${BFCL_REVISION}#subdirectory=berkeley-function-call-leaderboard" \
+    && python -c "from huggingface_hub import snapshot_download; snapshot_download('sentence-transformers/all-MiniLM-L6-v2', revision='${EMBEDDING_REVISION}')" \
+    && mkdir -p /opt/huggingface/hub/models--sentence-transformers--all-MiniLM-L6-v2/refs \
+    && printf '%s' "${EMBEDDING_REVISION}" > /opt/huggingface/hub/models--sentence-transformers--all-MiniLM-L6-v2/refs/main \
+    && apt-get purge --yes --auto-remove git \
+    && rm -rf /var/lib/apt/lists/* /tmp/requirements-agentic.txt
+
+ENV HF_HUB_DISABLE_XET=1 \
+    HF_HUB_OFFLINE=1
+
+RUN useradd --create-home --uid 1000 sandbox \
+    && mkdir --parents /workspace \
+    && chown sandbox:sandbox /workspace
+
+COPY runner.py /opt/openbench/bfcl_runner.py
+
+USER sandbox
+WORKDIR /workspace
+CMD ["sleep", "infinity"]