Python: [Bug]: AG-UI transport executes approved tool calls before `before_run` injects provider tools — approved calls to provider-injected tools silently fail

[antsok]

Description

1. Summary

In agent_framework_ag_ui, run_agent_stream resolves approval responses and executes the approved tool calls before agent.run runs the agent's before_run hooks. The tool map used for that execution is built from the static server tools only:

# _agent_run.py:1341
server_tools = collect_server_tools(agent)        # static server tools ONLY
...
# _agent_run.py:1381-1384
tools_for_execution = tools if tools is not None else server_tools
resolved_approval_results = await _resolve_approval_responses(
    messages, tools_for_execution, agent, run_kwargs, pending_approvals, thread_id
)

collect_server_tools returns the tools declared on the agent up front. It does not include tools that a ContextProvider/FileAccessProvider/TodoProvider-style harness provider registers on the per-invocation SessionContext during before_run — those tools only exist once agent.run is underway. So when a user approves a call to one of those provider-injected tools (file_access_write, file_memory_*, todos_*, load_skill, …), _resolve_approval_responses → _try_execute_function_calls cannot find the tool in tools_for_execution, produces no result for that call, and the fallback at _agent_run.py:951-952 injects:

Content.from_function_result(call_id=call_id, result="Error: Tool call invocation failed.")

The approved action never runs. The model sees the error, apologizes ("Let me retry that."), re-issues the same call, and the user is presented with a second approval card for the same action — a loop. The user believes they saved a file; nothing was written.

sequenceDiagram
  autonumber
  participant C as Client (AG-UI)
  participant EP as AG-UI endpoint
  participant R as _resolve_approval_responses
  participant AR as agent.run (before_run)

  C->>EP: assistant tool_call file_access_write + user approval
  EP->>EP: collect_server_tools(agent) -> {waf_pillars, caf_methodologies, update_mode, ask_user_questions, microsoft_docs_*}
  Note over EP: file_access_* NOT in the map (registered in before_run, which has not run)
  EP->>R: _resolve_approval_responses(tools_for_execution=server_tools)
  R->>R: _try_execute_function_calls: tool_map has no "file_access_write" -> no result
  R-->>EP: inject "Error: Tool call invocation failed." for the approved call
  EP->>AR: agent.run(messages with the error result)
  AR->>AR: before_run NOW registers file_access_write on SessionContext
  AR->>AR: model reads error, re-issues file_access_write
  AR-->>C: another approval card for the same action (loop)

Loading

2. Symptoms

• The user approves a provider-injected tool call in the AG-UI client.
• The tool does not execute: no side effect (no file written, no memory entry, no todo mutation).
• The chat transcript shows a tool result of "Error: Tool call invocation failed." for that call_id.
• The agent retries the identical call, producing a second approval card; the interaction loops until the model gives up or the user cancels.
• A client that optimistically renders the action from the call arguments (e.g. a "Saved" card drawn from file_access_write's args before the result arrives) will show a success affordance that is not backed by a real write — the optimistic UI directly contradicts the failed tool result.

Provider-agnostic: reproduced on Ollama (glm-5.2:cloud); the failure is entirely server-side in the transport and independent of the model provider.

3. Evidence — live capture (2026-07-09)

Harness advisor agent with FileAccessProvider (registers file_access_write with approval_mode="always_require" in before_run), hosted via the AG-UI endpoint, CopilotKit frontend. User asked the agent to save approval-fix.md; the model emitted file_access_write; the user clicked Approve. From aspire otel logs ats-web (Aspire DEBUG telemetry, ats.* bridged to OTel):

Log id	Source	Message (abridged)
75	agent_framework	Checking function call: type=function_call, name=file_access_write, in approval_tools=True
76	agent_framework	Approval needed for function: file_access_write
79	agent	conditional_auto_approval_rules: function_call=file_access_write (user approval arrived)
82	web_approvals_continuity	suppressed synthetic results for ['file_access_write::0::7e9f7b66']
84	web_approvals_continuity	recovered 1 provider-tool approval(s) the AG-UI transport could not execute (deferred to harness in-run)
85	web_approvals_continuity	function middleware saw tool='file_access_write' … result='File written.' (this is our app-side recovery, not the transport)
99	agent_framework	Function file_access_write succeeded.
105	agent_framework	tool result: "File 'approval-fix.md' written."
108	web_history	file-store[file_access]: write ok path='approval-fix.md' bytes=23

Rows 84–108 are the app-side workaround kicking in (see §6). The transport's own behavior is rows 75–82 plus, in the pre-fix repro, the diagnostic pair below. From an earlier unmitigated run on the same flow (before the workaround):

Log id	Message
38, 39	two identical tool_call_response for the same call_id with "Error: Tool call invocation failed." (transport executed nothing, injected the error, and the snapshot re-sent it)
47	assistant: "Let me retry that." + a new file_access_write tool_call
50	Approval needed for function: file_access_write (second approval card)

The tell in framework DEBUG: _try_execute_function_calls runs with a tool_map whose keys are exactly the static server tools and do not include file_access_write, immediately followed by "Error: Tool call invocation failed.".

3a. Second capture — load_skill (2026-07-10, agent-framework @ 9a5312b)

Same flow with the harness skills provider: the model called load_skill {"skill_name": "landing-zone-review"} (approval-gated) alongside two static tools; the user approved all three. Telemetry chronology:

1. Transport pre-execution failed for the provider-injected tool and seated the placeholder: TOOL_CALL_RESULT toolCallId=load_skill::0::3c0ff140 content='Error: Tool call invocation failed.'
2. Our shim deferred it: WARNING … recovered 1 provider-tool approval(s) the AG-UI transport could not execute (deferred to harness in-run).
3. The deferred in-run execution succeeded: INFO agent_framework._skills: Loading skill: landing-zone-review → Function load_skill succeeded. → shim captured the real result.
4. Residual artifact even with the workaround: the run-boundary TOOL_CALL_RESULT event emitted to the client still carried the transport's step-1 error placeholder, so the UI shows a red Error card — and the client-held history round-trips the error string — for a call that actually executed. The two static tools in the same turn show Done with real results.

This confirms the defect is alive on current main and demonstrates the placeholder is not just a transcript artifact: it is what the transport emits to the AG-UI client for the affected call. Tracked app-side as sokolaidev/ats-maf#56.

4. Reproduction (minimal, stock components)

1. Build a harness agent with a ContextProvider that registers a tool in before_run and marks it approval-required. The stock FileAccessProvider is the canonical case (_harness/_file_access.py: before_run registers file_access_write with approval_mode="always_require").
2. Host it with add_agent_framework_fastapi_endpoint(app, agent, …) (tool approval enabled).
3. Send a prompt that makes the agent call file_access_write.
4. Approve the call in the AG-UI client.
5. Observe: the tool result is "Error: Tool call invocation failed.", no file is written, and the agent re-issues the call (second approval).

No custom middleware required — default FileAccessProvider over the stock endpoint. Any provider-injected, approval-gated tool exhibits the same failure.

Code Sample

Error Messages / Stack Traces

Package Versions

agent-framework-ag-ui: 1.0.0rc7, agent-framework-core: 1.10.0

Python Version

Python 3.12

Additional Context

5. Root cause

run_agent_stream builds the execution tool map from collect_server_tools(agent) and resolves approvals before agent.run:

• _agent_run.py:1341 — server_tools = collect_server_tools(agent) (static server tools only).
• _agent_run.py:1381-1384 — tools_for_execution = tools if tools is not None else server_tools; _resolve_approval_responses(messages, tools_for_execution, …) runs here, ahead of agent.run.
• _agent_run.py:926-933 — _try_execute_function_calls(function_calls=approved_responses, tools=tools, …); tools is tools_for_execution, which lacks the provider-injected tool, so the call yields no result.
• _agent_run.py:941-953 — for any approved response without a matching function_result, the fallback injects Content.from_function_result(call_id=call_id, result="Error: Tool call invocation failed.").

The provider-injected tools are added to the SessionContext by ContextProvider.before_run / FileAccessProvider.before_run, which runs inside agent.run — after the transport has already tried (and failed) to execute the approval. The transport and the harness disagree on when the tool set is complete: the transport assumes the tool set is knowable before the run; the harness providers deliberately defer tool registration to before_run so they can be per-invocation and session-aware. The approval path straddles that seam and falls through it.

Note the asymmetry: a provider-injected tool call that does not require approval is fine — it is executed later by the harness chat client's _process_function_requests (_tools.py) with the full per-invocation tool set, after before_run has run. Only the approval path is broken, because the transport short-circuits approval resolution ahead of agent.run.

6. Application-level workaround (what we do today)

We recover the failed approval in our AG-UI continuity shim (ats.web_approvals_continuity, AGUIContinuityAgentFrameworkAgent._run_with_continuity):

1. Before the run's incoming-result scan, detect approved provider-injected tool calls whose result is the transport's "Error: Tool call invocation failed." marker and whose name is not in collect_server_tools(agent) (i.e. provider-injected).
2. Drop that error tool_call_response from messages (it is a transport artifact, not a real tool result).
3. Reconstruct a Content.from_function_approval_response(approved=True, id=call_id, function_call=<the still-present function_call>) and push it into session.state["tool_approval"]["collected_approval_responses"] (dedup by call_id).
4. Skip that call_id in the shim's gated-call backfill so it is not double-handled.
5. Let agent.run proceed. before_run now registers the tool, and the harness ToolApprovalMiddleware._inject_collected_responses injects the queued approval response, so the chat client executes the call with the full per-invocation tool set — the approved action actually runs.

This reuses the shim's existing deferred-approval mechanism; the only addition is recognizing the transport's failure marker for provider-injected tools and converting it back into a queued approval. Verified on disk: approval-fix.md is written with the approved content, and the retry loop is gone (no second approval card).

7. Proposed upstream fix

Any of the following, in our order of preference:

1. Run before_run (or otherwise gather provider-injected tools) before approval resolution. Hoist the per-invocation tool set — including tools that harness ContextProviders register on the SessionContext — ahead of _resolve_approval_responses, so the execution tool map is complete when an approved call is executed. This is the direct fix: the transport and the harness then agree on the tool set at approval time.
2. Defer provider-tool approvals to the harness ToolApprovalMiddleware. If the transport cannot know the provider-injected tool set ahead of agent.run, leave the approval resolution to the harness in-run path (which already executes approval responses via _process_function_requests with the full tool set) instead of short-circuiting it. The transport would only resolve approvals for tools it can actually execute (the static server/client set).
3. At minimum, distinguish "tool not found" from a real execution failure. _try_execute_function_calls returning no result for a call_id should not be reported to the model as "Error: Tool call invocation failed." indistinguishable from a tool that threw. A tool_not_found/deferred signal (or simply leaving the approval content in place for the harness to handle in-run) would let the existing harness approval path complete the action and avoid the retry loop. Today the failure is silent and looks like the tool itself is broken.

8. Related issues

Issue / PR	State	Relationship
6910	open (this repo)	approval state lost across fresh per-request sessions; this report is the tool-set timing gap in the same approval path. Independent — fixing 6910 does not fix this, and vice versa
6920	open (this repo)	harness provider state buckets lost across requests; this report is about harness provider tools missing from the transport's pre-run tool map. Same provider family, different seam
#6471	merged	Python AG-UI thread snapshot persistence — the snapshot seam our workaround leverages; does not address pre-run tool-set completeness
#3199 / PR #3212	closed / merged 2026-01-15	Direct prior art — the MCP variant of this exact defect. "MCP Tools with approval_mode set to Always_requires are not executed when using AG-UI": approved MCP tool calls failed with the same "Tool call invocation failed" because collect_server_tools() didn't see them. Fixed by adding ChatAgent.mcp_tools to collect_server_tools(). ContextProvider-injected tools cannot be fixed the same way statically (they only exist per-invocation after before_run), which is why §7 option 1 or 2 is needed

(Upstream duplicate search performed 2026-07-10 — queries over issues/PRs for collect_server_tools/approval, before_run tool injection, provider-tool approval, and the error string: no Python issue or PR covers the ContextProvider/before_run variant; #3199/#3212 is the nearest and covers MCP tools only.)

9. Environment

• agent-framework-core 1.10.0, agent-framework-ag-ui 1.0.0rc7, Python 3.12, Windows 11
• Frontend: CopilotKit over the FastAPI AG-UI endpoint; evidence captured via .NET Aspire telemetry with DEBUG logging (ats.* bridged to OTel)
• Provider: Ollama (glm-5.2:cloud); failure is provider-agnostic

[github-actions]

🤖 Automated triage reproduction notes (agent-authored — trust but verify)

Agent analysis

Repro: _resolve_approval_responses in packages/ag-ui/agent_framework_ag_ui/_agent_run.py (called at line 1857) receives tools_for_execution built from collect_server_tools (line 1814/1855) which only includes static default_options tools and MCP tools — provider-injected tools registered in before_run are absent. When an approved function_approval_response names a provider-injected tool (e.g. file_access_write), _try_execute_function_calls (core/_tools.py:1633) cannot find it in tool_map, returns no result for that call, and the fallback at _agent_run.py:1343 injects the error placeholder. Minimal repro: call _resolve_approval_responses with a tool list that excludes the approved tool's name and observe the error result.

• Failing test: python/packages/ag-ui/tests/ag_ui/test_provider_tool_approval_bug.py
• Files examined: python/packages/ag-ui/agent_framework_ag_ui/_agent_run.py, python/packages/ag-ui/agent_framework_ag_ui/_orchestration/_tooling.py, python/packages/core/agent_framework/_tools.py, python/packages/ag-ui/tests/ag_ui/conftest.py, python/packages/ag-ui/tests/ag_ui/test_approval_result_event.py
• Tests run: tests/ag_ui/test_provider_tool_approval_bug.py::TestProviderToolApprovalBug::test_resolve_approval_for_provider_tool_produces_error, tests/ag_ui/test_provider_tool_approval_bug.py::TestProviderToolApprovalBug::test_resolve_approval_works_when_tool_is_available, tests/ag_ui/test_provider_tool_approval_bug.py::TestProviderToolApprovalBug::test_full_stream_provider_tool_approval_produces_error_in_messages
• Reported version: agent-framework-ag-ui 1.0.0rc7, agent-framework-core 1.10.0
• Current version: agent-framework-ag-ui 1.0.0rc8, agent-framework-core 1.11.0