Repository navigation
Ship StoryboardHarness — built-in compliance test harness for adopters via httpx.ASGITransport #662
Description
Activity
Triage
Classification: Feature request
Bucket(s): handlers, testing
Status: drafting-pr
Milestone: omitted — no target version in issue textWhat the experts said:
- dx-expert: place in
adcp.testing(not a newadcp.server.testingmodule); name itSellerTestClientrather thanStoryboardHarnessto avoid implying scenario bundles that don't exist yet; shipinvoke()only — norun_scenario()stub; typedAdcpErrorPayloaddataclass return;from_platform()classmethod for DecisioningPlatform adopters - code-reviewer: safe to execute with hard constraints — (a)
adcp/testing/only, (b) norun_scenario()anywhere, (c) MCP-only for v1 (transport="a2a"excluded — A2A has a separate ASGI app), (d) use in-processmcp.call_tool()to avoid SSE frame parsing, (e)passedis a computed property not a stored field, (f)adcp_errorfield is a dataclass/TypedDict not the exception class
My take: The core pain is real —
build_test_clienthandles ASGI wiring but each adopter still has to write the JSON-RPC envelope construction +structuredContent["adcp_error"]extraction boilerplate by hand.SellerTestClient.invoke()eliminates exactly that. Therun_scenario()half of the issue is blocked: there's no bundled scenario playbook cache in the Python SDK (the storyboard runner referenced in the issue is the external AAO compliance tool). Shippinginvoke()now + noting therun_scenario()dependency in source is the right split.Drafting PR for
SellerTestClient.invoke()(MCP transport, inadcp.testing).run_scenario()and A2A transport deferred — this PR usesRefs #662, notCloses. A status comment will list what shipped vs. what remains.
Triaged by Claude Code. Session: https://claude.ai/code/session_01VZRaadSyoHDm7CU945e3Ba
Generated by Claude Code
- dx-expert: place in
Triage status update — partial rollout via #666
Shipped in #666:
SellerTestClient— in-process MCP harness; wrapsDecisioningPlatform, callsmcp.call_tool(), returnsToolInvokeResultwith.passed,.data,.adcp_error,.rawToolInvokeResult— typed result containerAdcpErrorPayload— typedadcp_errorenvelope fields (code,message,recovery,field,suggestion,retry_after,details)- All three symbols exported from
adcp.testing - 18 new tests covering success path, error extraction, lazy init, instance reuse, and public import path
Not yet shipped (queued for follow-up):
run_scenario()— blocked on bundled compliance scenario playbooks; the Python SDK currently has schema files inschemas/cache/3.0/but no scenario playbooks, and noadcp storyboard run --jsonCLI equivalent. Tracked as Ship StoryboardHarness — built-in compliance test harness for adopters via httpx.ASGITransport #662.- A2A transport support — A2A is served via a separate Starlette app (
create_a2a_server()), not the same ASGI app. An in-process A2A test client needs its own primitive. Tracked as Ship StoryboardHarness — built-in compliance test harness for adopters via httpx.ASGITransport #662.
The class docstring in
harness.pycalls both out explicitly so adopters who read source aren't surprised.
Generated by Claude Code
- added a commit that references this issue
on May 12, 2026
Summary
Every adopter of `adcp-client-python` who wants to run the compliance storyboard against their own seller in CI has to invent the same harness:
This is reinvented in every salesagent test file we've added today (see PR #330's `tests/integration/test_delegate_wire_envelope_cross_transport.py`). It's tedious and prone to drift across adopters.
Proposed surface
```python
from adcp.server.testing import StoryboardHarness
@pytest.fixture
def harness():
return StoryboardHarness(app=build_app(), seller_config=...)
def test_create_media_buy_compliance(harness):
result = harness.run_scenario(
"media_buy_seller/refine_products",
transport="mcp", # or "a2a", or "both"
)
assert result.passed, result.failures
def test_typed_error_translates(harness):
response = harness.invoke(
tool="update_media_buy",
payload={"media_buy_id": "does-not-exist", ...},
transport="a2a",
)
assert response.adcp_error.code == "MEDIA_BUY_NOT_FOUND"
```
The harness:
Why this matters
Reference implementation
PR #330 in bokelley/salesagent (`tests/integration/test_delegate_wire_envelope_cross_transport.py`) is a working version for a single tool. Extending to scenario-level execution is the obvious next step.
Related