Skip to content

docs: document Studio's WebMCP agent tools, proven end-to-end in a browser - #3521

Draft
miguel-heygen wants to merge 4 commits into
feat/studio-webmcp-animationfrom
feat/studio-webmcp-docs
Draft

docs: document Studio's WebMCP agent tools, proven end-to-end in a browser#3521
miguel-heygen wants to merge 4 commits into
feat/studio-webmcp-animationfrom
feat/studio-webmcp-docs

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

What

The webmcp docs page, an end-to-end proof that the tool set works in a real browser, and the one bug that proof found.

Stacked on #3520. Last unit of ten.

End-to-end proof

Driven with Claude in Chrome against Studio's dev server on 127.0.0.1:5190, using a minimal GSAP fixture composition. Not a mock anywhere.

1. All twelve tools register

> const tools = await document.modelContext.getTools()
> tools.map(t => t.name).sort()

["studio_add_animation", "studio_add_keyframe", "studio_delete_animation",
 "studio_frame", "studio_inspect", "studio_look", "studio_seek",
 "studio_select", "studio_set_style", "studio_set_text",
 "studio_transform", "studio_update_animation"]

2. studio_look returns real Studio state

{ "ok": true, "project": "webmcp-proof", "playhead": 0, "duration": 4,
  "canWrite": true, "elementCount": 1,
  "elements": [{ "handle": "hf:hf-10wy", "label": "Ship it", "tag": "div",
                 "start": 0, "duration": 4, "track": 0 }],
  "selection": null }

3. Select, inspect, write

selected  { "ok": true, "label": "Headline" }
inspect   { "text": "Ship it", "canEditText": true, "fieldKey": "self:0:h1" }
wrote     { "ok": true, "text": "Shipped it", "changed": true }

Studio's own URL state moved with it, so this is Studio's real selection, not a shadow copy:

#project/webmcp-proof?...&selFile=index.html&selId=headline&selSelector=%23headline

4. The file on disk changed

md5 before  b0d086373b574f887b93b52fcd8d2066
md5 after   d37d443d880eda060792386ebf4cd195

-  <h1 id="headline" data-hf-id="headline">Ship it</h1>
+  <h1 id="headline" data-hf-id="headline">Shipped it</h1>

5. And it renders

studio_frame's URL, against the real Puppeteer renderer, before and after:

t=0.500  ->  HTTP 200  image/png  14,424 bytes  1920x1080   "Ship it"
t=1.500  ->  HTTP 200  image/png  17,680 bytes  1920x1080   "Shipped it"

Both PNGs were opened and read. The frame shows the agent's edit.

The bug the proof found

The first run failed, and failed honestly:

{ "ok": false, "kind": "failed",
  "reason": "the text was not applied: persist-failed",
  "hint": "The write did not reach the file. Check Studio." }

The file was unchanged, checksum identical. So the reporting work in #3510 and #3518 did exactly its job: before that, this would have returned success.

Intercepting fetch gave the cause:

POST .../file-mutations/patch-element  ->  400 {"error":"target and operations required"}

An element's text lives in a child field, keyed self:0:h1 here. studio_set_text passed no field key, so buildNextDomTextFields planned zero operations and the patch went out empty.

The unit tests could not have caught this, because they mock setText and never cross that boundary. This is the class of failure only a browser run finds, which is why the plan insisted on this unit.

Fixed: the tool resolves the field, using the one named or the element's single field. Several fields, it asks which. No fields, it reports blocked. A named field that does not exist is rejected with the list of ones that do. Four regression tests, including the exact failing shape. One existing assertion expected the key to be undefined, which was the bug, so it now expects the resolved key.

Two real-world behaviours, both documented

Registration is asynchronous. A caller reading getTools() the instant Studio loads sees a partial list. Observed at 10 of 12 mid-registration. The docs page says to wait for toolchange or poll.

Tools on the current selection need a render between select and edit. Firing select-then-edit back to back from one script returns "nothing is selected", because applyDomSelection schedules a React state update. A real agent gets this for free, since its calls arrive as separate messages. This is the live confirmation of the select-first design rather than a defect.

The docs page

guides/webmcp, under Developers → Agent setup.

Its first job is defusing a name collision: guides/mcp already covers HeyGen's hosted MCP connector, which builds a video from a chat. This page is an agent working inside Studio. The page says what it is not before what it is.

Also flags the two things easiest to get wrong: the API is document.modelContext not navigator.modelContext, and Studio should stay visible because some write paths report failure through a toast the human sees rather than a return value.

Test plan

  • Unit tests added/updated

  • Manual testing performed

  • Documentation updated

  • Full end-to-end run above, in Chrome, against a live Studio.

  • npx mint validate and npx mint broken-links --check-redirects both pass.

  • Package suite 4606 passing across 414 files.

  • bunx tsc --noEmit, bunx oxlint, bunx fallow audit --fail-on-issues all clean.

Still not covered

Native Chrome behind chrome://flags/#enable-webmcp-testing uses a different code path from the polyfill, and has not been tried. Neither has a bridge extension calling from the extension context. The polyfill path is now proven end to end; those two are not.

@miguel-heygen miguel-heygen changed the title docs: document Studio's WebMCP agent tools, and verify registration in a real browser docs: document Studio's WebMCP agent tools, proven end-to-end in a browser Aug 27, 2026
@miguel-heygen
miguel-heygen marked this pull request as draft August 27, 2026 01:32
Adds `guides/webmcp`, under Developers > Agent setup.

Its first job is to defuse a name collision. `guides/mcp` already exists and
covers HeyGen's HOSTED MCP connector, which builds a video from a chat. This
page is about an agent working inside Studio on a composition already open in
front of you. Different feature, confusingly similar name, so the page says
what it is not before it says what it is.

Written to DOCS_GUIDELINES: one-sentence intro, outcome before implementation,
real values rather than placeholders, and three callouts.

The three things a reader most needs are the ones easiest to get wrong:

The API is `document.modelContext`, not `navigator.modelContext`. Most
published examples use the second, which is a polyfill compatibility shim
rather than a spec member, so feature-detecting it misleads.

Select first, then edit. Most editing tools act on the current selection, and
an agent that skips it gets an error rather than a wrong-element write.

Leave Studio visible. Some of Studio's write paths report failure through a
toast rather than a return value, so the human is the one who sees it. That is
a real property of the co-pilot design, not a nicety, so the page says it
plainly.

Verified with `npx mint validate` and `npx mint broken-links --check-redirects`,
both passing.
Found by running the tools end to end in a browser, which is the only way it
could have been found: the unit tests mock `setText`, so they never crossed the
boundary where this breaks.

An element's text usually lives in a CHILD field, keyed like `self:0:h1` or
`child:0:h1`. `studio_set_text` passed no field key, so
`buildNextDomTextFields` planned zero operations, the request went out with an
empty patch, and the server answered:

  POST /api/projects/<id>/file-mutations/patch-element
  -> 400 {"error":"target and operations required"}

Which surfaced as `persist-failed`. The tool was telling the truth, so the
reporting work in the earlier PRs did its job, but the failure looked like a
server problem and was not.

The tool now resolves the field: the one the caller named, or the element's
single field when it has exactly one. An element with several fields is asked
to name one; an element with none is reported blocked. Naming a field the
element does not have is rejected with the list of the ones it does have,
rather than silently writing nowhere.

Four regression tests, including the exact `child:0:h1` shape that failed. One
existing assertion changed: it expected the field to be `undefined`, which is
precisely the bug, so it now expects the resolved key.

Also documents two things the browser run surfaced, both real and neither a
defect: registration is asynchronous, so a caller reading `getTools()` too
early sees a partial list; and the tools that act on the current selection need
a render between the select and the edit, which a real agent gets for free
because its calls arrive as separate messages.
The page told readers to set agentToolsEnabled in Studio's preferences.
Nothing writes that flag: it is read in useStudioAgentTools and parsed in
studioUiPreferences, but there is no settings UI and no toggle, so the
instruction could not be followed. Replace it with the localStorage write
that actually flips it, and spell out the merge, since overwriting the key
drops every other stored preference.
The page said the browser asks before any agent calls a tool. Prompt
granularity is browser-specific and unsettled during the origin trial, and
we have not observed it on the native path. Say what holds, that access is
gated, and name the part that is still moving.
@miguel-heygen
miguel-heygen force-pushed the feat/studio-webmcp-animation branch from 721c271 to 90cef4f Compare August 27, 2026 04:49
@miguel-heygen
miguel-heygen force-pushed the feat/studio-webmcp-docs branch from 7498855 to 8a6342b Compare August 27, 2026 04:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant