docs: document Studio's WebMCP agent tools, proven end-to-end in a browser - #3521
Draft
miguel-heygen wants to merge 4 commits into
Draft
docs: document Studio's WebMCP agent tools, proven end-to-end in a browser#3521miguel-heygen wants to merge 4 commits into
miguel-heygen wants to merge 4 commits into
Conversation
miguel-heygen
marked this pull request as draft
August 27, 2026 01:32
Adds `guides/webmcp`, under Developers > Agent setup. Its first job is to defuse a name collision. `guides/mcp` already exists and covers HeyGen's HOSTED MCP connector, which builds a video from a chat. This page is about an agent working inside Studio on a composition already open in front of you. Different feature, confusingly similar name, so the page says what it is not before it says what it is. Written to DOCS_GUIDELINES: one-sentence intro, outcome before implementation, real values rather than placeholders, and three callouts. The three things a reader most needs are the ones easiest to get wrong: The API is `document.modelContext`, not `navigator.modelContext`. Most published examples use the second, which is a polyfill compatibility shim rather than a spec member, so feature-detecting it misleads. Select first, then edit. Most editing tools act on the current selection, and an agent that skips it gets an error rather than a wrong-element write. Leave Studio visible. Some of Studio's write paths report failure through a toast rather than a return value, so the human is the one who sees it. That is a real property of the co-pilot design, not a nicety, so the page says it plainly. Verified with `npx mint validate` and `npx mint broken-links --check-redirects`, both passing.
Found by running the tools end to end in a browser, which is the only way it
could have been found: the unit tests mock `setText`, so they never crossed the
boundary where this breaks.
An element's text usually lives in a CHILD field, keyed like `self:0:h1` or
`child:0:h1`. `studio_set_text` passed no field key, so
`buildNextDomTextFields` planned zero operations, the request went out with an
empty patch, and the server answered:
POST /api/projects/<id>/file-mutations/patch-element
-> 400 {"error":"target and operations required"}
Which surfaced as `persist-failed`. The tool was telling the truth, so the
reporting work in the earlier PRs did its job, but the failure looked like a
server problem and was not.
The tool now resolves the field: the one the caller named, or the element's
single field when it has exactly one. An element with several fields is asked
to name one; an element with none is reported blocked. Naming a field the
element does not have is rejected with the list of the ones it does have,
rather than silently writing nowhere.
Four regression tests, including the exact `child:0:h1` shape that failed. One
existing assertion changed: it expected the field to be `undefined`, which is
precisely the bug, so it now expects the resolved key.
Also documents two things the browser run surfaced, both real and neither a
defect: registration is asynchronous, so a caller reading `getTools()` too
early sees a partial list; and the tools that act on the current selection need
a render between the select and the edit, which a real agent gets for free
because its calls arrive as separate messages.
The page told readers to set agentToolsEnabled in Studio's preferences. Nothing writes that flag: it is read in useStudioAgentTools and parsed in studioUiPreferences, but there is no settings UI and no toggle, so the instruction could not be followed. Replace it with the localStorage write that actually flips it, and spell out the merge, since overwriting the key drops every other stored preference.
The page said the browser asks before any agent calls a tool. Prompt granularity is browser-specific and unsettled during the origin trial, and we have not observed it on the native path. Say what holds, that access is gated, and name the part that is still moving.
miguel-heygen
force-pushed
the
feat/studio-webmcp-animation
branch
from
August 27, 2026 04:49
721c271 to
90cef4f
Compare
miguel-heygen
force-pushed
the
feat/studio-webmcp-docs
branch
from
August 27, 2026 04:49
7498855 to
8a6342b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The
webmcpdocs page, an end-to-end proof that the tool set works in a real browser, and the one bug that proof found.Stacked on #3520. Last unit of ten.
End-to-end proof
Driven with Claude in Chrome against Studio's dev server on
127.0.0.1:5190, using a minimal GSAP fixture composition. Not a mock anywhere.1. All twelve tools register
2.
studio_lookreturns real Studio state{ "ok": true, "project": "webmcp-proof", "playhead": 0, "duration": 4, "canWrite": true, "elementCount": 1, "elements": [{ "handle": "hf:hf-10wy", "label": "Ship it", "tag": "div", "start": 0, "duration": 4, "track": 0 }], "selection": null }3. Select, inspect, write
Studio's own URL state moved with it, so this is Studio's real selection, not a shadow copy:
4. The file on disk changed
5. And it renders
studio_frame's URL, against the real Puppeteer renderer, before and after:Both PNGs were opened and read. The frame shows the agent's edit.
The bug the proof found
The first run failed, and failed honestly:
{ "ok": false, "kind": "failed", "reason": "the text was not applied: persist-failed", "hint": "The write did not reach the file. Check Studio." }The file was unchanged, checksum identical. So the reporting work in #3510 and #3518 did exactly its job: before that, this would have returned success.
Intercepting
fetchgave the cause:An element's text lives in a child field, keyed
self:0:h1here.studio_set_textpassed no field key, sobuildNextDomTextFieldsplanned zero operations and the patch went out empty.The unit tests could not have caught this, because they mock
setTextand never cross that boundary. This is the class of failure only a browser run finds, which is why the plan insisted on this unit.Fixed: the tool resolves the field, using the one named or the element's single field. Several fields, it asks which. No fields, it reports blocked. A named field that does not exist is rejected with the list of ones that do. Four regression tests, including the exact failing shape. One existing assertion expected the key to be
undefined, which was the bug, so it now expects the resolved key.Two real-world behaviours, both documented
Registration is asynchronous. A caller reading
getTools()the instant Studio loads sees a partial list. Observed at 10 of 12 mid-registration. The docs page says to wait fortoolchangeor poll.Tools on the current selection need a render between select and edit. Firing select-then-edit back to back from one script returns
"nothing is selected", becauseapplyDomSelectionschedules a React state update. A real agent gets this for free, since its calls arrive as separate messages. This is the live confirmation of the select-first design rather than a defect.The docs page
guides/webmcp, under Developers → Agent setup.Its first job is defusing a name collision:
guides/mcpalready covers HeyGen's hosted MCP connector, which builds a video from a chat. This page is an agent working inside Studio. The page says what it is not before what it is.Also flags the two things easiest to get wrong: the API is
document.modelContextnotnavigator.modelContext, and Studio should stay visible because some write paths report failure through a toast the human sees rather than a return value.Test plan
Unit tests added/updated
Manual testing performed
Documentation updated
Full end-to-end run above, in Chrome, against a live Studio.
npx mint validateandnpx mint broken-links --check-redirectsboth pass.Package suite 4606 passing across 414 files.
bunx tsc --noEmit,bunx oxlint,bunx fallow audit --fail-on-issuesall clean.Still not covered
Native Chrome behind
chrome://flags/#enable-webmcp-testinguses a different code path from the polyfill, and has not been tried. Neither has a bridge extension calling from the extension context. The polyfill path is now proven end to end; those two are not.