fix(core): press CUA keypress combinations as a single chord#2266
Open
yawbtng wants to merge 1 commit into
Open
fix(core): press CUA keypress combinations as a single chord#2266yawbtng wants to merge 1 commit into
yawbtng wants to merge 1 commit into
Conversation
CUA keypress actions describe a single key chord, but the executor pressed each key in the array separately, releasing modifiers before the main key. Combinations like ["Control", "A"] sent Ctrl alone and then typed a literal "a" instead of select-all. This broke the OpenAI, Google (key_combination), and Microsoft computer-use clients, which emit multi-element key arrays; Anthropic (single +-joined string) was fine. Join the mapped keys into one +-delimited combination so page.keyPress holds modifiers down for the main key, and record the chord as a single replay step. mapKeyToPlaywright is idempotent, so already-combined and single-key inputs are unchanged. Adds unit coverage for chord, alias normalization, single-key, already-combined, and empty-array cases.
🦋 Changeset detectedLatest commit: 4f921ee The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
|
This PR is from an external contributor and must be approved by a stagehand team member with write access before CI can run. |
Contributor
There was a problem hiding this comment.
No issues found across 3 files
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Architecture diagram
sequenceDiagram
participant Provider as CUA Provider
participant Handler as V3CuaAgentHandler
participant ActionMapper as mapKeyToPlaywright
participant Browser as Playwright Page
Note over Provider,Browser: CUA Keypress Action Flow
Provider->>Handler: executeAction({ type: "keypress", keys })
alt keys is array (OpenAI, Google, Microsoft shape)
Handler->>Handler: keyList = keys (already array)
else keys is string (Anthropic shape)
Handler->>Handler: keyList = [keys]
end
alt keyList.length > 0
Handler->>ActionMapper: map each element via mapKeyToPlaywright()
ActionMapper-->>Handler: normalized key strings
Handler->>Handler: Join mapped keys with "+"
Note over Handler: e.g. ["Control", "A"] → "Control+A"
Handler->>Browser: page.keyPress("Control+A")
Note over Browser: page.keyPress holds modifiers<br/>down for final key
Browser-->>Handler: keypress complete
alt recording enabled
Handler->>Handler: recordCuaActStep("press Control+A")
end
else keyList.length === 0
Handler->>Handler: Skip (no keyPress call)
end
Handler-->>Provider: { success: true }
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
why
CUA
keypressactions describe a single key chord (modifiers held down while the main key is pressed), butV3CuaAgentHandler.executeActionpressed each key in the array separately.page.keyPress(modifier)presses and releases the modifier, so by the time the main key was pressed the modifier was already up.The concrete failure: a
["Control", "A"]keypress sendsControlon its own (a no-op) and thenAthrough the plain typing path — so instead of select-all, the agent types a literalainto the focused field. Any select-all / copy / paste / cut / shortcut pattern silently fails and corrupts input. Because the agent-replay cache recorded the broken per-key sequence, replays reproduced the bug too.This is provider-dependent, based on the shape each client emits:
keys: ["CTRL", "A"]Ctrlthen literalakey_combination).split("+")→["Control", "A"]Ctrlthen literalafara-7b)keys: string[](per-key)Ctrlthen literalakeys: ["ctrl+s"](single+-joined string)Anthropic only worked by accident — it pre-joins with
+, whichpage.keyPressalready chords internally.what changed
packages/core/lib/v3/handlers/v3CuaAgentHandler.ts— in thekeypresscase, map each key and join into one+-delimited combination, then callpage.keyPressonce.page.keyPressalready holds modifiers down for the final key and already special-cases the literal+key, so single keys, already-combined strings, andCtrl++-style inputs all stay correct.mapKeyToPlaywrightis idempotent (CTRL/Control→Control), so Google's pre-mapped arrays and Anthropic's combined string are unchanged. The recorded replay step is now a singlepress Control+Ainstead of the brokenpress Control, press A.test plan
New
packages/core/tests/unit/cua-keypress-chord.test.ts(5 cases, all passing):["Control", "A"]→ singlekeyPress("Control+A")["CTRL", "A"]→keyPress("Control+A")["Enter"]→keyPress("Enter")(unchanged)["ctrl+s"]→keyPress("ctrl+s")(Anthropic shape, unchanged)[]→ nokeyPresscallExisting CUA suites (
anthropic-cua-triple-click,openai-cua-client,microsoft-cua-client,anthropic-cua-adaptive-thinking) — 25 tests still green.Related: this is exactly the class of provider-specific CUA regression that #2188 proposes catching with a deterministic bench task.
Summary by cubic
Fixes CUA keypress combos by pressing them as one chord, not as separate keys. Shortcuts like Ctrl+A now work across OpenAI, Google
key_combination, and Microsoft clients instead of typing letters.page.keyPresscall; supports arrays, already-joined strings, and the literal "+" key.Written for commit 4f921ee. Summary will update on new commits.