Validate desktop tool use in Ace and track known gaps #427

Activity

1. iamnbutler

   added 6 commits that reference this issue on Oct 15, 2026

   fix: say why a key has no receiver and point to the menu (#14)

   ...

   25851fd

   fix: name the unmatched menu title and its siblings (#15)

   ...

   034a695

   fix(channel): pin desktop-tools with clearer menu and key refusals (#441

   ...

   8fbfcc3

   fix: refuse file URLs whose query or fragment would be dropped (#20)

   ...

   380e665

   fix: re-read a restored window until the Dock animation ends (#22)

   ...

   4618b33

   fix: inspect windows whose title changes during the read (#26)

   ...

   73c0513
2. iamnbutler commented on Oct 15, 2026

   

   iamnbutler

   on Oct 15, 2026

   ContributorAuthor

   More actions

   Desktop tools study, single player (2026-10-15)

   Setup. One Mac, Ace Canary, model anthropic/claude-opus-5-5, channels dt and dt2 in a sandbox project.

   • A Codex computer-use tester drove Ace's UI; the lead watched from outside and scripted disruptions.
   • Builds: 0.0.46.50 → .51 (fix(channel): pin desktop-tools with clearer menu and key refusals #441) → .52 (fix(channel): pin desktop-tools with file URL, disabled menu and restore fixes #444, desktop-tools 4618b33).
   • Scenarios: D0–D9 plus re-checks R1–R3.

   Validation results

   Check	Result	Build
   Tools off refuses every desktop_* call; on again without restart	Pass (D2)	.50
   Teammate invocation with sharing on and off	Not tested: single player	–
   Stop, or a lost worker, mid-action settles and isn't replayed	Pass (D3b Stop halted typing at once; D3c kill -9 of the worker mid-insert → unknown, the agent inspected and didn't retype)	.50
   Two channels don't act on each other's targets	Pass for safety (D4, R1: no cross-target action). Usability gap: agents can't tell whose new window is whose (desktop-tools#17)	.51, .52
   Patch tests pass at the pin	Pass at 8801b31: Bridge 115/115 and 55/55, AutomationKit 2033/2033 swift-testing, 296/297 XCTest (the known upstream failure). 4618b33 adds only the restore patch, and its bridge tests pass	–

   Also passed:

   • D5: Open panel, PDF export and menus.
   • D6: Safari form fields exact, the user's clipboard preserved.
   • D8: clipboard text, image and file references.
   • D9: target closed mid-task (refused before dispatch, the agent asked).
   • D9b: target minimized mid-task (refused, restored, finished with no gaps or duplicates).

   Shipped during the study

   • desktop-tools:
     • Make directory changes appear promptly #14: key and insert refusal hints.
     • Verify repository-based project identity in the native app and across two hosts #15: a missing menu title lists its siblings and the ellipsis form.
     • Expose several chats in a channel's UI #19: insert re-reads a momentary window-ID refusal.
     • Expose Git and PR actions in the UI #20: refuse a file URL with a query or fragment instead of dropping it.
     • Let agents ask participants questions in the chat #21: a disabled-menu refusal suggests activating the app.
     • Add a File tab for browsing files #22: restore re-reads until the Dock animation ends.
     • Support non-image uploads in the composer #26: Accessibility inspect no longer fails when a window's title changes during the read (R1, Chrome with a ticking title). Not yet pinned in Ace or checked live.
   • Ace:
     • fix(channel): pin desktop-tools with clearer menu and key refusals #441 and fix(channel): pin desktop-tools with file URL, disabled menu and restore fixes #444: pin bumps, with tool description updates.
     • fix(host): close a client socket at its backpressure limit instead of dropping frames #446: gateway backpressure, the lost run-end behind Composer keeps showing Stop response after a run finishes #443.
     • fix(channel): caption only the calls in a step that took effect #447: captions stop counting refused calls (Timeline captions count refused tool calls as done #445).
     • fix(ui): make a leaving composer action inert before its exit animation ends #448: the Stop button stops lingering in the accessibility tree.

   Open findings

   • Security: the bridge serves any same-user process, including input (desktop-tools#24). A channel shell with desktop tools off, or any CLI, can click and type under Ace's grants. This extends Turning desktop tools off doesn't stop shell screen capture #379 from capture to input.
   • Concurrency: desktop-tools#17, attributing new windows between concurrent agents. Commented with a per-app lease proposal.
   • Focus after restore times out while another window of the app is key (desktop-tools#23).
   • A minimized target refuses with a raw AX error (desktop-tools#25).
   • Peekaboo tests and dev builds share per-user locks with Ace's runtime, which quarantines its capture or times out every operation (desktop-tools#16).
   • Escape in Finder is refused (desktop-tools#13).
   • Ace UI:
     • Screenshots are buried (Show desktop tool screenshots without digging through tool output #440) and Show when a run was stopped or interrupted, and tell the agent #442.
     • The context gauge reads as a loading spinner after a run, which is a design call.
     • fix(ui): make a leaving composer action inert before its exit animation ends #448 isn't in a Canary yet.
   • Chrome AX titles (desktop-tools#6); WebKit (desktop-tools#5); Ace's own Open Folder panel is refused by design (desktop-tools#4 comment).

   Stretch: Claude Code and Codex on desktop-tools

   Both work today through the MCP server in desktop-tools#12, pointed at Ace's client and socket.

   • Claude Code (Sonnet 5.5): a TextEdit task end to end in about 40 s.
   • Codex (gpt-6-astra): the same in 72 s, including Save and quit prompts.

   That only works because of #24. My recommendation:

   1. A standalone signed Desktop Tools app with its own grants and the MCP server bundled, for outside agents.
   2. Later, possibly Ace-hosted MCP that pairs a session with a channel.
   3. desktop-tools should own the tool definitions so Ace and MCP stop drifting.

   Details are on desktop-tools#12.

   Next steps

   1. Pin desktop-tools 73c0513 in Ace, re-run the patch tests suite by suite, and cut a Canary with fix(ui): make a leaving composer action inert before its exit animation ends #448.
   2. Re-check D7/R1 (Chrome with a ticking title) and D4 on that build.
   3. Run the sharing check with a second person.
   4. Decide on desktop-tools#24 and Add project lobbies for presence and channel lifecycle #17 before marking the concurrency check done.
3. iamnbutler commented on Oct 16, 2026

   

   iamnbutler

   on Oct 16, 2026

   ContributorAuthor

   More actions

   Re-checks on Canary 0.0.46.53 (68d9450, desktop-tools 73c0513):

   • Chrome with a ticking title: pass. fix: inspect windows whose title changes during the read desktop-tools#26 confirmed live: inspect, fill and Send with no retry, and the page log matched.
   • Stuck Stop button: pass. fix(ui): make a leaving composer action inert before its exit animation ends #448 fixed it. The button clears within about 3 s and no longer lingers in the accessibility tree. The ring that looked like a spinner is the context-usage gauge (Composer keeps showing Stop response after a run finishes #443 has details).
   • Two channels at once: still no cross-target actions. Say which windows a menu command, launch or open created desktop-tools#17 still blocks one of the two agents; there's a new detail on that issue about TextEdit opening a window by itself at launch.
   • Since this Canary: fix(channel): build desktop tools from desktop-tools' shared definitions #450 makes Ace import its desktop tool definitions from @githubnext/desktop-tools/tools, which the new MCP server and Desktop Tools.app (Ship a signed Desktop Tools app for agents outside Ace desktop-tools#27) also use.