Simulator: expose InjectTouch/InjectKey through the simulator-cli control protocol (hands-off UI automation)

[ObjSal]

Summary

Please expose the gui-server's existing input-injection messages through the simulator's control protocol (simulator-cli), so UI automation against the hosted simulator doesn't have to synthesize macOS-level input events.

Everything needed already exists:

• The gui-server API ships inject_touch() / inject_key() (SDK 1.0.0, api/gui-server/src/lib.rs), with wire messages InjectTouch (id 55) and InjectKey (id 56) in the service manifest.
• simulator-cli already has an automation-flavored control protocol — ping (proto/caps), run <AppId>, screenshots, recording, scale, theme.

What's missing is the last inch: the messages are ungrouped (so a third-party app or tool can't be granted them), and simulator-cli has no command that forwards to them. A tap <x> <y> [down|up|move] and key <keycode> <down|up> in the control protocol — or alternatively a grantable permission group for the two messages in hosted builds — would close the gap.

Why

We drive full UI test suites against the hosted simulator (the harness behind the apps we've sideloaded on Passport Prime). Today the only way to deliver input is posting global CoreGraphics events at screen coordinates, which has real costs:

• The suite owns the machine: the cursor is stolen, the sim window must be frontmost and unobstructed, and one stray human click mid-run silently diverts a tap and cascades failures.
• Runs die with the screen locked or the Mac asleep — a CGEventPost into a locked session blocks indefinitely, so an overnight run just hangs.
• Targeted delivery is a dead end: we measured CGEventPostToPid directly against the sim window (correct coordinates, unoccluded, before/after window-id captures byte-identical) — pid-posted events never reach the winit canvas.
• It's also how host-input quirks leak into automation (e.g. the spacebar forwarding issue in Simulator drops the physical (host) spacebar — NamedKey::Space not forwarded to the device #7).

With tap/key in the control protocol, suites become deterministic, headless-friendly, and machine-independent — no Accessibility permission, no screen geometry, no window management, and the sim could even run on a locked/CI machine.

Sketch

simulator-cli stdin commands, mirroring the existing style:

tap 240 400          # down+up at screen coords (480x800 space)
tapd 240 400         # down only
tapu 240 400         # up (for drags: tapd, tapm..., tapu)
tapm 240 380         # move while down
key 36 d / key 36 u  # InjectKey press/release

forwarding to GuiServer::inject_touch / inject_key. caps=runInput in the ping reply would let harnesses feature-detect it.

Happy to test a build against our suites (nine of them, ~130 legs) and report back.

[ObjSal]

Follow-up after testing, to reframe this from "please build" to "please mirror what already exists":

On real hardware this is already fully solved. The usb-debug service exposes a Command::Tap { x, y, kind } and Command::InputText protocol that calls gui_server::inject_touch/inject_key, and the SDK ships foundation-passport-drive as a complete host-side driver for it — tap, swipe, input-text, screenshot, tap-screenshot, run <json-sequence>, plus an mcp mode for AI-driven automation. That's exactly the hands-off, coordinate-based, screen-independent input path we want, and it works today over USB in Developer Mode.

The gap is only the hosted simulator. usb-debug is ARM-only, so passport-drive can't drive the sim, and the sim's control process (simulator-cli, PID 26, reads stdin) has screenshot/scale/theme commands but no input verb. So the request narrows to: mirror the usb-debug Tap/InputText commands into simulator-cli (or route them through the existing control channel), so the same automation works against the sim without synthesizing macOS CGEvents.

Confirmed for context: CGEventPostToPid aimed at the sim's winit window is a dead end (measured — pid-posted events never reach the canvas), which is why the sim currently forces global CGEvent posting (steals the cursor, needs an unlocked/awake Mac). A simulator-cli tap/key closes that gap with the injection primitives gui-server already has.

Happy to test a build and, if useful, contribute the simulator-cli side — it looks like a small addition mirroring the usb-debug dispatch.

[georgesFoundation]

This is already merged but not yet deployed;)