Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion .github/workflows/ios.yml
Original file line number Diff line number Diff line change
Expand Up @@ -155,8 +155,13 @@ jobs:
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotPresentationOwnsScopeAndRelativeDepth \
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPresentationRefusesAnAcquisitionCapturedForTheOtherProjection \
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testCaptureHintIsTheOnlyAcquisitionViewOfARequest \
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRegularFoldClipsScrollOverflowReparentsAndBooksHints \
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRegularFoldKeepsWindowCarriersButNeverHittableOutsideClip \
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRegularFoldDropsSubPixelContentlessDecorationOnEveryBackend \
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPlainViewportPolicyFoldsWithoutAncestorCursor \
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRegularPresentationRoutesThroughVisibilityFold \
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotScopePolicyMatchesGoldenParityTable \
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testFlatSnapshotProjectionMatchesElementReverseScrollCapture \
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testScrollContainerTypeNamesMatchElementTypeSet \
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPrivateAXDepthLimitedRequiresEveryFrontierResolved \
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testDeepExtensionCountsMissedFrontiers \
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPreferredPrivateAXBackendPlansAsPenalized \
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
- `--udid` with `--platform android` (and `--serial` with an Apple platform) now fails as the flag mistake it is — `INVALID_ARGS` naming the right flag — instead of reaching device resolution and answering `No Apple device with UDID emulator-5580` for an explicitly Android request. `--udid` addresses Apple devices, `--serial` addresses Android and HarmonyOS; matching pairs and requests that name no platform are unchanged.
- Breaking (`--session-lock strip`): a device selector that names a different device than the bound session is no longer silently discarded. `strip` exists to drop redundant platform/scope selectors; when it also dropped `--udid`/`--serial`/`--device`, the command kept running against the *bound* device instead of the one the caller named — a wrong-device action that looks like a success. Such a request now fails with `INVALID_ARGS` under both `reject` and `strip`, and the error carries the two identities structurally (`requestedDevice`, `boundDevice`) plus a hint offering the two real recoveries: close the bound session if the requested device is intended, or remove the selector if the bound device is. The hint no longer suggests `--session-lock strip` for an identity conflict, since following that advice is what produced the wrong-device run. Scope-only stripping (`--platform`, `--target`, `--ios-simulator-device-set`, `--android-device-allowlist`) is unchanged.
- iOS regular snapshots now apply one backend-neutral eligibility rule after every capture backend: a node survives when its accessibility type is interactive or it carries a non-empty label, identifier, or value. This removes the tree backend's extra "hittable non-Other" membership path and drops unlabeled decorative nodes consistently; labeled images, identifier-only nodes, and value-only nodes still survive. Raw snapshot membership is unchanged.
- iOS regular snapshots now run one shared clip fold inside presentation for every capture backend (#1797). Backends serialize reported facts -- every traversed node, at raw traversal depth -- and presentation alone decides what the viewport and scroll clips hide, books the scroll hints, and collapses depth; no backend carries its own copy of that interpretation anymore (the copies are what produced the scroll-overflow leak class, #1784). Three intentional edge deltas ride along, all in the direction of one backend-neutral rule: sub-pixel content-free decorations are now dropped by every backend (previously private-AX only); labeled offscreen Application/Window carriers now survive on every backend (previously tree only), still never hittable; and a query-sweep recovery snapshot without `-i` no longer lists offscreen elements. Nothing outside its clip, and nothing without geometry, is ever `hittable` in a regular snapshot, whatever the backend reported.
- iOS `snapshot --raw` is now the acquired accessibility tree on every backend that can serve it (#1797). A raw request that recovered onto the private-AX backend — the route an app whose XCTest tree capture fails takes — returned the *regular* projection's viewport-pruned nodes labeled raw: everything scrolled out of the viewport, and every sub-pixel decoration, was missing from the one view whose purpose is showing what the pruned view hid. Raw now keeps every node the backend serialized, at traversal depth, and `--depth` still narrows it (for raw, presented depth *is* traversal depth). Two structural rules replace the hand-synchronized ones: the raw capture plan is derived from each backend's declared ability to serve raw, so the interactive query sweep — which has no hierarchy to return — cannot be planned for a raw request; and presentation refuses an acquisition captured for the other projection instead of relabeling it, dropping that tier with a structured failure. Breaking in the same direction: `snapshot --raw -i` now returns the acquired tree instead of an interactive-filtered one — `-i` narrows the regular projection, and the pair used to produce a third membership rule that differed per backend. Regular and `-i` output is unchanged. Backends now read one derived capture hint rather than the request itself, so what a capture is allowed to skip is stated once, next to the proof that skipping it keeps the projection complete.
- Android `snapshot --raw` is now the acquired accessibility tree (#1832 C3): the three regular-projection pruners — nodes Android marks invisible, stale application windows, and covered same-window surfaces — no longer run at parse time, so `--raw` keeps everything the helper serialized (normalization only) and pruned content is recoverable for diagnosis. Regular and `-i` output is unchanged (property-checked identical to the previous implementation over 12,000 random tree × projection pairs, hidden-content hints included). Also: Android blocking-dialog recovery now reads the same daemon presentation an agent's `snapshot` sees instead of a hand-rolled subset, and acts on its occlusion result — a stale "App isn't responding" surface left under the foreground one no longer triggers recovery, and a covered "Close app" is never tapped ahead of the visible one; the Android freshness route signature no longer keys on `role`/`selected`, fields the Android backend never carries; and the Android helper's declared fidelity residues (no `checked`/`checkable`/`selected`/`long-clickable`, 5000-node cap before scoping, API-level cache-reset divergence) are recorded in `CONTEXT.md`.
- `agent-device mcp` now carries its own usage guidance, so MCP-only clients (Codex CLI, Cursor, custom agents) no longer depend on a separately installed skill (#1833). The handshake `instructions` — returned by both `server/discover` and, newly, the legacy `initialize` — is a compact (< 2 KB, the Claude Code truncation limit) workflow card: start with `open {app, foreground: true}` instead of probing, act with `settle: true` and continue from the diff, verify with `wait`/`is`/`get`/`find`, copy `@refs` byte-for-byte, recover from sparse/AX-unavailable, follow error hints, `close`. A new MCP-only `help` tool serves the full guides on demand: no `topic` returns the CLI's decision card; `topic` returns `agent-device help <topic|command>` verbatim (workflow, gestures, scripting, tv, macos, web, remote, debugging, …, or any tool name for its complete flag reference), prefixed with the one-line CLI→tool-property mapping. `help` is router-owned rather than a command descriptor, so it appears in `tools/list` only — not in the CLI, Node client, or `batch` — and its description tells the model it is not a startup step. Legacy `initialize` gains the optional `instructions` field; no other legacy field changes.
Expand Down
5 changes: 5 additions & 0 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,11 @@ narrowing complete.
**Snapshot eligibility**:
Membership in a presented snapshot projection, independent of whether a node is currently hittable.

**Clip fold**:
The regular projection's single visibility interpreter, run inside presentation for every backend:
viewport and scroll-container clipping, ancestor projection, scroll hints, and collapsed depth.
Platform differences enter as a fold policy, never as a backend exception.

**Presented node**:
A wire-facing snapshot value produced at the presentation boundary.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,8 @@ extension RunnerTests {
hint: hint,
viewport: viewport
)
// Serialization-level emptiness only: an acquired-but-fully-clipped tree is presentation's
// verdict now, surfaced by the plan's sparse classifier on the presented payload (#1797).
if nodes.count <= 1 {
NSLog("AGENT_DEVICE_RUNNER_PRIVATE_AX_SNAPSHOT_SPARSE=%ld", nodes.count)
return nil
Expand Down Expand Up @@ -213,7 +215,8 @@ extension RunnerTests {
effectiveDepth: depthLimited ? effectiveDepth : nil,
customActions: Self.privateAXCustomActionCoverage(
response[RunnerAXSnapshotCustomActionsKey]
)
),
viewport: viewport
)
#else
return nil
Expand Down Expand Up @@ -752,14 +755,20 @@ extension RunnerTests {
]
],
]
let nodes = privateAXAcquisition(
rawRoot: tree,
hint: CaptureHint(
projection: .regular, depth: nil, interactiveOnly: true, customActions: false),
viewport: CGRect(x: 0, y: 0, width: 390, height: 844)
let viewport = CGRect(x: 0, y: 0, width: 390, height: 844)
let hint = CaptureHint(
projection: .regular, depth: nil, interactiveOnly: true, customActions: false)
let acquired = privateAXAcquisition(rawRoot: tree, hint: hint, viewport: viewport)
// Acquisition serializes the drawer too; the shared fold is what hides it (#1797).
XCTAssertTrue(acquired.compactMap(\.label).contains("Admin settings"))

let capture = SnapshotPresentation.presentRegular(
SnapshotAcquisition(
hint: hint, nodes: acquired, truncated: false, effectiveDepth: nil, viewport: viewport),
options: PresentationOptions(interactiveOnly: true, depth: nil, scope: nil, raw: false),
policy: .cursorProjected
)

let labels = nodes.compactMap { $0.label }
let labels = (capture.payload.nodes ?? []).compactMap { $0.label }
XCTAssertEqual(
labels,
["Blue Sky", "Callstack", "Welcome back", "Email", "Password", "Sign in", "Forgot password?"]
Expand Down
Loading
Loading