From 935bc26ee568674633d96c240a544244db8309de Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Pierzcha=C5=82a?= Date: Fri, 21 Aug 2026 08:23:10 +0200 Subject: [PATCH 1/2] feat(ios): move the regular-projection clip fold into snapshot presentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both iOS snapshot backends carried their own copy of the visibility fold: the tree walker and the private-AX serializer each computed viewport-and-scroll-clip intersection, ancestor projection, hidden-content hints, and collapsed depth during acquisition. Hand-synchronized copies of that interpretation are what produced the scroll-overflow leak class (#1784), and C1 (fact-availability neutrality) could not hold while acquisition decided what a screen shows. Acquisition backends are now fact serializers: every traversed node is emitted at raw traversal depth with its reported frame, and SnapshotAcquisition carries the viewport. presentRegular runs the one clip fold for every backend -- viewport ∩ scroll clip, the ancestor cursor (an out-of-clip Cell or scroll container hides its clamped descendants), the sub-pixel decoration rule, scroll hints booked onto anchors, reparenting with collapsed depth -- and narrows the emitted hittable to the clip: nothing outside its clip, and nothing without geometry, is ever hittable, whatever the backend reported. Platform differences are a SnapshotFoldPolicy input to the shared algorithm (iOS cursor-projected; macOS/tvOS plain viewport), never a backend exception. The private-AX backend collapses to ONE serializer for both projections, and the flat filter-decision family dies with the acquisition gates it fed. Three intentional edge deltas, each toward one backend-neutral rule: sub-pixel content-free decorations now drop on every backend (was private-AX only); labeled offscreen Application/Window carriers survive on every backend (was tree only), never hittable; query-sweep regular without -i is viewport-folded. Declared acquisition residues: the traversal-depth budget cut, the sweep's frameless-element drop, the private-AX bridge's device-side cap. Refs #1797 (migration step 3, clip-fold delta). --- .github/workflows/ios.yml | 5 + CHANGELOG.md | 1 + CONTEXT.md | 5 + .../RunnerTests+AXSnapshotFallback.swift | 25 +- .../RunnerTests+FlatSnapshotFiltering.swift | 146 ++------- .../RunnerTests+PrivateAXPresentation.swift | 151 +++------ .../RunnerTests+Snapshot.swift | 294 +++++------------- .../RunnerTests+SnapshotPresentation.swift | 204 +++++++++++- ...unnerTests+SnapshotPresentationTests.swift | 147 +++++++++ .../adr/0004-ios-snapshot-backend-strategy.md | 15 + 10 files changed, 527 insertions(+), 466 deletions(-) diff --git a/.github/workflows/ios.yml b/.github/workflows/ios.yml index a61c9efdab..c0f8b34aae 100644 --- a/.github/workflows/ios.yml +++ b/.github/workflows/ios.yml @@ -155,8 +155,13 @@ jobs: -only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotPresentationOwnsScopeAndRelativeDepth \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testPresentationRefusesAnAcquisitionCapturedForTheOtherProjection \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testCaptureHintIsTheOnlyAcquisitionViewOfARequest \ + -only-testing:AgentDeviceRunnerUITests/RunnerTests/testRegularFoldClipsScrollOverflowReparentsAndBooksHints \ + -only-testing:AgentDeviceRunnerUITests/RunnerTests/testRegularFoldKeepsWindowCarriersButNeverHittableOutsideClip \ + -only-testing:AgentDeviceRunnerUITests/RunnerTests/testRegularFoldDropsSubPixelContentlessDecorationOnEveryBackend \ + -only-testing:AgentDeviceRunnerUITests/RunnerTests/testPlainViewportPolicyFoldsWithoutAncestorCursor \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotScopePolicyMatchesGoldenParityTable \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testFlatSnapshotProjectionMatchesElementReverseScrollCapture \ + -only-testing:AgentDeviceRunnerUITests/RunnerTests/testScrollContainerTypeNamesMatchElementTypeSet \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testPrivateAXDepthLimitedRequiresEveryFrontierResolved \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testDeepExtensionCountsMissedFrontiers \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testPreferredPrivateAXBackendPlansAsPenalized \ diff --git a/CHANGELOG.md b/CHANGELOG.md index 34212bc4b0..382f9bc05c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,7 @@ - `--udid` with `--platform android` (and `--serial` with an Apple platform) now fails as the flag mistake it is — `INVALID_ARGS` naming the right flag — instead of reaching device resolution and answering `No Apple device with UDID emulator-5580` for an explicitly Android request. `--udid` addresses Apple devices, `--serial` addresses Android and HarmonyOS; matching pairs and requests that name no platform are unchanged. - Breaking (`--session-lock strip`): a device selector that names a different device than the bound session is no longer silently discarded. `strip` exists to drop redundant platform/scope selectors; when it also dropped `--udid`/`--serial`/`--device`, the command kept running against the *bound* device instead of the one the caller named — a wrong-device action that looks like a success. Such a request now fails with `INVALID_ARGS` under both `reject` and `strip`, and the error carries the two identities structurally (`requestedDevice`, `boundDevice`) plus a hint offering the two real recoveries: close the bound session if the requested device is intended, or remove the selector if the bound device is. The hint no longer suggests `--session-lock strip` for an identity conflict, since following that advice is what produced the wrong-device run. Scope-only stripping (`--platform`, `--target`, `--ios-simulator-device-set`, `--android-device-allowlist`) is unchanged. - iOS regular snapshots now apply one backend-neutral eligibility rule after every capture backend: a node survives when its accessibility type is interactive or it carries a non-empty label, identifier, or value. This removes the tree backend's extra "hittable non-Other" membership path and drops unlabeled decorative nodes consistently; labeled images, identifier-only nodes, and value-only nodes still survive. Raw snapshot membership is unchanged. +- iOS regular snapshots now run one shared clip fold inside presentation for every capture backend (#1797). Backends serialize reported facts -- every traversed node, at raw traversal depth -- and presentation alone decides what the viewport and scroll clips hide, books the scroll hints, and collapses depth; no backend carries its own copy of that interpretation anymore (the copies are what produced the scroll-overflow leak class, #1784). Three intentional edge deltas ride along, all in the direction of one backend-neutral rule: sub-pixel content-free decorations are now dropped by every backend (previously private-AX only); labeled offscreen Application/Window carriers now survive on every backend (previously tree only), still never hittable; and a query-sweep recovery snapshot without `-i` no longer lists offscreen elements. Nothing outside its clip, and nothing without geometry, is ever `hittable` in a regular snapshot, whatever the backend reported. - iOS `snapshot --raw` is now the acquired accessibility tree on every backend that can serve it (#1797). A raw request that recovered onto the private-AX backend — the route an app whose XCTest tree capture fails takes — returned the *regular* projection's viewport-pruned nodes labeled raw: everything scrolled out of the viewport, and every sub-pixel decoration, was missing from the one view whose purpose is showing what the pruned view hid. Raw now keeps every node the backend serialized, at traversal depth, and `--depth` still narrows it (for raw, presented depth *is* traversal depth). Two structural rules replace the hand-synchronized ones: the raw capture plan is derived from each backend's declared ability to serve raw, so the interactive query sweep — which has no hierarchy to return — cannot be planned for a raw request; and presentation refuses an acquisition captured for the other projection instead of relabeling it, dropping that tier with a structured failure. Breaking in the same direction: `snapshot --raw -i` now returns the acquired tree instead of an interactive-filtered one — `-i` narrows the regular projection, and the pair used to produce a third membership rule that differed per backend. Regular and `-i` output is unchanged. Backends now read one derived capture hint rather than the request itself, so what a capture is allowed to skip is stated once, next to the proof that skipping it keeps the projection complete. - Android `snapshot --raw` is now the acquired accessibility tree (#1832 C3): the three regular-projection pruners — nodes Android marks invisible, stale application windows, and covered same-window surfaces — no longer run at parse time, so `--raw` keeps everything the helper serialized (normalization only) and pruned content is recoverable for diagnosis. Regular and `-i` output is unchanged (property-checked identical to the previous implementation over 12,000 random tree × projection pairs, hidden-content hints included). Also: Android blocking-dialog recovery now reads the same daemon presentation an agent's `snapshot` sees instead of a hand-rolled subset, and acts on its occlusion result — a stale "App isn't responding" surface left under the foreground one no longer triggers recovery, and a covered "Close app" is never tapped ahead of the visible one; the Android freshness route signature no longer keys on `role`/`selected`, fields the Android backend never carries; and the Android helper's declared fidelity residues (no `checked`/`checkable`/`selected`/`long-clickable`, 5000-node cap before scoping, API-level cache-reset divergence) are recorded in `CONTEXT.md`. - `agent-device mcp` now carries its own usage guidance, so MCP-only clients (Codex CLI, Cursor, custom agents) no longer depend on a separately installed skill (#1833). The handshake `instructions` — returned by both `server/discover` and, newly, the legacy `initialize` — is a compact (< 2 KB, the Claude Code truncation limit) workflow card: start with `open {app, foreground: true}` instead of probing, act with `settle: true` and continue from the diff, verify with `wait`/`is`/`get`/`find`, copy `@refs` byte-for-byte, recover from sparse/AX-unavailable, follow error hints, `close`. A new MCP-only `help` tool serves the full guides on demand: no `topic` returns the CLI's decision card; `topic` returns `agent-device help ` verbatim (workflow, gestures, scripting, tv, macos, web, remote, debugging, …, or any tool name for its complete flag reference), prefixed with the one-line CLI→tool-property mapping. `help` is router-owned rather than a command descriptor, so it appears in `tools/list` only — not in the CLI, Node client, or `batch` — and its description tells the model it is not a startup step. Legacy `initialize` gains the optional `instructions` field; no other legacy field changes. diff --git a/CONTEXT.md b/CONTEXT.md index eee240082c..cf3fab1829 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -198,6 +198,11 @@ narrowing complete. **Snapshot eligibility**: Membership in a presented snapshot projection, independent of whether a node is currently hittable. +**Clip fold**: +The regular projection's single visibility interpreter, run inside presentation for every backend: +viewport and scroll-container clipping, ancestor projection, scroll hints, and collapsed depth. +Platform differences enter as a fold policy, never as a backend exception. + **Presented node**: A wire-facing snapshot value produced at the presentation boundary. diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+AXSnapshotFallback.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+AXSnapshotFallback.swift index 357a378e5c..4e95076b33 100644 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+AXSnapshotFallback.swift +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+AXSnapshotFallback.swift @@ -184,6 +184,8 @@ extension RunnerTests { hint: hint, viewport: viewport ) + // Serialization-level emptiness only: an acquired-but-fully-clipped tree is presentation's + // verdict now, surfaced by the plan's sparse classifier on the presented payload (#1797). if nodes.count <= 1 { NSLog("AGENT_DEVICE_RUNNER_PRIVATE_AX_SNAPSHOT_SPARSE=%ld", nodes.count) return nil @@ -213,7 +215,8 @@ extension RunnerTests { effectiveDepth: depthLimited ? effectiveDepth : nil, customActions: Self.privateAXCustomActionCoverage( response[RunnerAXSnapshotCustomActionsKey] - ) + ), + viewport: viewport ) #else return nil @@ -752,14 +755,20 @@ extension RunnerTests { ] ], ] - let nodes = privateAXAcquisition( - rawRoot: tree, - hint: CaptureHint( - projection: .regular, depth: nil, interactiveOnly: true, customActions: false), - viewport: CGRect(x: 0, y: 0, width: 390, height: 844) + let viewport = CGRect(x: 0, y: 0, width: 390, height: 844) + let hint = CaptureHint( + projection: .regular, depth: nil, interactiveOnly: true, customActions: false) + let acquired = privateAXAcquisition(rawRoot: tree, hint: hint, viewport: viewport) + // Acquisition serializes the drawer too; the shared fold is what hides it (#1797). + XCTAssertTrue(acquired.compactMap(\.label).contains("Admin settings")) + + let capture = SnapshotPresentation.presentRegular( + SnapshotAcquisition( + hint: hint, nodes: acquired, truncated: false, effectiveDepth: nil, viewport: viewport), + options: PresentationOptions(interactiveOnly: true, depth: nil, scope: nil, raw: false), + policy: .cursorProjected ) - - let labels = nodes.compactMap { $0.label } + let labels = (capture.payload.nodes ?? []).compactMap { $0.label } XCTAssertEqual( labels, ["Blue Sky", "Callstack", "Welcome back", "Email", "Password", "Sign in", "Forgot password?"] diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+FlatSnapshotFiltering.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+FlatSnapshotFiltering.swift index 9d0eff5b9e..982584341a 100644 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+FlatSnapshotFiltering.swift +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+FlatSnapshotFiltering.swift @@ -1,19 +1,5 @@ import XCTest -struct FlatSnapshotFilterNode { - let isRoot: Bool - let visible: Bool -} - -struct FlatSnapshotFilterDecision { - let include: Bool -} - -enum FlatSnapshotVisibilityPolicy { - case interactiveOnly - case viewportProjected -} - enum FlatSnapshotGeometry { case geometryless case framed(intersectsViewportAndScrollClip: Bool) @@ -65,7 +51,7 @@ struct FlatSnapshotProjectionTransition { } extension RunnerTests { - func flatSnapshotGeometry( + static func flatSnapshotGeometry( frame: CGRect, intersectsViewportAndScrollClip: Bool ) -> FlatSnapshotGeometry { @@ -73,21 +59,23 @@ extension RunnerTests { return .framed(intersectsViewportAndScrollClip: intersectsViewportAndScrollClip) } - func flatSnapshotDescendantVisibility( - elementType: XCUIElement.ElementType?, + /// Wire-name variant: the presentation fold works on `RawAXNode.type` strings, the one node + /// vocabulary every backend already shares (#1797 — backend-neutral by construction). + static func flatSnapshotDescendantVisibility( + typeName: String?, hasChildren: Bool ) -> FlatSnapshotDescendantVisibility { - guard hasChildren, let elementType else { return .independent } - if elementType == .cell || Self.scrollContainerTypes.contains(elementType) { + guard hasChildren, let typeName else { return .independent } + if typeName == "Cell" || Self.scrollContainerTypeNames.contains(typeName) { return .owned } return .independent } - func flatSnapshotProjectionTransition( + static func flatSnapshotProjectionTransition( frame: CGRect, intersectsViewportAndScrollClip: Bool, - elementType: XCUIElement.ElementType?, + typeName: String?, hasChildren: Bool, cursor: FlatSnapshotProjectionCursor ) -> FlatSnapshotProjectionTransition { @@ -99,7 +87,7 @@ extension RunnerTests { intersectsViewportAndScrollClip: intersectsViewportAndScrollClip ), descendantVisibility: flatSnapshotDescendantVisibility( - elementType: elementType, + typeName: typeName, hasChildren: hasChildren ) ), @@ -108,27 +96,6 @@ extension RunnerTests { ) } - /// Regular-projection acquisition gate. The raw projection never reaches it: raw acquisition is - /// the acquired tree, so it has no visibility filter to consult (#1797 D4). - func flatSnapshotFilterDecision( - _ node: FlatSnapshotFilterNode, - hint: CaptureHint, - visibilityPolicy: FlatSnapshotVisibilityPolicy - ) -> FlatSnapshotFilterDecision { - let include: Bool - if node.isRoot { - include = true - } else if !node.visible - && (hint.interactiveOnly || visibilityPolicy == .viewportProjected) - { - include = false - } else { - include = true - } - - return FlatSnapshotFilterDecision(include: include) - } - func privateAXInteractiveCandidate(rawElementType: Int) -> Bool { guard let type = flatSnapshotElementType(rawElementType: rawElementType) else { return false @@ -146,6 +113,17 @@ extension RunnerTests { } #if AGENT_DEVICE_RUNNER_UNIT_TESTS + /// The fold's string vocabulary must track the XCUIElement type sets it mirrors: a scroll + /// container renamed or added in one place but not the other silently changes which subtrees + /// own their descendants' visibility. + func testScrollContainerTypeNamesMatchElementTypeSet() { + XCTAssertEqual( + Self.scrollContainerTypeNames, + Set(Self.scrollContainerTypes.map(elementTypeName)) + ) + XCTAssertEqual(elementTypeName(.cell), "Cell") + } + func testFlatSnapshotProjectionMatchesElementReverseScrollCapture() { struct FixtureNode { let label: String @@ -167,21 +145,21 @@ extension RunnerTests { cursor: FlatSnapshotProjectionCursor, scrollAnchor: (index: Int, rect: CGRect)? ) { - let intersects = isVisibleInRegularSnapshot( + let intersects = Self.isVisibleInRegularSnapshot( node.frame, viewport: CGRect(x: 0, y: 0, width: 402, height: 874), scrollContainerAnchor: scrollAnchor ) - let transition = flatSnapshotProjectionTransition( + let transition = Self.flatSnapshotProjectionTransition( frame: node.frame, intersectsViewportAndScrollClip: intersects, - elementType: node.type, + typeName: elementTypeName(node.type), hasChildren: !node.children.isEmpty, cursor: cursor ) if transition.decision.presentationVisible { visibleLabels.append(node.label) } if let hiddenFrame = transition.hiddenContentFrame, let scrollAnchor { - rememberHiddenContentHint( + Self.rememberHiddenContentHint( for: hiddenFrame, relativeTo: scrollAnchor, hints: &hints @@ -269,89 +247,23 @@ extension RunnerTests { XCTAssertEqual(hints[0]?.above, false) XCTAssertEqual(hints[0]?.below, true) - let scopedRoot = flatSnapshotProjectionTransition( + let scopedRoot = Self.flatSnapshotProjectionTransition( frame: offscreenThemeRowFrame, intersectsViewportAndScrollClip: false, - elementType: .cell, + typeName: "Cell", hasChildren: true, cursor: .root ) - let scopedClampedChild = flatSnapshotProjectionTransition( + let scopedClampedChild = Self.flatSnapshotProjectionTransition( frame: clampedThemeFrame, intersectsViewportAndScrollClip: true, - elementType: .staticText, + typeName: "StaticText", hasChildren: false, cursor: scopedRoot.decision.descendants ) XCTAssertFalse(scopedClampedChild.decision.presentationVisible) } - func testFlatSnapshotFilterDecisionMatrixCoversOptions() { - func hint(interactiveOnly: Bool) -> CaptureHint { - CaptureHint( - projection: .regular, depth: nil, interactiveOnly: interactiveOnly, customActions: false) - } - let visibleContent = FlatSnapshotFilterNode( - isRoot: false, - visible: true - ) - let hiddenInteractive = FlatSnapshotFilterNode( - isRoot: false, - visible: false - ) - let decorative = FlatSnapshotFilterNode( - isRoot: false, - visible: true - ) - let hiddenRoot = FlatSnapshotFilterNode( - isRoot: true, - visible: false - ) - - XCTAssertTrue( - flatSnapshotFilterDecision( - visibleContent, - hint: hint(interactiveOnly: false), - visibilityPolicy: .interactiveOnly - ).include - ) - XCTAssertFalse( - flatSnapshotFilterDecision( - hiddenInteractive, - hint: hint(interactiveOnly: true), - visibilityPolicy: .interactiveOnly - ).include - ) - XCTAssertFalse( - flatSnapshotFilterDecision( - hiddenInteractive, - hint: hint(interactiveOnly: false), - visibilityPolicy: .viewportProjected - ).include - ) - XCTAssertTrue( - flatSnapshotFilterDecision( - hiddenInteractive, - hint: hint(interactiveOnly: false), - visibilityPolicy: .interactiveOnly - ).include - ) - XCTAssertTrue( - flatSnapshotFilterDecision( - hiddenRoot, - hint: hint(interactiveOnly: false), - visibilityPolicy: .viewportProjected - ).include - ) - XCTAssertTrue( - flatSnapshotFilterDecision( - decorative, - hint: hint(interactiveOnly: false), - visibilityPolicy: .interactiveOnly - ).include - ) - } - func testPrivateAXInteractiveCandidatesPreserveBackendInputs() { XCTAssertTrue( privateAXInteractiveCandidate(rawElementType: Int(XCUIElement.ElementType.scrollView.rawValue)), diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXPresentation.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXPresentation.swift index bf4e7591e8..390e035f05 100644 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXPresentation.swift +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXPresentation.swift @@ -1,7 +1,7 @@ import XCTest -/// Reported facts for one private-AX element, read once so both projections describe a node the -/// same way. Only membership and clip-derived state differ between them. +/// Reported facts for one private-AX element, read once so every consumer describes a node the +/// same way. struct PrivateAXFields { let rect: CGRect let label: String @@ -20,116 +20,36 @@ struct PrivateAXFields { } extension RunnerTests { - private static let privateAXProjectionTolerance: CGFloat = 1 - - /// Acquisition entry point for the private AX backend. - /// - /// The regular projection folds the viewport and scroll clips and records scroll hints; the raw - /// projection is the acquired tree, so it drops nothing and keeps traversal depth (#1797 D4 — a - /// recovered `snapshot --raw` used to return viewport-pruned nodes labeled raw). + /// Private-AX acquisition: ONE serializer for both projections. Reported facts at traversal + /// depth -- membership, the clip fold, scroll hints, and collapsed depth are + /// `SnapshotPresentation`'s alone (#1797). The traversal-depth cut is the backend's one + /// narrowing, complete for raw (raw depth *is* traversal depth) and a declared residue for + /// regular; `hittable` is the pre-clip geometric fact the fold narrows per projection. func privateAXAcquisition(rawRoot: [String: Any], hint: CaptureHint, viewport: CGRect) -> [RawAXNode] { var nodes: [RawAXNode] = [] - switch hint.projection { - case .raw: - appendPrivateAXRawNode(rawRoot, to: &nodes, hint: hint, viewport: viewport, - depth: 0, parentIndex: nil) - return nodes - case .regular: - var hints: [Int: (above: Bool, below: Bool)] = [:] - appendPrivateAXNode(rawRoot, to: &nodes, hints: &hints, hint: hint, viewport: viewport, - depth: 0, parentIndex: nil, scrollContext: nil, - projectionCursor: .root) - return applyHiddenContentHints(hints, to: nodes) - } + appendPrivateAXNode(rawRoot, to: &nodes, hint: hint, viewport: viewport, + depth: 0, parentIndex: nil) + return nodes } - /// Raw projection: every serialized node, in traversal order, at traversal depth. Depth is the - /// one narrowing a raw acquisition can prove complete, because raw depth *is* traversal depth. - private func appendPrivateAXRawNode(_ raw: [String: Any], to nodes: inout [RawAXNode], + private func appendPrivateAXNode(_ raw: [String: Any], to nodes: inout [RawAXNode], hint: CaptureHint, viewport: CGRect, depth: Int, parentIndex: Int?) { if let limit = hint.depth, depth > limit { return } let fields = privateAXFields(raw) - // No clip fold outside the regular projection: hittability stays the reported geometric fact. - let onScreen = fields.hasFrame && isVisibleInViewport(fields.rect, viewport) + let onScreen = fields.hasFrame && Self.isVisibleInViewport(fields.rect, viewport) let index = nodes.count nodes.append( privateAXNode(fields, index: index, depth: depth, parentIndex: parentIndex, onScreen: onScreen) ) for child in fields.children { - appendPrivateAXRawNode(child, to: &nodes, hint: hint, viewport: viewport, + appendPrivateAXNode(child, to: &nodes, hint: hint, viewport: viewport, depth: depth + 1, parentIndex: index) } } - private func appendPrivateAXNode(_ raw: [String: Any], to nodes: inout [RawAXNode], - hints: inout [Int: (above: Bool, below: Bool)], hint: CaptureHint, viewport: CGRect, - depth: Int, parentIndex: Int?, - scrollContext: (index: Int, rect: CGRect)?, projectionCursor: FlatSnapshotProjectionCursor) - { - if let limit = hint.depth, depth > limit { return } - let fields = privateAXFields(raw) - let rect = fields.rect - let children = fields.children - let elementType = fields.elementType - let negligibleDecoration = parentIndex != nil - && !fields.hasSemanticContent - && (!fields.hasFrame - || rect.width <= Self.privateAXProjectionTolerance - || rect.height <= Self.privateAXProjectionTolerance) - let onScreen = fields.hasFrame - && isVisibleInRegularSnapshot( - rect, - viewport: viewport, - scrollContainerAnchor: scrollContext - ) - let projectionTransition = flatSnapshotProjectionTransition( - frame: rect, - intersectsViewportAndScrollClip: onScreen, - elementType: elementType, - hasChildren: !children.isEmpty, - cursor: projectionCursor - ) - let projection = projectionTransition.decision - let presentationVisible = projection.presentationVisible && !negligibleDecoration - let decision = flatSnapshotFilterDecision( - FlatSnapshotFilterNode(isRoot: parentIndex == nil, visible: presentationVisible), - hint: hint, visibilityPolicy: .viewportProjected) - let include = decision.include - - if let hiddenFrame = projectionTransition.hiddenContentFrame, let scrollContext { - rememberHiddenContentHint(for: hiddenFrame, relativeTo: scrollContext, hints: &hints) - } - - let currentIndex: Int? - if include { - currentIndex = nodes.count - nodes.append( - privateAXNode( - fields, index: nodes.count, depth: depth, parentIndex: parentIndex, onScreen: onScreen) - ) - } else { currentIndex = parentIndex } - - let nextScrollContext: (index: Int, rect: CGRect)? - if include, let elementType, let currentIndex { - nextScrollContext = scrollContainerAnchor( - for: elementType, - hasChildren: !children.isEmpty, - visible: onScreen, - frame: rect, - nodeIndex: currentIndex - ) ?? scrollContext - } else { nextScrollContext = scrollContext } - for child in children { - appendPrivateAXNode(child, to: &nodes, hints: &hints, hint: hint, viewport: viewport, - depth: depth + 1, parentIndex: currentIndex, - scrollContext: nextScrollContext, - projectionCursor: projection.descendants) - } - } - private func privateAXFields(_ raw: [String: Any]) -> PrivateAXFields { let rawType = privateAXPresentationInt(raw["type"]) ?? 0 return PrivateAXFields( @@ -201,10 +121,28 @@ extension RunnerTests { "frame": frame(16, 900, 360, 44)]]]]] } + /// Acquire with the private-AX serializer, then present through the shared regular fold -- + /// the production route for this backend since the fold moved into presentation (#1797). + fileprivate func privateAXRegularPresentation( + rawRoot: [String: Any], + viewport: CGRect, + interactiveOnly: Bool = false + ) -> [PresentedNode] { + let hint = CaptureHint( + projection: .regular, depth: nil, interactiveOnly: interactiveOnly, customActions: false) + let acquired = privateAXAcquisition(rawRoot: rawRoot, hint: hint, viewport: viewport) + return SnapshotPresentation.presentRegular( + SnapshotAcquisition( + hint: hint, nodes: acquired, truncated: false, effectiveDepth: nil, viewport: viewport), + options: PresentationOptions( + interactiveOnly: interactiveOnly, depth: nil, scope: nil, raw: false), + policy: .cursorProjected + ).payload.nodes ?? [] + } + func testPrivateAXRegularPresentationProjectsToViewportAndKeepsScrollHint() { - let nodes = privateAXAcquisition(rawRoot: Self.privateAXScrolledFixture, - hint: CaptureHint(projection: .regular, depth: nil, interactiveOnly: false, - customActions: false), + let nodes = privateAXRegularPresentation( + rawRoot: Self.privateAXScrolledFixture, viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) XCTAssertEqual(nodes.compactMap(\.label), ["Element", "Profile picture"]) let scrollView = nodes.first { $0.type == "ScrollView" } @@ -214,14 +152,12 @@ extension RunnerTests { /// #1797 D4: the raw projection is the acquired tree. The offscreen row and the sub-pixel /// decoration the regular projection folds away are both present, at traversal depth, and every - /// regular node still appears — `regular ⊆ raw` on the same capture. + /// regular node still appears -- `regular ⊆ raw` on the same capture. func testPrivateAXRawProjectionKeepsEveryAcquiredNode() { let viewport = CGRect(x: 0, y: 0, width: 402, height: 874) let root = Self.privateAXScrolledFixture - let regular = privateAXAcquisition(rawRoot: root, - hint: CaptureHint(projection: .regular, depth: nil, interactiveOnly: true, - customActions: false), - viewport: viewport) + let regular = privateAXRegularPresentation(rawRoot: root, viewport: viewport, + interactiveOnly: true) let raw = privateAXAcquisition(rawRoot: root, hint: CaptureHint(projection: .raw, depth: nil, interactiveOnly: false, customActions: false), viewport: viewport) @@ -267,10 +203,8 @@ extension RunnerTests { ["type": Int(XCUIElement.ElementType.switch.rawValue), "label": "Theme", "frame": frame(340, 96, 46, 44)]]]]]]] - let nodes = privateAXAcquisition(rawRoot: root, - hint: CaptureHint(projection: .regular, depth: nil, interactiveOnly: false, - customActions: false), - viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) + let nodes = privateAXRegularPresentation( + rawRoot: root, viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) XCTAssertEqual(nodes.compactMap(\.label), ["Element"]) XCTAssertEqual(nodes.first { $0.type == "Table" }?.hiddenContentBelow, true) @@ -284,10 +218,9 @@ extension RunnerTests { "label": "Settings semantics", "frame": zero, "children": [[ "type": Int(XCUIElement.ElementType.button.rawValue), "label": "Theme", "frame": zero], ["type": Int(XCUIElement.ElementType.other.rawValue), "frame": zero]]]]] - let nodes = privateAXAcquisition(rawRoot: root, - hint: CaptureHint(projection: .regular, depth: nil, interactiveOnly: true, - customActions: false), - viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) + let nodes = privateAXRegularPresentation( + rawRoot: root, viewport: CGRect(x: 0, y: 0, width: 402, height: 874), + interactiveOnly: true) XCTAssertEqual(nodes.compactMap(\.label), ["Element", "Settings semantics", "Theme"]) XCTAssertEqual(nodes.filter { $0.index != 0 }.map(\.hittable), [false, false]) XCTAssertFalse(nodes.contains { $0.type == "Other" }) diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift index 428b4ef3d2..7770e7b707 100644 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift @@ -27,16 +27,12 @@ extension RunnerTests { let hittable: Bool let focused: Bool let selected: Bool - let visible: Bool } private struct SnapshotTraversalEntry { let snapshot: XCUIElementSnapshot let depth: Int - let visibleDepth: Int let parentIndex: Int? - let nearestScrollAnchor: (index: Int, rect: CGRect)? - let projectionCursor: FlatSnapshotProjectionCursor } struct SnapshotCaptureFailure: Error { @@ -110,6 +106,14 @@ extension RunnerTests { .table ] + /// `RawAXNode.type` vocabulary for the presentation fold; pinned to `scrollContainerTypes` + /// through `elementTypeName` by a unit test. + static let scrollContainerTypeNames: Set = [ + "CollectionView", + "ScrollView", + "Table", + ] + static let flatInteractiveFallbackBudget: TimeInterval = 1.0 // The single production entry point -- always compiled, no unit-test overload. A unit test @@ -147,8 +151,12 @@ extension RunnerTests { return result.elements } + // Acquisition serializes facts: every traversed node is emitted at raw traversal depth, and + // the regular projection's clip fold runs once inside `SnapshotPresentation` (#1797). The two + // walks this backend keeps are budget and augmentation, never membership: the traversal-depth + // cut (declared residue -- regular presentation emits collapsed depth) and the collapsed-tab + // expansion, which needs live element handles. var nodes: [RawAXNode] = [] - var hiddenContentHintsByNodeIndex: [Int: (above: Bool, below: Bool)] = [:] let rootEvaluation = evaluateSnapshot(context.rootSnapshot, in: context) nodes.append( makeSnapshotNode( @@ -170,87 +178,21 @@ extension RunnerTests { } var seen = Set() - let rootScrollAnchor = scrollContainerAnchor( - for: context.rootSnapshot, - visible: rootEvaluation.visible, - nodeIndex: 0 - ) - #if os(iOS) - let rootProjection = flatSnapshotProjectionTransition( - frame: context.rootSnapshot.frame, - intersectsViewportAndScrollClip: rootEvaluation.visible, - elementType: context.rootSnapshot.elementType, - hasChildren: !context.rootSnapshot.children.isEmpty, - cursor: .root - ) - let rootDescendantProjectionCursor = rootProjection.decision.descendants - #else - let rootDescendantProjectionCursor = FlatSnapshotProjectionCursor.root - #endif var stack: [SnapshotTraversalEntry] = context.rootSnapshot.children.map { SnapshotTraversalEntry( snapshot: $0, depth: 1, - visibleDepth: 1, - parentIndex: 0, - nearestScrollAnchor: rootScrollAnchor, - projectionCursor: rootDescendantProjectionCursor + parentIndex: 0 ) } while let entry = stack.popLast() { let snapshot = entry.snapshot let depth = entry.depth - let visibleDepth = entry.visibleDepth let parentIndex = entry.parentIndex - let nearestScrollAnchor = entry.nearestScrollAnchor - let projectionCursor = entry.projectionCursor if let limit = hint.depth, depth > limit { continue } let evaluation = evaluateSnapshot(snapshot, in: context) - let intersectsViewportAndScrollClip = isVisibleInRegularSnapshot( - snapshot.frame, - viewport: context.viewport, - scrollContainerAnchor: nearestScrollAnchor - ) - #if os(iOS) - let projectionTransition = flatSnapshotProjectionTransition( - frame: snapshot.frame, - intersectsViewportAndScrollClip: intersectsViewportAndScrollClip, - elementType: snapshot.elementType, - hasChildren: !snapshot.children.isEmpty, - cursor: projectionCursor - ) - let projection = projectionTransition.decision - #else - let projectionTransition = FlatSnapshotProjectionTransition( - decision: FlatSnapshotProjectionDecision( - presentationVisible: intersectsViewportAndScrollClip, - descendants: .root - ), - hiddenContentFrame: !intersectsViewportAndScrollClip ? snapshot.frame : nil - ) - let projection = projectionTransition.decision - #endif - if let hiddenFrame = projectionTransition.hiddenContentFrame, let nearestScrollAnchor - { - rememberHiddenContentHint( - for: hiddenFrame, - relativeTo: nearestScrollAnchor, - hints: &hiddenContentHintsByNodeIndex - ) - } - let include = shouldAcquireSnapshotNode( - snapshot: snapshot, - label: evaluation.label, - identifier: evaluation.identifier, - valueText: evaluation.valueText, - hint: hint, - hittable: evaluation.hittable, - visible: projection.presentationVisible, - regularSnapshot: true - ) - let key = Self.snapshotTraversalIdentity( elementType: snapshot.elementType, label: evaluation.label, @@ -262,64 +204,48 @@ extension RunnerTests { seen.insert(key) } - let currentIndex = include && !isDuplicate ? nodes.count : parentIndex + let currentIndex = !isDuplicate ? nodes.count : parentIndex if depth < context.maxDepth { - let nextVisibleDepth = include && !isDuplicate ? visibleDepth + 1 : visibleDepth - let nextScrollContainerAnchor: (index: Int, rect: CGRect)? - if include && !isDuplicate { - nextScrollContainerAnchor = - scrollContainerAnchor( - for: snapshot, - visible: intersectsViewportAndScrollClip, - nodeIndex: currentIndex - ) - ?? nearestScrollAnchor - } else { - nextScrollContainerAnchor = nearestScrollAnchor - } for child in snapshot.children.reversed() { stack.append( SnapshotTraversalEntry( snapshot: child, depth: depth + 1, - visibleDepth: nextVisibleDepth, - parentIndex: currentIndex, - nearestScrollAnchor: nextScrollContainerAnchor, - projectionCursor: projection.descendants + parentIndex: currentIndex ) ) } } - if !include || isDuplicate { continue } + if isDuplicate { continue } let index = nodes.count nodes.append( makeSnapshotNode( snapshot: snapshot, evaluation: evaluation, - depth: min(context.maxDepth, visibleDepth), + depth: depth, index: index, parentIndex: parentIndex ) ) - if visibleDepth < context.maxDepth { + if depth < context.maxDepth { appendCollapsedTabFallbackNodes( to: &nodes, containerSnapshot: snapshot, resolveElements: collapsedTabDescendants, - depth: visibleDepth + 1, + depth: depth + 1, parentIndex: index ) } - } return SnapshotAcquisition( hint: hint, - nodes: applyHiddenContentHints(hiddenContentHintsByNodeIndex, to: nodes), + nodes: nodes, truncated: false, - effectiveDepth: nil + effectiveDepth: nil, + viewport: context.viewport ) } @@ -430,30 +356,19 @@ extension RunnerTests { if let limit = hint.depth, depth > limit { return } let evaluation = evaluateSnapshot(snapshot, in: context) - let include = shouldAcquireSnapshotNode( - snapshot: snapshot, - label: evaluation.label, - identifier: evaluation.identifier, - valueText: evaluation.valueText, - hint: hint, - hittable: evaluation.hittable, - visible: evaluation.visible - ) - let currentIndex = include ? nodes.count : parentIndex - if include { - if nodes.count >= Self.rawSnapshotMaxNodes { - throw rawSnapshotTooLargeFailure(nodeCount: nodes.count + 1) - } - nodes.append( - makeSnapshotNode( - snapshot: snapshot, - evaluation: evaluation, - depth: depth, - index: nodes.count, - parentIndex: parentIndex - ) - ) + if nodes.count >= Self.rawSnapshotMaxNodes { + throw rawSnapshotTooLargeFailure(nodeCount: nodes.count + 1) } + let currentIndex = nodes.count + nodes.append( + makeSnapshotNode( + snapshot: snapshot, + evaluation: evaluation, + depth: depth, + index: currentIndex, + parentIndex: parentIndex + ) + ) let children = snapshot.children for child in children { @@ -462,7 +377,13 @@ extension RunnerTests { } try walk(context.rootSnapshot, depth: 0, parentIndex: nil) - return SnapshotAcquisition(hint: hint, nodes: nodes, truncated: false, effectiveDepth: nil) + return SnapshotAcquisition( + hint: hint, + nodes: nodes, + truncated: false, + effectiveDepth: nil, + viewport: context.viewport + ) } func querySweepSnapshotAcquisition( @@ -474,7 +395,13 @@ extension RunnerTests { interactiveRootNode(rect: .zero) ] if hint.depth == 0 { - return SnapshotAcquisition(hint: hint, nodes: nodes, truncated: false, effectiveDepth: nil) + return SnapshotAcquisition( + hint: hint, + nodes: nodes, + truncated: false, + effectiveDepth: nil, + viewport: .infinite + ) } // Bounded by both its own sweep budget and the umbrella capture-plan deadline, so a @@ -498,8 +425,7 @@ extension RunnerTests { element: element, index: 0, parentIndex: 0, - viewport: viewport, - hint: hint + viewport: viewport ) else { continue } @@ -545,7 +471,13 @@ extension RunnerTests { ) ) } - return SnapshotAcquisition(hint: hint, nodes: nodes, truncated: truncated, effectiveDepth: nil) + return SnapshotAcquisition( + hint: hint, + nodes: nodes, + truncated: truncated, + effectiveDepth: nil, + viewport: viewport + ) } func snapshotAccessibilityUnavailable(failure: SnapshotCaptureFailure) -> DataPayload { @@ -880,44 +812,6 @@ extension RunnerTests { // MARK: - Snapshot Filtering - /// Conservative acquisition gate. Regular capture keeps every node its visibility policy can - /// supply; `SnapshotPresentation` alone decides eligibility after the backend returns. - private func shouldAcquireSnapshotNode( - snapshot: XCUIElementSnapshot, - label: String, - identifier: String, - valueText: String?, - hint: CaptureHint, - hittable: Bool, - visible: Bool, - regularSnapshot: Bool = false - ) -> Bool { - let type = snapshot.elementType - let hasContent = !label.isEmpty || !identifier.isEmpty || (valueText != nil) - #if os(iOS) - if regularSnapshot && !visible && type != .application && type != .window { - return false - } - #endif - if regularSnapshot { - #if os(macOS) - if hint.interactiveOnly && !visible && type != .application { - return false - } - #endif - if type == .application || type == .window { return true } - return visible - } - if hint.interactiveOnly { - if isScrollableContainer(snapshot, visible: visible) { return true } - if interactiveTypes.contains(type) { return true } - if hittable && type != .other { return true } - if hasContent { return true } - return false - } - return true - } - private func computedSnapshotHittable( _ snapshot: XCUIElementSnapshot, viewport: CGRect, @@ -1119,8 +1013,7 @@ extension RunnerTests { valueText: valueText, hittable: computedSnapshotHittable(snapshot, viewport: context.viewport, laterNodes: laterNodes), focused: snapshotHasFocus(snapshot), - selected: snapshotIsSelected(snapshot), - visible: isVisibleInViewport(snapshot.frame, context.viewport) + selected: snapshotIsSelected(snapshot) ) } @@ -1235,12 +1128,12 @@ extension RunnerTests { return nil } - func isVisibleInViewport(_ rect: CGRect, _ viewport: CGRect) -> Bool { + static func isVisibleInViewport(_ rect: CGRect, _ viewport: CGRect) -> Bool { if rect.isNull || rect.isEmpty { return false } return rect.intersects(viewport) } - func isVisibleInRegularSnapshot( + static func isVisibleInRegularSnapshot( _ rect: CGRect, viewport: CGRect, scrollContainerAnchor: (index: Int, rect: CGRect)? @@ -1267,37 +1160,21 @@ extension RunnerTests { nodes.append(contentsOf: fallbackNodes) } - private func scrollContainerAnchor( - for snapshot: XCUIElementSnapshot, - visible: Bool, - nodeIndex: Int? - ) -> (index: Int, rect: CGRect)? { - return scrollContainerAnchor( - for: snapshot.elementType, - hasChildren: !snapshot.children.isEmpty, - visible: visible, - frame: snapshot.frame, - nodeIndex: nodeIndex - ) - } - - func scrollContainerAnchor( - for elementType: XCUIElement.ElementType, + static func scrollContainerAnchor( + forTypeName typeName: String, hasChildren: Bool, visible: Bool, frame: CGRect, nodeIndex: Int? ) -> (index: Int, rect: CGRect)? { guard let nodeIndex else { return nil } - if !isScrollableContainer( - elementType: elementType, - hasChildren: hasChildren, - visible: visible - ) { return nil } + guard visible, hasChildren, Self.scrollContainerTypeNames.contains(typeName) else { + return nil + } return (nodeIndex, frame) } - func rememberHiddenContentHint( + static func rememberHiddenContentHint( for frame: CGRect, relativeTo scrollContainerAnchor: (index: Int, rect: CGRect), hints: inout [Int: (above: Bool, below: Bool)] @@ -1314,7 +1191,7 @@ extension RunnerTests { hints[scrollContainerAnchor.index] = hint } - func applyHiddenContentHints( + static func applyHiddenContentHints( _ hints: [Int: (above: Bool, below: Bool)], to nodes: [RawAXNode] ) -> [RawAXNode] { @@ -1576,36 +1453,22 @@ extension RunnerTests { element: XCUIElement, index: Int, parentIndex: Int?, - viewport: CGRect, - hint: CaptureHint + viewport: CGRect ) -> RawAXNode? { var node: RawAXNode? let exceptionMessage = RunnerObjCExceptionCatcher.catchException({ if !element.exists { return } + // Declared residue: a flat element query has no hierarchy for geometryless semantics to + // attach to, so frameless elements are dropped at acquisition rather than presented. let frame = element.frame if frame.isNull || frame.isEmpty { return } - let visible = isVisibleInViewport(frame, viewport) - if hint.interactiveOnly && !visible { return } - #if os(macOS) - if !visible { return } - #endif + let visible = Self.isVisibleInViewport(frame, viewport) let label = element.label.trimmingCharacters(in: .whitespacesAndNewlines) let identifier = element.identifier.trimmingCharacters(in: .whitespacesAndNewlines) let valueText = snapshotValueText(element) let elementType = element.elementType let enabled = element.isEnabled let hittable = visible && enabled && element.isHittable - let filterNode = FlatSnapshotFilterNode( - isRoot: false, - visible: visible - ) - if !flatSnapshotFilterDecision( - filterNode, - hint: hint, - visibilityPolicy: .interactiveOnly - ).include { - return - } node = RawAXNode( index: index, @@ -1631,19 +1494,4 @@ extension RunnerTests { return node } - private func isScrollableContainer(_ snapshot: XCUIElementSnapshot, visible: Bool) -> Bool { - return isScrollableContainer( - elementType: snapshot.elementType, - hasChildren: !snapshot.children.isEmpty, - visible: visible - ) - } - - private func isScrollableContainer( - elementType: XCUIElement.ElementType, - hasChildren: Bool, - visible: Bool - ) -> Bool { - return visible && hasChildren && Self.scrollContainerTypes.contains(elementType) - } } diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentation.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentation.swift index fc6884a2d4..6af16a2204 100644 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentation.swift +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentation.swift @@ -69,6 +69,29 @@ struct SnapshotAcquisition { let truncated: Bool let effectiveDepth: Int? var customActions: SnapshotCustomActionCoverage? = nil + /// Viewport the regular projection's clip fold runs against. `.infinite` disables the fold -- + /// legitimate only for raw acquisitions and depth-0 probes, where no fold applies. + var viewport: CGRect = .infinite +} + +/// Platform policy for the regular projection's visibility fold (#1797: macOS is a policy input +/// to the shared algorithm, never a backend exception). +enum SnapshotFoldPolicy { + /// iOS: ancestor projection cursor -- an out-of-clip Cell or scroll container hides the + /// descendants whose clamped frames would otherwise leak back into the viewport -- plus the + /// sub-pixel decoration rule. + case cursorProjected + /// macOS / tvOS: plain viewport intersection per node, no ancestor cursor, and interactive-only + /// requests additionally drop out-of-viewport Window carriers. + case plainViewport + + static var platformDefault: SnapshotFoldPolicy { + #if os(iOS) + return .cursorProjected + #else + return .plainViewport + #endif + } } /// The only snapshot node shape accepted by response payload assembly. @@ -169,14 +192,26 @@ enum SnapshotPresentation { } } - /// Visible projection: eligibility (an interactive type or non-empty semantic content, below the - /// root carrier), scope, and the acquired scroll hints. The only interpreter of what a screen - /// currently shows. + /// Visible projection: the clip fold (viewport ∩ scroll clip, ancestor cursor, scroll hints), + /// eligibility (an interactive type or non-empty semantic content, below the root carrier), and + /// scope. The only interpreter of what a screen currently shows -- no backend folds its own + /// visibility (#1797). static func presentRegular( _ acquisition: SnapshotAcquisition, - options: PresentationOptions + options: PresentationOptions, + policy: SnapshotFoldPolicy = .platformDefault ) -> SnapshotBackendCapture { - project(acquisition, options: options, projection: .regular) + project( + foldRegularVisibility( + acquisition.nodes, + viewport: acquisition.viewport, + interactiveOnly: options.interactiveOnly, + policy: policy + ), + acquisition: acquisition, + options: options, + projection: .regular + ) } /// Diagnostic projection: the acquired tree, normalized. Scope and depth apply when explicitly @@ -186,7 +221,7 @@ enum SnapshotPresentation { _ acquisition: SnapshotAcquisition, options: PresentationOptions ) -> SnapshotBackendCapture { - project(acquisition, options: options, projection: .raw) + project(acquisition.nodes, acquisition: acquisition, options: options, projection: .raw) } /// Derives the one acquisition-facing view of a request, so no backend re-reads @@ -206,15 +241,16 @@ enum SnapshotPresentation { } private static func project( - _ acquisition: SnapshotAcquisition, + _ projectionNodes: [RawAXNode], + acquisition: SnapshotAcquisition, options: PresentationOptions, projection: CaptureHint.Projection ) -> SnapshotBackendCapture { - let scopedRawNodes = applyScope(to: acquisition.nodes, options: options, projection: projection) + let scopedRawNodes = applyScope(to: projectionNodes, options: options, projection: projection) let nodes = presentedNodes(from: scopedRawNodes, projection: projection) let qualityPayload: DataPayload? = SnapshotScopePolicy.isActive(options.scope) ? DataPayload( - nodes: presentedNodes(from: acquisition.nodes, projection: projection), + nodes: presentedNodes(from: projectionNodes, projection: projection), truncated: acquisition.truncated ) : nil @@ -229,6 +265,156 @@ enum SnapshotPresentation { ) } + /// A decoration this small carries no information a user could act on: sub-pixel separators and + /// spacer views. Content-carrying nodes are exempt regardless of size. + private static let negligibleDecorationTolerance = 1.0 + private static let visibilityExemptCarrierTypes: Set = ["Application", "Window"] + + /// The regular projection's clip fold -- the ONE visibility interpreter for every backend + /// (#1797). Consumes reported facts (frames at raw traversal depth), drops out-of-clip and + /// negligible-decoration nodes, reparents survivors to their nearest kept ancestor with + /// collapsed depth, books scroll hints onto the anchors that hide the dropped content, and + /// narrows `hittable` to the clip: a node outside its clip -- or with no geometry at all -- + /// is never actionable, whatever the backend reported. + static func foldRegularVisibility( + _ nodes: [RawAXNode], + viewport: CGRect, + interactiveOnly: Bool, + policy: SnapshotFoldPolicy + ) -> [RawAXNode] { + struct BranchState { + let cursor: FlatSnapshotProjectionCursor + let anchor: (index: Int, rect: CGRect)? + let keptIndex: Int? + let keptDepth: Int + } + + var hasChildren = [Bool](repeating: false, count: nodes.count) + for node in nodes { + if let parentIndex = node.parentIndex, parentIndex >= 0, parentIndex < nodes.count { + hasChildren[parentIndex] = true + } + } + + var states = [BranchState?](repeating: nil, count: nodes.count) + var kept: [RawAXNode] = [] + var hints: [Int: (above: Bool, below: Bool)] = [:] + + for (offset, node) in nodes.enumerated() { + let parentState = node.parentIndex.flatMap { states[$0] } + let parentCursor = parentState?.cursor ?? .root + let parentAnchor = parentState?.anchor + let rect = CGRect( + x: node.rect.x, y: node.rect.y, width: node.rect.width, height: node.rect.height + ) + let intersects = RunnerTests.isVisibleInRegularSnapshot( + rect, + viewport: viewport, + scrollContainerAnchor: parentAnchor + ) + + let transition: FlatSnapshotProjectionTransition + switch policy { + case .cursorProjected: + transition = RunnerTests.flatSnapshotProjectionTransition( + frame: rect, + intersectsViewportAndScrollClip: intersects, + typeName: node.type, + hasChildren: hasChildren[offset], + cursor: parentCursor + ) + case .plainViewport: + transition = FlatSnapshotProjectionTransition( + decision: FlatSnapshotProjectionDecision( + presentationVisible: intersects, + descendants: .root + ), + hiddenContentFrame: !intersects ? rect : nil + ) + } + + let negligibleDecoration = policy == .cursorProjected + && node.parentIndex != nil + && !hasSemanticContent(node) + && (rect.isEmpty + || rect.width <= Self.negligibleDecorationTolerance + || rect.height <= Self.negligibleDecorationTolerance) + let visible = transition.decision.presentationVisible && !negligibleDecoration + + let include: Bool + if node.parentIndex == nil { + include = true + } else { + switch policy { + case .cursorProjected: + include = visibilityExemptCarrierTypes.contains(node.type) || visible + case .plainViewport: + if interactiveOnly && !visible && node.type != "Application" { + include = false + } else { + include = visibilityExemptCarrierTypes.contains(node.type) || visible + } + } + } + + if let hiddenFrame = transition.hiddenContentFrame, let parentAnchor { + RunnerTests.rememberHiddenContentHint( + for: hiddenFrame, + relativeTo: parentAnchor, + hints: &hints + ) + } + + var keptIndex = parentState?.keptIndex + var keptDepth = parentState?.keptDepth ?? -1 + if include { + let outIndex = kept.count + let outDepth = keptDepth + 1 + kept.append( + RawAXNode( + index: outIndex, + type: node.type, + label: node.label, + identifier: node.identifier, + value: node.value, + rect: node.rect, + enabled: node.enabled, + focused: node.focused, + selected: node.selected, + hittable: node.hittable && intersects, + depth: outDepth, + parentIndex: keptIndex, + hiddenContentAbove: node.hiddenContentAbove, + hiddenContentBelow: node.hiddenContentBelow, + actions: node.actions + ) + ) + keptIndex = outIndex + keptDepth = outDepth + } + + var anchor = parentAnchor + if include, + let newAnchor = RunnerTests.scrollContainerAnchor( + forTypeName: node.type, + hasChildren: hasChildren[offset], + visible: intersects, + frame: rect, + nodeIndex: keptIndex + ) + { + anchor = newAnchor + } + states[offset] = BranchState( + cursor: transition.decision.descendants, + anchor: anchor, + keptIndex: keptIndex, + keptDepth: keptDepth + ) + } + return RunnerTests.applyHiddenContentHints(hints, to: kept) + } + /// Explicit carve-out for selector queries and system-modal reads that intentionally return one /// already-resolved element instead of traversing a snapshot backend. static func singleElementRead(_ node: RawAXNode) -> PresentedNode { diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentationTests.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentationTests.swift index ae2a858931..0529781640 100644 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentationTests.swift +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentationTests.swift @@ -245,6 +245,153 @@ extension RunnerTests { XCTAssertNil(RunnerTests.sparsePayloadReason(try XCTUnwrap(missing.qualityPayload))) } + private static func foldNode( + _ index: Int, + type: String, + label: String? = nil, + rect: SnapshotRect, + hittable: Bool = false, + depth: Int, + parentIndex: Int? + ) -> RawAXNode { + RawAXNode( + index: index, + type: type, + label: label, + identifier: nil, + value: nil, + rect: rect, + enabled: true, + focused: nil, + selected: nil, + hittable: hittable, + depth: depth, + parentIndex: parentIndex, + hiddenContentAbove: nil, + hiddenContentBelow: nil + ) + } + + private static func presentedRegular( + _ nodes: [RawAXNode], + viewport: CGRect, + interactiveOnly: Bool = false, + policy: SnapshotFoldPolicy = .cursorProjected + ) -> [PresentedNode] { + let hint = CaptureHint( + projection: .regular, depth: nil, interactiveOnly: interactiveOnly, customActions: false) + return SnapshotPresentation.presentRegular( + SnapshotAcquisition( + hint: hint, nodes: nodes, truncated: false, effectiveDepth: nil, viewport: viewport), + options: PresentationOptions( + interactiveOnly: interactiveOnly, depth: nil, scope: nil, raw: false), + policy: policy + ).payload.nodes ?? [] + } + + /// The clip fold is presentation's, fed by any backend's reported facts: out-of-clip rows are + /// dropped, owned containers hide their clamped descendants, the scroll anchor books the hint, + /// and survivors are reparented with collapsed depth. Non-vacuity: bypassing + /// `foldRegularVisibility` in `presentRegular` fails every assertion below except the raw count. + func testRegularFoldClipsScrollOverflowReparentsAndBooksHints() throws { + let nodes = [ + Self.foldNode(0, type: "Application", label: "App", + rect: SnapshotRect(x: 0, y: 0, width: 402, height: 874), depth: 0, parentIndex: nil), + Self.foldNode(1, type: "ScrollView", + rect: SnapshotRect(x: 0, y: 96, width: 402, height: 700), depth: 1, parentIndex: 0), + Self.foldNode(2, type: "Cell", label: "Visible row", + rect: SnapshotRect(x: 0, y: 120, width: 402, height: 52), hittable: true, + depth: 2, parentIndex: 1), + Self.foldNode(3, type: "StaticText", label: "Detail", + rect: SnapshotRect(x: 16, y: 130, width: 200, height: 20), depth: 3, parentIndex: 2), + Self.foldNode(4, type: "Cell", label: "Offscreen row", + rect: SnapshotRect(x: 0, y: 900, width: 402, height: 52), hittable: true, + depth: 2, parentIndex: 1), + // Element reports the offscreen row's child clamped back inside the viewport; the owned + // container's projection is what keeps it hidden (#1784's leak, now fold-owned). + Self.foldNode(5, type: "StaticText", label: "Clamped child", + rect: SnapshotRect(x: 16, y: 96, width: 100, height: 20), depth: 3, parentIndex: 4), + ] + let presented = Self.presentedRegular( + nodes, viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) + + XCTAssertEqual(presented.map(\.type), ["Application", "ScrollView", "Cell", "StaticText"]) + XCTAssertEqual(presented.compactMap(\.label), ["App", "Visible row", "Detail"]) + XCTAssertEqual(presented.map(\.depth), [0, 1, 2, 3]) + XCTAssertEqual(presented.map(\.parentIndex), [nil, 0, 1, 2]) + XCTAssertEqual(presented.first { $0.type == "ScrollView" }?.hiddenContentBelow, true) + } + + /// Application/Window carriers survive the fold off-clip (they carry geometry other layers + /// need), but nothing outside its clip is ever hittable, whatever the backend reported. + func testRegularFoldKeepsWindowCarriersButNeverHittableOutsideClip() throws { + let nodes = [ + Self.foldNode(0, type: "Application", label: "App", + rect: SnapshotRect(x: 0, y: 0, width: 402, height: 874), depth: 0, parentIndex: nil), + Self.foldNode(1, type: "Window", label: "Second screen", + rect: SnapshotRect(x: 402, y: 0, width: 402, height: 874), hittable: true, + depth: 1, parentIndex: 0), + Self.foldNode(2, type: "Button", label: "Gone", + rect: SnapshotRect(x: 500, y: 100, width: 100, height: 44), hittable: true, + depth: 2, parentIndex: 1), + ] + let presented = Self.presentedRegular( + nodes, viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) + + XCTAssertEqual(presented.map(\.type), ["Application", "Window"]) + XCTAssertEqual(presented.last?.hittable, false) + XCTAssertFalse(presented.contains { $0.label == "Gone" }) + } + + /// The sub-pixel decoration rule is fold-owned and backend-neutral: a contentless separator is + /// dropped even when its type is interactive (eligibility alone would keep it), while + /// content-carrying degenerate and geometryless nodes survive -- never hittable. + func testRegularFoldDropsSubPixelContentlessDecorationOnEveryBackend() throws { + let nodes = [ + Self.foldNode(0, type: "Application", label: "App", + rect: SnapshotRect(x: 0, y: 0, width: 402, height: 874), depth: 0, parentIndex: nil), + Self.foldNode(1, type: "Button", + rect: SnapshotRect(x: 0, y: 100, width: 402, height: 1), hittable: true, + depth: 1, parentIndex: 0), + Self.foldNode(2, type: "StaticText", label: "Hairline caption", + rect: SnapshotRect(x: 0, y: 200, width: 402, height: 1), depth: 1, parentIndex: 0), + Self.foldNode(3, type: "StaticText", label: "Frameless semantics", + rect: SnapshotRect(x: 0, y: 0, width: 0, height: 0), hittable: true, + depth: 1, parentIndex: 0), + ] + let presented = Self.presentedRegular( + nodes, viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) + + XCTAssertEqual( + presented.compactMap(\.label), ["App", "Hairline caption", "Frameless semantics"]) + XCTAssertFalse(presented.contains { $0.type == "Button" }) + XCTAssertEqual(presented.first { $0.label == "Frameless semantics" }?.hittable, false) + } + + /// The platform split is a policy input, not a backend exception: plain-viewport platforms have + /// no ancestor cursor (a clamped child of an offscreen row stays visible) and drop offscreen + /// Window carriers only for interactive-only requests. + func testPlainViewportPolicyFoldsWithoutAncestorCursor() throws { + let nodes = [ + Self.foldNode(0, type: "Application", label: "App", + rect: SnapshotRect(x: 0, y: 0, width: 800, height: 600), depth: 0, parentIndex: nil), + Self.foldNode(1, type: "Window", label: "Offscreen window", + rect: SnapshotRect(x: 900, y: 0, width: 800, height: 600), depth: 1, parentIndex: 0), + Self.foldNode(2, type: "Cell", label: "Offscreen row", + rect: SnapshotRect(x: 0, y: 900, width: 800, height: 52), depth: 1, parentIndex: 0), + Self.foldNode(3, type: "StaticText", label: "Clamped child", + rect: SnapshotRect(x: 16, y: 100, width: 100, height: 20), depth: 2, parentIndex: 2), + ] + let viewport = CGRect(x: 0, y: 0, width: 800, height: 600) + + let regular = Self.presentedRegular(nodes, viewport: viewport, policy: .plainViewport) + XCTAssertEqual(regular.compactMap(\.label), ["App", "Offscreen window", "Clamped child"]) + + let interactive = Self.presentedRegular( + nodes, viewport: viewport, interactiveOnly: true, policy: .plainViewport) + XCTAssertFalse(interactive.contains { $0.label == "Offscreen window" }) + } + /// #1797 D4: a backend that answers a `--raw` request with a regular capture (or the reverse) /// loses its tier instead of having its output relabeled. Non-vacuity: dropping the projection /// guard makes both `XCTAssertNil` assertions fail, and the raw projection then returns the diff --git a/docs/adr/0004-ios-snapshot-backend-strategy.md b/docs/adr/0004-ios-snapshot-backend-strategy.md index 4d6a723022..706a9f053b 100644 --- a/docs/adr/0004-ios-snapshot-backend-strategy.md +++ b/docs/adr/0004-ios-snapshot-backend-strategy.md @@ -143,6 +143,21 @@ compares the requested projection with the hint the acquisition was captured und tier with a structured `IOS_SNAPSHOT_PROJECTION_MISMATCH` failure instead of presenting it under the requested label. +The fourth semantic layer moves the clip fold itself into presentation. Acquisition backends are +fact serializers: every traversed node is emitted at raw traversal depth with its reported frame, +and `SnapshotAcquisition` carries the viewport. `presentRegular` runs the one visibility +interpreter for every backend — viewport ∩ scroll-container clip, the ancestor projection cursor +(an out-of-clip Cell or scroll container hides descendants whose clamped frames would otherwise +leak back into the viewport), the sub-pixel decoration rule, hidden-content hints booked onto +scroll anchors, and reparenting of survivors with collapsed depth. The fold also narrows the +emitted `hittable` to the clip: nothing outside its clip, and nothing without geometry, is ever +hittable regardless of what a backend reported. Platform differences are a `SnapshotFoldPolicy` +input to the shared algorithm (iOS cursor-projected; macOS/tvOS plain viewport intersection), +never a backend exception. The remaining acquisition-side narrowings are declared: the +traversal-depth budget cut, the flat query sweep's frameless-element drop (a flat query has no +hierarchy for geometryless semantics to attach to), and the private-AX bridge's device-side node +cap. + Declared residue: a regular-projection `--depth` request still cuts the traversal at that depth, while regular presentation emits collapsed depth. A node whose presented depth would be within the limit can therefore be dropped when structural wrappers put it deeper in the raw tree. The cut is From e4b19df6c955ee6c91c8ca70798c24b0f76269ce Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20Pierzcha=C5=82a?= Date: Fri, 21 Aug 2026 09:10:00 +0200 Subject: [PATCH 2/2] refactor(ios): isolate snapshot visibility fold --- .github/workflows/ios.yml | 2 +- .../RunnerTests+FlatSnapshotFiltering.swift | 274 ---------------- .../RunnerTests+PrivateAXElementTypes.swift | 28 ++ .../RunnerTests+PrivateAXPresentation.swift | 3 +- .../RunnerTests+Snapshot.swift | 78 ----- .../RunnerTests+SnapshotPresentation.swift | 191 +----------- ...unnerTests+SnapshotPresentationTests.swift | 200 +++--------- .../RunnerTests+SnapshotVisibilityFold.swift | 294 ++++++++++++++++++ ...nerTests+SnapshotVisibilityFoldTests.swift | 132 ++++++++ 9 files changed, 511 insertions(+), 691 deletions(-) delete mode 100644 apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+FlatSnapshotFiltering.swift create mode 100644 apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXElementTypes.swift create mode 100644 apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotVisibilityFold.swift create mode 100644 apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotVisibilityFoldTests.swift diff --git a/.github/workflows/ios.yml b/.github/workflows/ios.yml index c0f8b34aae..6f2806b581 100644 --- a/.github/workflows/ios.yml +++ b/.github/workflows/ios.yml @@ -159,8 +159,8 @@ jobs: -only-testing:AgentDeviceRunnerUITests/RunnerTests/testRegularFoldKeepsWindowCarriersButNeverHittableOutsideClip \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testRegularFoldDropsSubPixelContentlessDecorationOnEveryBackend \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testPlainViewportPolicyFoldsWithoutAncestorCursor \ + -only-testing:AgentDeviceRunnerUITests/RunnerTests/testRegularPresentationRoutesThroughVisibilityFold \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotScopePolicyMatchesGoldenParityTable \ - -only-testing:AgentDeviceRunnerUITests/RunnerTests/testFlatSnapshotProjectionMatchesElementReverseScrollCapture \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testScrollContainerTypeNamesMatchElementTypeSet \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testPrivateAXDepthLimitedRequiresEveryFrontierResolved \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testDeepExtensionCountsMissedFrontiers \ diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+FlatSnapshotFiltering.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+FlatSnapshotFiltering.swift deleted file mode 100644 index 982584341a..0000000000 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+FlatSnapshotFiltering.swift +++ /dev/null @@ -1,274 +0,0 @@ -import XCTest - -enum FlatSnapshotGeometry { - case geometryless - case framed(intersectsViewportAndScrollClip: Bool) -} - -enum FlatSnapshotDescendantVisibility: Equatable { - case independent - case owned -} - -struct FlatSnapshotProjectionCursor { - static let root = FlatSnapshotProjectionCursor(ancestorProjectedOut: false) - - private let ancestorProjectedOut: Bool - - var isProjectedOut: Bool { ancestorProjectedOut } - - func project( - geometry: FlatSnapshotGeometry, - descendantVisibility: FlatSnapshotDescendantVisibility - ) -> FlatSnapshotProjectionDecision { - let nodeProjectedOut: Bool - switch geometry { - case .geometryless: - nodeProjectedOut = false - case .framed(let intersectsViewportAndScrollClip): - nodeProjectedOut = !intersectsViewportAndScrollClip - } - let presentationVisible = !ancestorProjectedOut && !nodeProjectedOut - let descendantsProjectedOut = ancestorProjectedOut - || (nodeProjectedOut && descendantVisibility == .owned) - return FlatSnapshotProjectionDecision( - presentationVisible: presentationVisible, - descendants: FlatSnapshotProjectionCursor( - ancestorProjectedOut: descendantsProjectedOut - ) - ) - } -} - -struct FlatSnapshotProjectionDecision { - let presentationVisible: Bool - let descendants: FlatSnapshotProjectionCursor -} - -struct FlatSnapshotProjectionTransition { - let decision: FlatSnapshotProjectionDecision - let hiddenContentFrame: CGRect? -} - -extension RunnerTests { - static func flatSnapshotGeometry( - frame: CGRect, - intersectsViewportAndScrollClip: Bool - ) -> FlatSnapshotGeometry { - if frame.isNull || frame.isEmpty { return .geometryless } - return .framed(intersectsViewportAndScrollClip: intersectsViewportAndScrollClip) - } - - /// Wire-name variant: the presentation fold works on `RawAXNode.type` strings, the one node - /// vocabulary every backend already shares (#1797 — backend-neutral by construction). - static func flatSnapshotDescendantVisibility( - typeName: String?, - hasChildren: Bool - ) -> FlatSnapshotDescendantVisibility { - guard hasChildren, let typeName else { return .independent } - if typeName == "Cell" || Self.scrollContainerTypeNames.contains(typeName) { - return .owned - } - return .independent - } - - static func flatSnapshotProjectionTransition( - frame: CGRect, - intersectsViewportAndScrollClip: Bool, - typeName: String?, - hasChildren: Bool, - cursor: FlatSnapshotProjectionCursor - ) -> FlatSnapshotProjectionTransition { - let hasFrame = !frame.isNull && !frame.isEmpty - return FlatSnapshotProjectionTransition( - decision: cursor.project( - geometry: flatSnapshotGeometry( - frame: frame, - intersectsViewportAndScrollClip: intersectsViewportAndScrollClip - ), - descendantVisibility: flatSnapshotDescendantVisibility( - typeName: typeName, - hasChildren: hasChildren - ) - ), - hiddenContentFrame: !cursor.isProjectedOut && hasFrame - && !intersectsViewportAndScrollClip ? frame : nil - ) - } - - func privateAXInteractiveCandidate(rawElementType: Int) -> Bool { - guard let type = flatSnapshotElementType(rawElementType: rawElementType) else { - return false - } - return interactiveTypes.contains(type) || Self.scrollContainerTypes.contains(type) - } - - func flatSnapshotElementType(rawElementType: Int) -> XCUIElement.ElementType? { - guard let raw = UInt(exactly: rawElementType), - let type = XCUIElement.ElementType(rawValue: raw) - else { - return nil - } - return type - } - -#if AGENT_DEVICE_RUNNER_UNIT_TESTS - /// The fold's string vocabulary must track the XCUIElement type sets it mirrors: a scroll - /// container renamed or added in one place but not the other silently changes which subtrees - /// own their descendants' visibility. - func testScrollContainerTypeNamesMatchElementTypeSet() { - XCTAssertEqual( - Self.scrollContainerTypeNames, - Set(Self.scrollContainerTypes.map(elementTypeName)) - ) - XCTAssertEqual(elementTypeName(.cell), "Cell") - } - - func testFlatSnapshotProjectionMatchesElementReverseScrollCapture() { - struct FixtureNode { - let label: String - let type: XCUIElement.ElementType - let frame: CGRect - let children: [FixtureNode] - } - - let tableFrame = CGRect(x: 0, y: 152.333, width: 402, height: 659.667) - let visibleRowFrame = CGRect(x: 0, y: 251.333, width: 402, height: 43.667) - let offscreenThemeRowFrame = CGRect(x: 0, y: 2044.333, width: 402, height: 44) - let clampedThemeFrame = CGRect(x: 0, y: 152.333, width: 53, height: 20.333) - let geometrylessFrame = CGRect.zero - var visibleLabels: [String] = [] - var hints: [Int: (above: Bool, below: Bool)] = [:] - - func visit( - _ node: FixtureNode, - cursor: FlatSnapshotProjectionCursor, - scrollAnchor: (index: Int, rect: CGRect)? - ) { - let intersects = Self.isVisibleInRegularSnapshot( - node.frame, - viewport: CGRect(x: 0, y: 0, width: 402, height: 874), - scrollContainerAnchor: scrollAnchor - ) - let transition = Self.flatSnapshotProjectionTransition( - frame: node.frame, - intersectsViewportAndScrollClip: intersects, - typeName: elementTypeName(node.type), - hasChildren: !node.children.isEmpty, - cursor: cursor - ) - if transition.decision.presentationVisible { visibleLabels.append(node.label) } - if let hiddenFrame = transition.hiddenContentFrame, let scrollAnchor { - Self.rememberHiddenContentHint( - for: hiddenFrame, - relativeTo: scrollAnchor, - hints: &hints - ) - } - let nextScrollAnchor = Self.scrollContainerTypes.contains(node.type) - && transition.decision.presentationVisible - ? (index: 0, rect: node.frame) : scrollAnchor - for child in node.children { - visit( - child, - cursor: transition.decision.descendants, - scrollAnchor: nextScrollAnchor - ) - } - } - - let fixture = FixtureNode( - label: "Table", - type: .table, - frame: tableFrame, - children: [ - FixtureNode( - label: "Profile Picture", - type: .cell, - frame: visibleRowFrame, - children: [ - FixtureNode( - label: "Profile semantics", - type: .staticText, - frame: geometrylessFrame, - children: [ - FixtureNode( - label: "Profile detail", - type: .staticText, - frame: visibleRowFrame, - children: [] - ) - ] - ) - ] - ), - // Element reports this row at y=2044 but clamps Theme and Auto to the table's top edge. - FixtureNode( - label: "Theme row", - type: .cell, - frame: offscreenThemeRowFrame, - children: [ - FixtureNode( - label: "Theme", - type: .staticText, - frame: clampedThemeFrame, - children: [] - ), - FixtureNode( - label: "Auto", - type: .staticText, - frame: clampedThemeFrame, - children: [] - ) - ] - ), - FixtureNode( - label: "Offscreen wrapper", - type: .other, - frame: offscreenThemeRowFrame, - children: [ - FixtureNode( - label: "Visible overlay", - type: .staticText, - frame: visibleRowFrame, - children: [] - ) - ] - ) - ] - ) - visit(fixture, cursor: .root, scrollAnchor: nil) - - XCTAssertEqual( - visibleLabels, - ["Table", "Profile Picture", "Profile semantics", "Profile detail", "Visible overlay"] - ) - XCTAssertEqual(hints.count, 1) - XCTAssertEqual(hints[0]?.above, false) - XCTAssertEqual(hints[0]?.below, true) - - let scopedRoot = Self.flatSnapshotProjectionTransition( - frame: offscreenThemeRowFrame, - intersectsViewportAndScrollClip: false, - typeName: "Cell", - hasChildren: true, - cursor: .root - ) - let scopedClampedChild = Self.flatSnapshotProjectionTransition( - frame: clampedThemeFrame, - intersectsViewportAndScrollClip: true, - typeName: "StaticText", - hasChildren: false, - cursor: scopedRoot.decision.descendants - ) - XCTAssertFalse(scopedClampedChild.decision.presentationVisible) - } - - func testPrivateAXInteractiveCandidatesPreserveBackendInputs() { - XCTAssertTrue( - privateAXInteractiveCandidate(rawElementType: Int(XCUIElement.ElementType.scrollView.rawValue)), - "private AX marks scroll containers as interactive candidates" - ) - } -#endif -} diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXElementTypes.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXElementTypes.swift new file mode 100644 index 0000000000..b00728df65 --- /dev/null +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXElementTypes.swift @@ -0,0 +1,28 @@ +import XCTest + +extension RunnerTests { + func privateAXInteractiveCandidate(rawElementType: Int) -> Bool { + guard let type = privateAXElementType(rawElementType: rawElementType) else { + return false + } + return interactiveTypes.contains(type) || Self.scrollContainerTypes.contains(type) + } + + func privateAXElementType(rawElementType: Int) -> XCUIElement.ElementType? { + guard let raw = UInt(exactly: rawElementType), + let type = XCUIElement.ElementType(rawValue: raw) + else { + return nil + } + return type + } + +#if AGENT_DEVICE_RUNNER_UNIT_TESTS + func testPrivateAXInteractiveCandidatesPreserveBackendInputs() { + XCTAssertTrue( + privateAXInteractiveCandidate(rawElementType: Int(XCUIElement.ElementType.scrollView.rawValue)), + "private AX marks scroll containers as interactive candidates" + ) + } +#endif +} diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXPresentation.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXPresentation.swift index 390e035f05..51295e180b 100644 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXPresentation.swift +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXPresentation.swift @@ -15,7 +15,6 @@ struct PrivateAXFields { let actions: [String]? let children: [[String: Any]] - var hasSemanticContent: Bool { [label, identifier, value].contains { !$0.isEmpty } } var hasFrame: Bool { !rect.isNull && !rect.isEmpty } } @@ -58,7 +57,7 @@ extension RunnerTests { identifier: privateAXPresentationString(raw["identifier"]), value: privateAXPresentationString(raw["value"]), rawType: rawType, - elementType: flatSnapshotElementType(rawElementType: rawType), + elementType: privateAXElementType(rawElementType: rawType), enabled: privateAXPresentationBool(raw["enabled"]) ?? true, focused: privateAXPresentationBool(raw["focused"]) == true ? true : nil, selected: privateAXPresentationBool(raw["selected"]) == true ? true : nil, diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift index 7770e7b707..9513c3a8b2 100644 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift @@ -106,14 +106,6 @@ extension RunnerTests { .table ] - /// `RawAXNode.type` vocabulary for the presentation fold; pinned to `scrollContainerTypes` - /// through `elementTypeName` by a unit test. - static let scrollContainerTypeNames: Set = [ - "CollectionView", - "ScrollView", - "Table", - ] - static let flatInteractiveFallbackBudget: TimeInterval = 1.0 // The single production entry point -- always compiled, no unit-test overload. A unit test @@ -1133,16 +1125,6 @@ extension RunnerTests { return rect.intersects(viewport) } - static func isVisibleInRegularSnapshot( - _ rect: CGRect, - viewport: CGRect, - scrollContainerAnchor: (index: Int, rect: CGRect)? - ) -> Bool { - if !isVisibleInViewport(rect, viewport) { return false } - guard let scrollContainerAnchor else { return true } - return isVisibleInViewport(rect, scrollContainerAnchor.rect) - } - private func appendCollapsedTabFallbackNodes( to nodes: inout [RawAXNode], containerSnapshot: XCUIElementSnapshot, @@ -1160,66 +1142,6 @@ extension RunnerTests { nodes.append(contentsOf: fallbackNodes) } - static func scrollContainerAnchor( - forTypeName typeName: String, - hasChildren: Bool, - visible: Bool, - frame: CGRect, - nodeIndex: Int? - ) -> (index: Int, rect: CGRect)? { - guard let nodeIndex else { return nil } - guard visible, hasChildren, Self.scrollContainerTypeNames.contains(typeName) else { - return nil - } - return (nodeIndex, frame) - } - - static func rememberHiddenContentHint( - for frame: CGRect, - relativeTo scrollContainerAnchor: (index: Int, rect: CGRect), - hints: inout [Int: (above: Bool, below: Bool)] - ) { - if frame.isNull || frame.isEmpty { return } - var hint = hints[scrollContainerAnchor.index] ?? (above: false, below: false) - if frame.maxY <= scrollContainerAnchor.rect.minY { - hint.above = true - } else if frame.minY >= scrollContainerAnchor.rect.maxY { - hint.below = true - } else { - return - } - hints[scrollContainerAnchor.index] = hint - } - - static func applyHiddenContentHints( - _ hints: [Int: (above: Bool, below: Bool)], - to nodes: [RawAXNode] - ) -> [RawAXNode] { - if hints.isEmpty { return nodes } - return nodes.map { node in - guard let hint = hints[node.index] else { return node } - let hiddenContentAbove: Bool? = (node.hiddenContentAbove == true || hint.above) ? true : nil - let hiddenContentBelow: Bool? = (node.hiddenContentBelow == true || hint.below) ? true : nil - return RawAXNode( - index: node.index, - type: node.type, - label: node.label, - identifier: node.identifier, - value: node.value, - rect: node.rect, - enabled: node.enabled, - focused: node.focused, - selected: node.selected, - hittable: node.hittable, - depth: node.depth, - parentIndex: node.parentIndex, - hiddenContentAbove: hiddenContentAbove, - hiddenContentBelow: hiddenContentBelow, - actions: node.actions - ) - } - } - private func collapsedTabFallbackNodes( for containerSnapshot: XCUIElementSnapshot, resolveElements: () -> [XCUIElement], diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentation.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentation.swift index 6af16a2204..6f9538cbe4 100644 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentation.swift +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentation.swift @@ -20,6 +20,12 @@ struct RawAXNode { let hiddenContentAbove: Bool? let hiddenContentBelow: Bool? var actions: [String]? = nil + + var hasSemanticContent: Bool { + [label, identifier, value].contains { + !($0?.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty ?? true) + } + } } /// The acquisition-facing view of a snapshot request, derived once by @@ -71,27 +77,7 @@ struct SnapshotAcquisition { var customActions: SnapshotCustomActionCoverage? = nil /// Viewport the regular projection's clip fold runs against. `.infinite` disables the fold -- /// legitimate only for raw acquisitions and depth-0 probes, where no fold applies. - var viewport: CGRect = .infinite -} - -/// Platform policy for the regular projection's visibility fold (#1797: macOS is a policy input -/// to the shared algorithm, never a backend exception). -enum SnapshotFoldPolicy { - /// iOS: ancestor projection cursor -- an out-of-clip Cell or scroll container hides the - /// descendants whose clamped frames would otherwise leak back into the viewport -- plus the - /// sub-pixel decoration rule. - case cursorProjected - /// macOS / tvOS: plain viewport intersection per node, no ancestor cursor, and interactive-only - /// requests additionally drop out-of-viewport Window carriers. - case plainViewport - - static var platformDefault: SnapshotFoldPolicy { - #if os(iOS) - return .cursorProjected - #else - return .plainViewport - #endif - } + let viewport: CGRect } /// The only snapshot node shape accepted by response payload assembly. @@ -199,10 +185,10 @@ enum SnapshotPresentation { static func presentRegular( _ acquisition: SnapshotAcquisition, options: PresentationOptions, - policy: SnapshotFoldPolicy = .platformDefault + policy: SnapshotVisibilityFold.Policy = .platformDefault ) -> SnapshotBackendCapture { project( - foldRegularVisibility( + SnapshotVisibilityFold.fold( acquisition.nodes, viewport: acquisition.viewport, interactiveOnly: options.interactiveOnly, @@ -265,156 +251,6 @@ enum SnapshotPresentation { ) } - /// A decoration this small carries no information a user could act on: sub-pixel separators and - /// spacer views. Content-carrying nodes are exempt regardless of size. - private static let negligibleDecorationTolerance = 1.0 - private static let visibilityExemptCarrierTypes: Set = ["Application", "Window"] - - /// The regular projection's clip fold -- the ONE visibility interpreter for every backend - /// (#1797). Consumes reported facts (frames at raw traversal depth), drops out-of-clip and - /// negligible-decoration nodes, reparents survivors to their nearest kept ancestor with - /// collapsed depth, books scroll hints onto the anchors that hide the dropped content, and - /// narrows `hittable` to the clip: a node outside its clip -- or with no geometry at all -- - /// is never actionable, whatever the backend reported. - static func foldRegularVisibility( - _ nodes: [RawAXNode], - viewport: CGRect, - interactiveOnly: Bool, - policy: SnapshotFoldPolicy - ) -> [RawAXNode] { - struct BranchState { - let cursor: FlatSnapshotProjectionCursor - let anchor: (index: Int, rect: CGRect)? - let keptIndex: Int? - let keptDepth: Int - } - - var hasChildren = [Bool](repeating: false, count: nodes.count) - for node in nodes { - if let parentIndex = node.parentIndex, parentIndex >= 0, parentIndex < nodes.count { - hasChildren[parentIndex] = true - } - } - - var states = [BranchState?](repeating: nil, count: nodes.count) - var kept: [RawAXNode] = [] - var hints: [Int: (above: Bool, below: Bool)] = [:] - - for (offset, node) in nodes.enumerated() { - let parentState = node.parentIndex.flatMap { states[$0] } - let parentCursor = parentState?.cursor ?? .root - let parentAnchor = parentState?.anchor - let rect = CGRect( - x: node.rect.x, y: node.rect.y, width: node.rect.width, height: node.rect.height - ) - let intersects = RunnerTests.isVisibleInRegularSnapshot( - rect, - viewport: viewport, - scrollContainerAnchor: parentAnchor - ) - - let transition: FlatSnapshotProjectionTransition - switch policy { - case .cursorProjected: - transition = RunnerTests.flatSnapshotProjectionTransition( - frame: rect, - intersectsViewportAndScrollClip: intersects, - typeName: node.type, - hasChildren: hasChildren[offset], - cursor: parentCursor - ) - case .plainViewport: - transition = FlatSnapshotProjectionTransition( - decision: FlatSnapshotProjectionDecision( - presentationVisible: intersects, - descendants: .root - ), - hiddenContentFrame: !intersects ? rect : nil - ) - } - - let negligibleDecoration = policy == .cursorProjected - && node.parentIndex != nil - && !hasSemanticContent(node) - && (rect.isEmpty - || rect.width <= Self.negligibleDecorationTolerance - || rect.height <= Self.negligibleDecorationTolerance) - let visible = transition.decision.presentationVisible && !negligibleDecoration - - let include: Bool - if node.parentIndex == nil { - include = true - } else { - switch policy { - case .cursorProjected: - include = visibilityExemptCarrierTypes.contains(node.type) || visible - case .plainViewport: - if interactiveOnly && !visible && node.type != "Application" { - include = false - } else { - include = visibilityExemptCarrierTypes.contains(node.type) || visible - } - } - } - - if let hiddenFrame = transition.hiddenContentFrame, let parentAnchor { - RunnerTests.rememberHiddenContentHint( - for: hiddenFrame, - relativeTo: parentAnchor, - hints: &hints - ) - } - - var keptIndex = parentState?.keptIndex - var keptDepth = parentState?.keptDepth ?? -1 - if include { - let outIndex = kept.count - let outDepth = keptDepth + 1 - kept.append( - RawAXNode( - index: outIndex, - type: node.type, - label: node.label, - identifier: node.identifier, - value: node.value, - rect: node.rect, - enabled: node.enabled, - focused: node.focused, - selected: node.selected, - hittable: node.hittable && intersects, - depth: outDepth, - parentIndex: keptIndex, - hiddenContentAbove: node.hiddenContentAbove, - hiddenContentBelow: node.hiddenContentBelow, - actions: node.actions - ) - ) - keptIndex = outIndex - keptDepth = outDepth - } - - var anchor = parentAnchor - if include, - let newAnchor = RunnerTests.scrollContainerAnchor( - forTypeName: node.type, - hasChildren: hasChildren[offset], - visible: intersects, - frame: rect, - nodeIndex: keptIndex - ) - { - anchor = newAnchor - } - states[offset] = BranchState( - cursor: transition.decision.descendants, - anchor: anchor, - keptIndex: keptIndex, - keptDepth: keptDepth - ) - } - return RunnerTests.applyHiddenContentHints(hints, to: kept) - } - /// Explicit carve-out for selector queries and system-modal reads that intentionally return one /// already-resolved element instead of traversing a snapshot backend. static func singleElementRead(_ node: RawAXNode) -> PresentedNode { @@ -517,13 +353,6 @@ enum SnapshotPresentation { // The top-level carrier owns viewport geometry and must survive even for query-sweep's // deliberately unlabeled synthetic Application node. if node.parentIndex == nil { return true } - return eligibleInteractiveTypes.contains(node.type) || hasSemanticContent(node) - } - - private static func hasSemanticContent(_ node: RawAXNode) -> Bool { - [node.label, node.identifier, node.value].contains { value in - guard let value else { return false } - return !value.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty - } + return eligibleInteractiveTypes.contains(node.type) || node.hasSemanticContent } } diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentationTests.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentationTests.swift index 0529781640..0bae0f2247 100644 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentationTests.swift +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentationTests.swift @@ -37,7 +37,8 @@ extension RunnerTests { candidates: 2, truncated: 0, blocked: false - ) + ), + viewport: .infinite ), options: PresentationOptions( interactiveOnly: true, @@ -108,7 +109,8 @@ extension RunnerTests { SnapshotAcquisition( hint: CaptureHint( projection: .regular, depth: nil, interactiveOnly: false, customActions: false), - nodes: acquired, truncated: false, effectiveDepth: nil), + nodes: acquired, truncated: false, effectiveDepth: nil, + viewport: CGRect(x: 0, y: 0, width: 1_000, height: 1_000)), options: PresentationOptions(interactiveOnly: false, depth: nil, scope: nil, raw: false) ).payload.nodes ) @@ -117,7 +119,8 @@ extension RunnerTests { SnapshotAcquisition( hint: CaptureHint( projection: .regular, depth: nil, interactiveOnly: true, customActions: false), - nodes: acquired, truncated: false, effectiveDepth: nil), + nodes: acquired, truncated: false, effectiveDepth: nil, + viewport: CGRect(x: 0, y: 0, width: 1_000, height: 1_000)), options: PresentationOptions(interactiveOnly: true, depth: nil, scope: nil, raw: false) ).payload.nodes ) @@ -139,7 +142,7 @@ extension RunnerTests { SnapshotAcquisition( hint: CaptureHint( projection: .raw, depth: nil, interactiveOnly: false, customActions: false), - nodes: acquired, truncated: false, effectiveDepth: nil), + nodes: acquired, truncated: false, effectiveDepth: nil, viewport: .infinite), options: PresentationOptions(interactiveOnly: true, depth: nil, scope: nil, raw: true) ).payload.nodes ) @@ -148,6 +151,37 @@ extension RunnerTests { XCTAssertEqual(raw.last?.parentIndex, 9) } + func testRegularPresentationRoutesThroughVisibilityFold() { + let nodes = [ + RawAXNode( + index: 0, type: "Application", label: "App", identifier: nil, value: nil, + rect: SnapshotRect(x: 0, y: 0, width: 100, height: 100), enabled: true, + focused: nil, selected: nil, hittable: false, depth: 0, parentIndex: nil, + hiddenContentAbove: nil, hiddenContentBelow: nil), + RawAXNode( + index: 1, type: "Button", label: "Outside viewport", identifier: nil, value: nil, + rect: SnapshotRect(x: 200, y: 200, width: 40, height: 40), enabled: true, + focused: nil, selected: nil, hittable: true, depth: 1, parentIndex: 0, + hiddenContentAbove: nil, hiddenContentBelow: nil), + ] + let acquisition = SnapshotAcquisition( + hint: CaptureHint( + projection: .regular, depth: nil, interactiveOnly: false, customActions: false), + nodes: nodes, + truncated: false, + effectiveDepth: nil, + viewport: CGRect(x: 0, y: 0, width: 100, height: 100) + ) + + let presented = SnapshotPresentation.presentRegular( + acquisition, + options: PresentationOptions( + interactiveOnly: false, depth: nil, scope: nil, raw: false) + ).payload.nodes + + XCTAssertEqual(presented?.compactMap(\.label), ["App"]) + } + func testSnapshotPresentationOwnsScopeAndRelativeDepth() throws { // Non-vacuity: disconnecting applyScope produces eight scope, depth, and raw-projection failures. func node( @@ -189,7 +223,8 @@ extension RunnerTests { node(6, type: "Button", label: "Outside sibling", depth: 1, parentIndex: 0), ], truncated: false, - effectiveDepth: nil + effectiveDepth: nil, + viewport: CGRect(x: 0, y: 0, width: 1_000, height: 1_000) ) let options = PresentationOptions( interactiveOnly: true, @@ -214,7 +249,8 @@ extension RunnerTests { projection: .raw, depth: nil, interactiveOnly: false, customActions: false), nodes: acquisition.nodes, truncated: false, - effectiveDepth: nil + effectiveDepth: nil, + viewport: .infinite ), options: PresentationOptions( interactiveOnly: true, @@ -245,153 +281,6 @@ extension RunnerTests { XCTAssertNil(RunnerTests.sparsePayloadReason(try XCTUnwrap(missing.qualityPayload))) } - private static func foldNode( - _ index: Int, - type: String, - label: String? = nil, - rect: SnapshotRect, - hittable: Bool = false, - depth: Int, - parentIndex: Int? - ) -> RawAXNode { - RawAXNode( - index: index, - type: type, - label: label, - identifier: nil, - value: nil, - rect: rect, - enabled: true, - focused: nil, - selected: nil, - hittable: hittable, - depth: depth, - parentIndex: parentIndex, - hiddenContentAbove: nil, - hiddenContentBelow: nil - ) - } - - private static func presentedRegular( - _ nodes: [RawAXNode], - viewport: CGRect, - interactiveOnly: Bool = false, - policy: SnapshotFoldPolicy = .cursorProjected - ) -> [PresentedNode] { - let hint = CaptureHint( - projection: .regular, depth: nil, interactiveOnly: interactiveOnly, customActions: false) - return SnapshotPresentation.presentRegular( - SnapshotAcquisition( - hint: hint, nodes: nodes, truncated: false, effectiveDepth: nil, viewport: viewport), - options: PresentationOptions( - interactiveOnly: interactiveOnly, depth: nil, scope: nil, raw: false), - policy: policy - ).payload.nodes ?? [] - } - - /// The clip fold is presentation's, fed by any backend's reported facts: out-of-clip rows are - /// dropped, owned containers hide their clamped descendants, the scroll anchor books the hint, - /// and survivors are reparented with collapsed depth. Non-vacuity: bypassing - /// `foldRegularVisibility` in `presentRegular` fails every assertion below except the raw count. - func testRegularFoldClipsScrollOverflowReparentsAndBooksHints() throws { - let nodes = [ - Self.foldNode(0, type: "Application", label: "App", - rect: SnapshotRect(x: 0, y: 0, width: 402, height: 874), depth: 0, parentIndex: nil), - Self.foldNode(1, type: "ScrollView", - rect: SnapshotRect(x: 0, y: 96, width: 402, height: 700), depth: 1, parentIndex: 0), - Self.foldNode(2, type: "Cell", label: "Visible row", - rect: SnapshotRect(x: 0, y: 120, width: 402, height: 52), hittable: true, - depth: 2, parentIndex: 1), - Self.foldNode(3, type: "StaticText", label: "Detail", - rect: SnapshotRect(x: 16, y: 130, width: 200, height: 20), depth: 3, parentIndex: 2), - Self.foldNode(4, type: "Cell", label: "Offscreen row", - rect: SnapshotRect(x: 0, y: 900, width: 402, height: 52), hittable: true, - depth: 2, parentIndex: 1), - // Element reports the offscreen row's child clamped back inside the viewport; the owned - // container's projection is what keeps it hidden (#1784's leak, now fold-owned). - Self.foldNode(5, type: "StaticText", label: "Clamped child", - rect: SnapshotRect(x: 16, y: 96, width: 100, height: 20), depth: 3, parentIndex: 4), - ] - let presented = Self.presentedRegular( - nodes, viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) - - XCTAssertEqual(presented.map(\.type), ["Application", "ScrollView", "Cell", "StaticText"]) - XCTAssertEqual(presented.compactMap(\.label), ["App", "Visible row", "Detail"]) - XCTAssertEqual(presented.map(\.depth), [0, 1, 2, 3]) - XCTAssertEqual(presented.map(\.parentIndex), [nil, 0, 1, 2]) - XCTAssertEqual(presented.first { $0.type == "ScrollView" }?.hiddenContentBelow, true) - } - - /// Application/Window carriers survive the fold off-clip (they carry geometry other layers - /// need), but nothing outside its clip is ever hittable, whatever the backend reported. - func testRegularFoldKeepsWindowCarriersButNeverHittableOutsideClip() throws { - let nodes = [ - Self.foldNode(0, type: "Application", label: "App", - rect: SnapshotRect(x: 0, y: 0, width: 402, height: 874), depth: 0, parentIndex: nil), - Self.foldNode(1, type: "Window", label: "Second screen", - rect: SnapshotRect(x: 402, y: 0, width: 402, height: 874), hittable: true, - depth: 1, parentIndex: 0), - Self.foldNode(2, type: "Button", label: "Gone", - rect: SnapshotRect(x: 500, y: 100, width: 100, height: 44), hittable: true, - depth: 2, parentIndex: 1), - ] - let presented = Self.presentedRegular( - nodes, viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) - - XCTAssertEqual(presented.map(\.type), ["Application", "Window"]) - XCTAssertEqual(presented.last?.hittable, false) - XCTAssertFalse(presented.contains { $0.label == "Gone" }) - } - - /// The sub-pixel decoration rule is fold-owned and backend-neutral: a contentless separator is - /// dropped even when its type is interactive (eligibility alone would keep it), while - /// content-carrying degenerate and geometryless nodes survive -- never hittable. - func testRegularFoldDropsSubPixelContentlessDecorationOnEveryBackend() throws { - let nodes = [ - Self.foldNode(0, type: "Application", label: "App", - rect: SnapshotRect(x: 0, y: 0, width: 402, height: 874), depth: 0, parentIndex: nil), - Self.foldNode(1, type: "Button", - rect: SnapshotRect(x: 0, y: 100, width: 402, height: 1), hittable: true, - depth: 1, parentIndex: 0), - Self.foldNode(2, type: "StaticText", label: "Hairline caption", - rect: SnapshotRect(x: 0, y: 200, width: 402, height: 1), depth: 1, parentIndex: 0), - Self.foldNode(3, type: "StaticText", label: "Frameless semantics", - rect: SnapshotRect(x: 0, y: 0, width: 0, height: 0), hittable: true, - depth: 1, parentIndex: 0), - ] - let presented = Self.presentedRegular( - nodes, viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) - - XCTAssertEqual( - presented.compactMap(\.label), ["App", "Hairline caption", "Frameless semantics"]) - XCTAssertFalse(presented.contains { $0.type == "Button" }) - XCTAssertEqual(presented.first { $0.label == "Frameless semantics" }?.hittable, false) - } - - /// The platform split is a policy input, not a backend exception: plain-viewport platforms have - /// no ancestor cursor (a clamped child of an offscreen row stays visible) and drop offscreen - /// Window carriers only for interactive-only requests. - func testPlainViewportPolicyFoldsWithoutAncestorCursor() throws { - let nodes = [ - Self.foldNode(0, type: "Application", label: "App", - rect: SnapshotRect(x: 0, y: 0, width: 800, height: 600), depth: 0, parentIndex: nil), - Self.foldNode(1, type: "Window", label: "Offscreen window", - rect: SnapshotRect(x: 900, y: 0, width: 800, height: 600), depth: 1, parentIndex: 0), - Self.foldNode(2, type: "Cell", label: "Offscreen row", - rect: SnapshotRect(x: 0, y: 900, width: 800, height: 52), depth: 1, parentIndex: 0), - Self.foldNode(3, type: "StaticText", label: "Clamped child", - rect: SnapshotRect(x: 16, y: 100, width: 100, height: 20), depth: 2, parentIndex: 2), - ] - let viewport = CGRect(x: 0, y: 0, width: 800, height: 600) - - let regular = Self.presentedRegular(nodes, viewport: viewport, policy: .plainViewport) - XCTAssertEqual(regular.compactMap(\.label), ["App", "Offscreen window", "Clamped child"]) - - let interactive = Self.presentedRegular( - nodes, viewport: viewport, interactiveOnly: true, policy: .plainViewport) - XCTAssertFalse(interactive.contains { $0.label == "Offscreen window" }) - } - /// #1797 D4: a backend that answers a `--raw` request with a regular capture (or the reverse) /// loses its tier instead of having its output relabeled. Non-vacuity: dropping the projection /// guard makes both `XCTAssertNil` assertions fail, and the raw projection then returns the @@ -417,10 +306,11 @@ extension RunnerTests { let regularAcquisition = SnapshotAcquisition( hint: SnapshotPresentation.captureHint(for: regularRequest), - nodes: nodes, truncated: false, effectiveDepth: nil) + nodes: nodes, truncated: false, effectiveDepth: nil, + viewport: CGRect(x: 0, y: 0, width: 100, height: 100)) let rawAcquisition = SnapshotAcquisition( hint: SnapshotPresentation.captureHint(for: rawRequest), - nodes: nodes, truncated: false, effectiveDepth: nil) + nodes: nodes, truncated: false, effectiveDepth: nil, viewport: .infinite) XCTAssertNil(SnapshotPresentation.present(regularAcquisition, options: rawRequest)) XCTAssertNil(SnapshotPresentation.present(rawAcquisition, options: regularRequest)) diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotVisibilityFold.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotVisibilityFold.swift new file mode 100644 index 0000000000..69dd658c40 --- /dev/null +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotVisibilityFold.swift @@ -0,0 +1,294 @@ +import Foundation + +/// The regular projection's single visibility interpreter for every snapshot backend. It applies +/// the platform clip policy, reparents survivors, and records hidden-content hints behind one pure +/// interface so acquisition backends cannot recreate only part of the projection contract. +enum SnapshotVisibilityFold { + enum Policy { + /// An out-of-clip Cell or scroll container owns its descendants' visibility. + case cursorProjected + /// Each node is intersected independently with the viewport. + case plainViewport + + static var platformDefault: Policy { + #if os(iOS) + return .cursorProjected + #else + return .plainViewport + #endif + } + } + + /// Wire-name vocabulary corresponding to XCTest's scroll-container element types. + static let scrollContainerTypeNames: Set = ["CollectionView", "ScrollView", "Table"] + + private enum Geometry { + case geometryless + case framed(intersectsClip: Bool) + } + + private enum DescendantVisibility { + case independent + case owned + } + + private struct ProjectionCursor { + static let root = ProjectionCursor(ancestorProjectedOut: false) + + private let ancestorProjectedOut: Bool + + var isProjectedOut: Bool { ancestorProjectedOut } + + func project( + geometry: Geometry, + descendantVisibility: DescendantVisibility + ) -> ProjectionDecision { + let nodeProjectedOut: Bool + switch geometry { + case .geometryless: + nodeProjectedOut = false + case .framed(let intersectsClip): + nodeProjectedOut = !intersectsClip + } + let presentationVisible = !ancestorProjectedOut && !nodeProjectedOut + let descendantsProjectedOut = ancestorProjectedOut + || (nodeProjectedOut && descendantVisibility == .owned) + return ProjectionDecision( + presentationVisible: presentationVisible, + descendants: ProjectionCursor(ancestorProjectedOut: descendantsProjectedOut) + ) + } + } + + private struct ProjectionDecision { + let presentationVisible: Bool + let descendants: ProjectionCursor + } + private struct ProjectionTransition { + let decision: ProjectionDecision + let hiddenContentFrame: CGRect? + } + private struct BranchState { + let cursor: ProjectionCursor + let anchor: (index: Int, rect: CGRect)? + let keptIndex: Int? + let keptDepth: Int + } + + private static let negligibleDecorationTolerance = 1.0 + private static let visibilityExemptCarrierTypes: Set = ["Application", "Window"] + + static func fold( + _ nodes: [RawAXNode], + viewport: CGRect, + interactiveOnly: Bool, + policy: Policy + ) -> [RawAXNode] { + var hasChildren = [Bool](repeating: false, count: nodes.count) + for node in nodes { + if let parentIndex = node.parentIndex, parentIndex >= 0, parentIndex < nodes.count { + hasChildren[parentIndex] = true + } + } + + var states = [BranchState?](repeating: nil, count: nodes.count) + var kept: [RawAXNode] = [] + var hints: [Int: (above: Bool, below: Bool)] = [:] + + for (offset, node) in nodes.enumerated() { + let parentState = node.parentIndex.flatMap { states[$0] } + let parentCursor = parentState?.cursor ?? .root + let parentAnchor = parentState?.anchor + let rect = CGRect( + x: node.rect.x, y: node.rect.y, width: node.rect.width, height: node.rect.height + ) + let intersects = intersectsClip(rect, viewport: viewport, scrollAnchor: parentAnchor) + let transition = projectionTransition( + frame: rect, + intersectsClip: intersects, + typeName: node.type, + hasChildren: hasChildren[offset], + cursor: parentCursor, + policy: policy + ) + + let negligibleDecoration = policy == .cursorProjected + && node.parentIndex != nil + && !node.hasSemanticContent + && (rect.isEmpty + || rect.width <= negligibleDecorationTolerance + || rect.height <= negligibleDecorationTolerance) + let visible = transition.decision.presentationVisible && !negligibleDecoration + let include = shouldInclude( + node, + visible: visible, + interactiveOnly: interactiveOnly, + policy: policy + ) + + if let hiddenFrame = transition.hiddenContentFrame, let parentAnchor { + rememberHiddenContentHint(for: hiddenFrame, relativeTo: parentAnchor, hints: &hints) + } + + var keptIndex = parentState?.keptIndex + var keptDepth = parentState?.keptDepth ?? -1 + if include { + let outIndex = kept.count + let outDepth = keptDepth + 1 + kept.append( + RawAXNode( + index: outIndex, + type: node.type, + label: node.label, + identifier: node.identifier, + value: node.value, + rect: node.rect, + enabled: node.enabled, + focused: node.focused, + selected: node.selected, + hittable: node.hittable && intersects, + depth: outDepth, + parentIndex: keptIndex, + hiddenContentAbove: node.hiddenContentAbove, + hiddenContentBelow: node.hiddenContentBelow, + actions: node.actions + ) + ) + keptIndex = outIndex + keptDepth = outDepth + } + + var anchor = parentAnchor + if include, + let newAnchor = scrollContainerAnchor( + forTypeName: node.type, + hasChildren: hasChildren[offset], + visible: intersects, + frame: rect, + nodeIndex: keptIndex + ) + { + anchor = newAnchor + } + states[offset] = BranchState( + cursor: transition.decision.descendants, + anchor: anchor, + keptIndex: keptIndex, + keptDepth: keptDepth + ) + } + return applyHiddenContentHints(hints, to: kept) + } + + private static func projectionTransition( + frame: CGRect, + intersectsClip: Bool, + typeName: String, + hasChildren: Bool, + cursor: ProjectionCursor, + policy: Policy + ) -> ProjectionTransition { + if policy == .plainViewport { + return ProjectionTransition( + decision: ProjectionDecision( + presentationVisible: intersectsClip, + descendants: .root + ), + hiddenContentFrame: !intersectsClip ? frame : nil + ) + } + + let hasFrame = !frame.isNull && !frame.isEmpty + let geometry: Geometry = hasFrame ? .framed(intersectsClip: intersectsClip) : .geometryless + let ownsDescendants = hasChildren + && (typeName == "Cell" || scrollContainerTypeNames.contains(typeName)) + return ProjectionTransition( + decision: cursor.project( + geometry: geometry, + descendantVisibility: ownsDescendants ? .owned : .independent + ), + hiddenContentFrame: !cursor.isProjectedOut && hasFrame && !intersectsClip ? frame : nil + ) + } + + private static func shouldInclude( + _ node: RawAXNode, + visible: Bool, + interactiveOnly: Bool, + policy: Policy + ) -> Bool { + if node.parentIndex == nil { return true } + if policy == .plainViewport && interactiveOnly && !visible && node.type != "Application" { + return false + } + return visibilityExemptCarrierTypes.contains(node.type) || visible + } + + private static func intersectsClip( + _ rect: CGRect, + viewport: CGRect, + scrollAnchor: (index: Int, rect: CGRect)? + ) -> Bool { + guard !rect.isNull, !rect.isEmpty, rect.intersects(viewport) else { return false } + guard let scrollAnchor else { return true } + return rect.intersects(scrollAnchor.rect) + } + + private static func scrollContainerAnchor( + forTypeName typeName: String, + hasChildren: Bool, + visible: Bool, + frame: CGRect, + nodeIndex: Int? + ) -> (index: Int, rect: CGRect)? { + guard let nodeIndex, + visible, + hasChildren, + scrollContainerTypeNames.contains(typeName) + else { return nil } + return (nodeIndex, frame) + } + + private static func rememberHiddenContentHint( + for frame: CGRect, + relativeTo scrollAnchor: (index: Int, rect: CGRect), + hints: inout [Int: (above: Bool, below: Bool)] + ) { + var hint = hints[scrollAnchor.index] ?? (above: false, below: false) + if frame.maxY <= scrollAnchor.rect.minY { + hint.above = true + } else if frame.minY >= scrollAnchor.rect.maxY { + hint.below = true + } else { + return + } + hints[scrollAnchor.index] = hint + } + + private static func applyHiddenContentHints( + _ hints: [Int: (above: Bool, below: Bool)], + to nodes: [RawAXNode] + ) -> [RawAXNode] { + if hints.isEmpty { return nodes } + return nodes.map { node in + guard let hint = hints[node.index] else { return node } + return RawAXNode( + index: node.index, + type: node.type, + label: node.label, + identifier: node.identifier, + value: node.value, + rect: node.rect, + enabled: node.enabled, + focused: node.focused, + selected: node.selected, + hittable: node.hittable, + depth: node.depth, + parentIndex: node.parentIndex, + hiddenContentAbove: node.hiddenContentAbove == true || hint.above ? true : nil, + hiddenContentBelow: node.hiddenContentBelow == true || hint.below ? true : nil, + actions: node.actions + ) + } + } +} diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotVisibilityFoldTests.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotVisibilityFoldTests.swift new file mode 100644 index 0000000000..aa12fce447 --- /dev/null +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotVisibilityFoldTests.swift @@ -0,0 +1,132 @@ +#if AGENT_DEVICE_RUNNER_UNIT_TESTS +import XCTest + +extension RunnerTests { + private static func foldNode( + _ index: Int, + type: String, + label: String? = nil, + rect: SnapshotRect, + hittable: Bool = false, + depth: Int, + parentIndex: Int? + ) -> RawAXNode { + RawAXNode( + index: index, type: type, label: label, identifier: nil, value: nil, rect: rect, + enabled: true, focused: nil, selected: nil, hittable: hittable, depth: depth, + parentIndex: parentIndex, hiddenContentAbove: nil, hiddenContentBelow: nil + ) + } + + private static func folded( + _ nodes: [RawAXNode], + viewport: CGRect, + interactiveOnly: Bool = false, + policy: SnapshotVisibilityFold.Policy = .cursorProjected + ) -> [RawAXNode] { + SnapshotVisibilityFold.fold( + nodes, viewport: viewport, interactiveOnly: interactiveOnly, policy: policy) + } + + func testRegularFoldClipsScrollOverflowReparentsAndBooksHints() { + let nodes = [ + Self.foldNode(0, type: "Application", label: "App", + rect: SnapshotRect(x: 0, y: 0, width: 402, height: 874), depth: 0, parentIndex: nil), + Self.foldNode(1, type: "ScrollView", + rect: SnapshotRect(x: 0, y: 96, width: 402, height: 700), depth: 1, parentIndex: 0), + Self.foldNode(2, type: "Cell", label: "Visible row", + rect: SnapshotRect(x: 0, y: 120, width: 402, height: 52), hittable: true, + depth: 2, parentIndex: 1), + Self.foldNode(3, type: "StaticText", label: "Detail", + rect: SnapshotRect(x: 16, y: 130, width: 200, height: 20), depth: 3, parentIndex: 2), + Self.foldNode(4, type: "Cell", label: "Offscreen row", + rect: SnapshotRect(x: 0, y: 900, width: 402, height: 52), hittable: true, + depth: 2, parentIndex: 1), + Self.foldNode(5, type: "StaticText", label: "Clamped child", + rect: SnapshotRect(x: 16, y: 96, width: 100, height: 20), depth: 3, parentIndex: 4), + // An offscreen generic wrapper does not own visibility; its visible overlay survives and is + // reparented to the scroll container. + Self.foldNode(6, type: "Other", + rect: SnapshotRect(x: 0, y: 900, width: 402, height: 52), depth: 2, parentIndex: 1), + Self.foldNode(7, type: "StaticText", label: "Visible overlay", + rect: SnapshotRect(x: 16, y: 200, width: 100, height: 20), depth: 3, parentIndex: 6), + ] + let folded = Self.folded(nodes, viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) + + XCTAssertEqual( + folded.map(\.type), ["Application", "ScrollView", "Cell", "StaticText", "StaticText"]) + XCTAssertEqual( + folded.compactMap(\.label), ["App", "Visible row", "Detail", "Visible overlay"]) + XCTAssertEqual(folded.map(\.depth), [0, 1, 2, 3, 2]) + XCTAssertEqual(folded.map(\.parentIndex), [nil, 0, 1, 2, 1]) + XCTAssertEqual(folded.first { $0.type == "ScrollView" }?.hiddenContentBelow, true) + } + + func testRegularFoldKeepsWindowCarriersButNeverHittableOutsideClip() { + let nodes = [ + Self.foldNode(0, type: "Application", label: "App", + rect: SnapshotRect(x: 0, y: 0, width: 402, height: 874), depth: 0, parentIndex: nil), + Self.foldNode(1, type: "Window", label: "Second screen", + rect: SnapshotRect(x: 402, y: 0, width: 402, height: 874), hittable: true, + depth: 1, parentIndex: 0), + Self.foldNode(2, type: "Button", label: "Gone", + rect: SnapshotRect(x: 500, y: 100, width: 100, height: 44), hittable: true, + depth: 2, parentIndex: 1), + ] + let folded = Self.folded(nodes, viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) + + XCTAssertEqual(folded.map(\.type), ["Application", "Window"]) + XCTAssertEqual(folded.last?.hittable, false) + XCTAssertFalse(folded.contains { $0.label == "Gone" }) + } + + func testRegularFoldDropsSubPixelContentlessDecorationOnEveryBackend() { + let nodes = [ + Self.foldNode(0, type: "Application", label: "App", + rect: SnapshotRect(x: 0, y: 0, width: 402, height: 874), depth: 0, parentIndex: nil), + Self.foldNode(1, type: "Button", + rect: SnapshotRect(x: 0, y: 100, width: 402, height: 1), hittable: true, + depth: 1, parentIndex: 0), + Self.foldNode(2, type: "StaticText", label: "Hairline caption", + rect: SnapshotRect(x: 0, y: 200, width: 402, height: 1), depth: 1, parentIndex: 0), + Self.foldNode(3, type: "StaticText", label: "Frameless semantics", + rect: SnapshotRect(x: 0, y: 0, width: 0, height: 0), hittable: true, + depth: 1, parentIndex: 0), + ] + let folded = Self.folded(nodes, viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) + + XCTAssertEqual(folded.compactMap(\.label), ["App", "Hairline caption", "Frameless semantics"]) + XCTAssertFalse(folded.contains { $0.type == "Button" }) + XCTAssertEqual(folded.first { $0.label == "Frameless semantics" }?.hittable, false) + } + + func testPlainViewportPolicyFoldsWithoutAncestorCursor() { + let nodes = [ + Self.foldNode(0, type: "Application", label: "App", + rect: SnapshotRect(x: 0, y: 0, width: 800, height: 600), depth: 0, parentIndex: nil), + Self.foldNode(1, type: "Window", label: "Offscreen window", + rect: SnapshotRect(x: 900, y: 0, width: 800, height: 600), depth: 1, parentIndex: 0), + Self.foldNode(2, type: "Cell", label: "Offscreen row", + rect: SnapshotRect(x: 0, y: 900, width: 800, height: 52), depth: 1, parentIndex: 0), + Self.foldNode(3, type: "StaticText", label: "Clamped child", + rect: SnapshotRect(x: 16, y: 100, width: 100, height: 20), depth: 2, parentIndex: 2), + ] + let viewport = CGRect(x: 0, y: 0, width: 800, height: 600) + + let regular = Self.folded(nodes, viewport: viewport, policy: .plainViewport) + XCTAssertEqual(regular.compactMap(\.label), ["App", "Offscreen window", "Clamped child"]) + + let interactive = Self.folded( + nodes, viewport: viewport, interactiveOnly: true, policy: .plainViewport) + XCTAssertFalse(interactive.contains { $0.label == "Offscreen window" }) + } + + func testScrollContainerTypeNamesMatchElementTypeSet() { + XCTAssertEqual( + SnapshotVisibilityFold.scrollContainerTypeNames, + Set(Self.scrollContainerTypes.map(elementTypeName)) + ) + XCTAssertEqual(elementTypeName(.cell), "Cell") + } +} +#endif