diff --git a/.github/workflows/ios.yml b/.github/workflows/ios.yml index a61c9efdab..6f2806b581 100644 --- a/.github/workflows/ios.yml +++ b/.github/workflows/ios.yml @@ -155,8 +155,13 @@ jobs: -only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotPresentationOwnsScopeAndRelativeDepth \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testPresentationRefusesAnAcquisitionCapturedForTheOtherProjection \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testCaptureHintIsTheOnlyAcquisitionViewOfARequest \ + -only-testing:AgentDeviceRunnerUITests/RunnerTests/testRegularFoldClipsScrollOverflowReparentsAndBooksHints \ + -only-testing:AgentDeviceRunnerUITests/RunnerTests/testRegularFoldKeepsWindowCarriersButNeverHittableOutsideClip \ + -only-testing:AgentDeviceRunnerUITests/RunnerTests/testRegularFoldDropsSubPixelContentlessDecorationOnEveryBackend \ + -only-testing:AgentDeviceRunnerUITests/RunnerTests/testPlainViewportPolicyFoldsWithoutAncestorCursor \ + -only-testing:AgentDeviceRunnerUITests/RunnerTests/testRegularPresentationRoutesThroughVisibilityFold \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotScopePolicyMatchesGoldenParityTable \ - -only-testing:AgentDeviceRunnerUITests/RunnerTests/testFlatSnapshotProjectionMatchesElementReverseScrollCapture \ + -only-testing:AgentDeviceRunnerUITests/RunnerTests/testScrollContainerTypeNamesMatchElementTypeSet \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testPrivateAXDepthLimitedRequiresEveryFrontierResolved \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testDeepExtensionCountsMissedFrontiers \ -only-testing:AgentDeviceRunnerUITests/RunnerTests/testPreferredPrivateAXBackendPlansAsPenalized \ diff --git a/CHANGELOG.md b/CHANGELOG.md index 34212bc4b0..382f9bc05c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,7 @@ - `--udid` with `--platform android` (and `--serial` with an Apple platform) now fails as the flag mistake it is — `INVALID_ARGS` naming the right flag — instead of reaching device resolution and answering `No Apple device with UDID emulator-5580` for an explicitly Android request. `--udid` addresses Apple devices, `--serial` addresses Android and HarmonyOS; matching pairs and requests that name no platform are unchanged. - Breaking (`--session-lock strip`): a device selector that names a different device than the bound session is no longer silently discarded. `strip` exists to drop redundant platform/scope selectors; when it also dropped `--udid`/`--serial`/`--device`, the command kept running against the *bound* device instead of the one the caller named — a wrong-device action that looks like a success. Such a request now fails with `INVALID_ARGS` under both `reject` and `strip`, and the error carries the two identities structurally (`requestedDevice`, `boundDevice`) plus a hint offering the two real recoveries: close the bound session if the requested device is intended, or remove the selector if the bound device is. The hint no longer suggests `--session-lock strip` for an identity conflict, since following that advice is what produced the wrong-device run. Scope-only stripping (`--platform`, `--target`, `--ios-simulator-device-set`, `--android-device-allowlist`) is unchanged. - iOS regular snapshots now apply one backend-neutral eligibility rule after every capture backend: a node survives when its accessibility type is interactive or it carries a non-empty label, identifier, or value. This removes the tree backend's extra "hittable non-Other" membership path and drops unlabeled decorative nodes consistently; labeled images, identifier-only nodes, and value-only nodes still survive. Raw snapshot membership is unchanged. +- iOS regular snapshots now run one shared clip fold inside presentation for every capture backend (#1797). Backends serialize reported facts -- every traversed node, at raw traversal depth -- and presentation alone decides what the viewport and scroll clips hide, books the scroll hints, and collapses depth; no backend carries its own copy of that interpretation anymore (the copies are what produced the scroll-overflow leak class, #1784). Three intentional edge deltas ride along, all in the direction of one backend-neutral rule: sub-pixel content-free decorations are now dropped by every backend (previously private-AX only); labeled offscreen Application/Window carriers now survive on every backend (previously tree only), still never hittable; and a query-sweep recovery snapshot without `-i` no longer lists offscreen elements. Nothing outside its clip, and nothing without geometry, is ever `hittable` in a regular snapshot, whatever the backend reported. - iOS `snapshot --raw` is now the acquired accessibility tree on every backend that can serve it (#1797). A raw request that recovered onto the private-AX backend — the route an app whose XCTest tree capture fails takes — returned the *regular* projection's viewport-pruned nodes labeled raw: everything scrolled out of the viewport, and every sub-pixel decoration, was missing from the one view whose purpose is showing what the pruned view hid. Raw now keeps every node the backend serialized, at traversal depth, and `--depth` still narrows it (for raw, presented depth *is* traversal depth). Two structural rules replace the hand-synchronized ones: the raw capture plan is derived from each backend's declared ability to serve raw, so the interactive query sweep — which has no hierarchy to return — cannot be planned for a raw request; and presentation refuses an acquisition captured for the other projection instead of relabeling it, dropping that tier with a structured failure. Breaking in the same direction: `snapshot --raw -i` now returns the acquired tree instead of an interactive-filtered one — `-i` narrows the regular projection, and the pair used to produce a third membership rule that differed per backend. Regular and `-i` output is unchanged. Backends now read one derived capture hint rather than the request itself, so what a capture is allowed to skip is stated once, next to the proof that skipping it keeps the projection complete. - Android `snapshot --raw` is now the acquired accessibility tree (#1832 C3): the three regular-projection pruners — nodes Android marks invisible, stale application windows, and covered same-window surfaces — no longer run at parse time, so `--raw` keeps everything the helper serialized (normalization only) and pruned content is recoverable for diagnosis. Regular and `-i` output is unchanged (property-checked identical to the previous implementation over 12,000 random tree × projection pairs, hidden-content hints included). Also: Android blocking-dialog recovery now reads the same daemon presentation an agent's `snapshot` sees instead of a hand-rolled subset, and acts on its occlusion result — a stale "App isn't responding" surface left under the foreground one no longer triggers recovery, and a covered "Close app" is never tapped ahead of the visible one; the Android freshness route signature no longer keys on `role`/`selected`, fields the Android backend never carries; and the Android helper's declared fidelity residues (no `checked`/`checkable`/`selected`/`long-clickable`, 5000-node cap before scoping, API-level cache-reset divergence) are recorded in `CONTEXT.md`. - `agent-device mcp` now carries its own usage guidance, so MCP-only clients (Codex CLI, Cursor, custom agents) no longer depend on a separately installed skill (#1833). The handshake `instructions` — returned by both `server/discover` and, newly, the legacy `initialize` — is a compact (< 2 KB, the Claude Code truncation limit) workflow card: start with `open {app, foreground: true}` instead of probing, act with `settle: true` and continue from the diff, verify with `wait`/`is`/`get`/`find`, copy `@refs` byte-for-byte, recover from sparse/AX-unavailable, follow error hints, `close`. A new MCP-only `help` tool serves the full guides on demand: no `topic` returns the CLI's decision card; `topic` returns `agent-device help ` verbatim (workflow, gestures, scripting, tv, macos, web, remote, debugging, …, or any tool name for its complete flag reference), prefixed with the one-line CLI→tool-property mapping. `help` is router-owned rather than a command descriptor, so it appears in `tools/list` only — not in the CLI, Node client, or `batch` — and its description tells the model it is not a startup step. Legacy `initialize` gains the optional `instructions` field; no other legacy field changes. diff --git a/CONTEXT.md b/CONTEXT.md index eee240082c..cf3fab1829 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -198,6 +198,11 @@ narrowing complete. **Snapshot eligibility**: Membership in a presented snapshot projection, independent of whether a node is currently hittable. +**Clip fold**: +The regular projection's single visibility interpreter, run inside presentation for every backend: +viewport and scroll-container clipping, ancestor projection, scroll hints, and collapsed depth. +Platform differences enter as a fold policy, never as a backend exception. + **Presented node**: A wire-facing snapshot value produced at the presentation boundary. diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+AXSnapshotFallback.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+AXSnapshotFallback.swift index 357a378e5c..4e95076b33 100644 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+AXSnapshotFallback.swift +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+AXSnapshotFallback.swift @@ -184,6 +184,8 @@ extension RunnerTests { hint: hint, viewport: viewport ) + // Serialization-level emptiness only: an acquired-but-fully-clipped tree is presentation's + // verdict now, surfaced by the plan's sparse classifier on the presented payload (#1797). if nodes.count <= 1 { NSLog("AGENT_DEVICE_RUNNER_PRIVATE_AX_SNAPSHOT_SPARSE=%ld", nodes.count) return nil @@ -213,7 +215,8 @@ extension RunnerTests { effectiveDepth: depthLimited ? effectiveDepth : nil, customActions: Self.privateAXCustomActionCoverage( response[RunnerAXSnapshotCustomActionsKey] - ) + ), + viewport: viewport ) #else return nil @@ -752,14 +755,20 @@ extension RunnerTests { ] ], ] - let nodes = privateAXAcquisition( - rawRoot: tree, - hint: CaptureHint( - projection: .regular, depth: nil, interactiveOnly: true, customActions: false), - viewport: CGRect(x: 0, y: 0, width: 390, height: 844) + let viewport = CGRect(x: 0, y: 0, width: 390, height: 844) + let hint = CaptureHint( + projection: .regular, depth: nil, interactiveOnly: true, customActions: false) + let acquired = privateAXAcquisition(rawRoot: tree, hint: hint, viewport: viewport) + // Acquisition serializes the drawer too; the shared fold is what hides it (#1797). + XCTAssertTrue(acquired.compactMap(\.label).contains("Admin settings")) + + let capture = SnapshotPresentation.presentRegular( + SnapshotAcquisition( + hint: hint, nodes: acquired, truncated: false, effectiveDepth: nil, viewport: viewport), + options: PresentationOptions(interactiveOnly: true, depth: nil, scope: nil, raw: false), + policy: .cursorProjected ) - - let labels = nodes.compactMap { $0.label } + let labels = (capture.payload.nodes ?? []).compactMap { $0.label } XCTAssertEqual( labels, ["Blue Sky", "Callstack", "Welcome back", "Email", "Password", "Sign in", "Forgot password?"] diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+FlatSnapshotFiltering.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+FlatSnapshotFiltering.swift deleted file mode 100644 index 9d0eff5b9e..0000000000 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+FlatSnapshotFiltering.swift +++ /dev/null @@ -1,362 +0,0 @@ -import XCTest - -struct FlatSnapshotFilterNode { - let isRoot: Bool - let visible: Bool -} - -struct FlatSnapshotFilterDecision { - let include: Bool -} - -enum FlatSnapshotVisibilityPolicy { - case interactiveOnly - case viewportProjected -} - -enum FlatSnapshotGeometry { - case geometryless - case framed(intersectsViewportAndScrollClip: Bool) -} - -enum FlatSnapshotDescendantVisibility: Equatable { - case independent - case owned -} - -struct FlatSnapshotProjectionCursor { - static let root = FlatSnapshotProjectionCursor(ancestorProjectedOut: false) - - private let ancestorProjectedOut: Bool - - var isProjectedOut: Bool { ancestorProjectedOut } - - func project( - geometry: FlatSnapshotGeometry, - descendantVisibility: FlatSnapshotDescendantVisibility - ) -> FlatSnapshotProjectionDecision { - let nodeProjectedOut: Bool - switch geometry { - case .geometryless: - nodeProjectedOut = false - case .framed(let intersectsViewportAndScrollClip): - nodeProjectedOut = !intersectsViewportAndScrollClip - } - let presentationVisible = !ancestorProjectedOut && !nodeProjectedOut - let descendantsProjectedOut = ancestorProjectedOut - || (nodeProjectedOut && descendantVisibility == .owned) - return FlatSnapshotProjectionDecision( - presentationVisible: presentationVisible, - descendants: FlatSnapshotProjectionCursor( - ancestorProjectedOut: descendantsProjectedOut - ) - ) - } -} - -struct FlatSnapshotProjectionDecision { - let presentationVisible: Bool - let descendants: FlatSnapshotProjectionCursor -} - -struct FlatSnapshotProjectionTransition { - let decision: FlatSnapshotProjectionDecision - let hiddenContentFrame: CGRect? -} - -extension RunnerTests { - func flatSnapshotGeometry( - frame: CGRect, - intersectsViewportAndScrollClip: Bool - ) -> FlatSnapshotGeometry { - if frame.isNull || frame.isEmpty { return .geometryless } - return .framed(intersectsViewportAndScrollClip: intersectsViewportAndScrollClip) - } - - func flatSnapshotDescendantVisibility( - elementType: XCUIElement.ElementType?, - hasChildren: Bool - ) -> FlatSnapshotDescendantVisibility { - guard hasChildren, let elementType else { return .independent } - if elementType == .cell || Self.scrollContainerTypes.contains(elementType) { - return .owned - } - return .independent - } - - func flatSnapshotProjectionTransition( - frame: CGRect, - intersectsViewportAndScrollClip: Bool, - elementType: XCUIElement.ElementType?, - hasChildren: Bool, - cursor: FlatSnapshotProjectionCursor - ) -> FlatSnapshotProjectionTransition { - let hasFrame = !frame.isNull && !frame.isEmpty - return FlatSnapshotProjectionTransition( - decision: cursor.project( - geometry: flatSnapshotGeometry( - frame: frame, - intersectsViewportAndScrollClip: intersectsViewportAndScrollClip - ), - descendantVisibility: flatSnapshotDescendantVisibility( - elementType: elementType, - hasChildren: hasChildren - ) - ), - hiddenContentFrame: !cursor.isProjectedOut && hasFrame - && !intersectsViewportAndScrollClip ? frame : nil - ) - } - - /// Regular-projection acquisition gate. The raw projection never reaches it: raw acquisition is - /// the acquired tree, so it has no visibility filter to consult (#1797 D4). - func flatSnapshotFilterDecision( - _ node: FlatSnapshotFilterNode, - hint: CaptureHint, - visibilityPolicy: FlatSnapshotVisibilityPolicy - ) -> FlatSnapshotFilterDecision { - let include: Bool - if node.isRoot { - include = true - } else if !node.visible - && (hint.interactiveOnly || visibilityPolicy == .viewportProjected) - { - include = false - } else { - include = true - } - - return FlatSnapshotFilterDecision(include: include) - } - - func privateAXInteractiveCandidate(rawElementType: Int) -> Bool { - guard let type = flatSnapshotElementType(rawElementType: rawElementType) else { - return false - } - return interactiveTypes.contains(type) || Self.scrollContainerTypes.contains(type) - } - - func flatSnapshotElementType(rawElementType: Int) -> XCUIElement.ElementType? { - guard let raw = UInt(exactly: rawElementType), - let type = XCUIElement.ElementType(rawValue: raw) - else { - return nil - } - return type - } - -#if AGENT_DEVICE_RUNNER_UNIT_TESTS - func testFlatSnapshotProjectionMatchesElementReverseScrollCapture() { - struct FixtureNode { - let label: String - let type: XCUIElement.ElementType - let frame: CGRect - let children: [FixtureNode] - } - - let tableFrame = CGRect(x: 0, y: 152.333, width: 402, height: 659.667) - let visibleRowFrame = CGRect(x: 0, y: 251.333, width: 402, height: 43.667) - let offscreenThemeRowFrame = CGRect(x: 0, y: 2044.333, width: 402, height: 44) - let clampedThemeFrame = CGRect(x: 0, y: 152.333, width: 53, height: 20.333) - let geometrylessFrame = CGRect.zero - var visibleLabels: [String] = [] - var hints: [Int: (above: Bool, below: Bool)] = [:] - - func visit( - _ node: FixtureNode, - cursor: FlatSnapshotProjectionCursor, - scrollAnchor: (index: Int, rect: CGRect)? - ) { - let intersects = isVisibleInRegularSnapshot( - node.frame, - viewport: CGRect(x: 0, y: 0, width: 402, height: 874), - scrollContainerAnchor: scrollAnchor - ) - let transition = flatSnapshotProjectionTransition( - frame: node.frame, - intersectsViewportAndScrollClip: intersects, - elementType: node.type, - hasChildren: !node.children.isEmpty, - cursor: cursor - ) - if transition.decision.presentationVisible { visibleLabels.append(node.label) } - if let hiddenFrame = transition.hiddenContentFrame, let scrollAnchor { - rememberHiddenContentHint( - for: hiddenFrame, - relativeTo: scrollAnchor, - hints: &hints - ) - } - let nextScrollAnchor = Self.scrollContainerTypes.contains(node.type) - && transition.decision.presentationVisible - ? (index: 0, rect: node.frame) : scrollAnchor - for child in node.children { - visit( - child, - cursor: transition.decision.descendants, - scrollAnchor: nextScrollAnchor - ) - } - } - - let fixture = FixtureNode( - label: "Table", - type: .table, - frame: tableFrame, - children: [ - FixtureNode( - label: "Profile Picture", - type: .cell, - frame: visibleRowFrame, - children: [ - FixtureNode( - label: "Profile semantics", - type: .staticText, - frame: geometrylessFrame, - children: [ - FixtureNode( - label: "Profile detail", - type: .staticText, - frame: visibleRowFrame, - children: [] - ) - ] - ) - ] - ), - // Element reports this row at y=2044 but clamps Theme and Auto to the table's top edge. - FixtureNode( - label: "Theme row", - type: .cell, - frame: offscreenThemeRowFrame, - children: [ - FixtureNode( - label: "Theme", - type: .staticText, - frame: clampedThemeFrame, - children: [] - ), - FixtureNode( - label: "Auto", - type: .staticText, - frame: clampedThemeFrame, - children: [] - ) - ] - ), - FixtureNode( - label: "Offscreen wrapper", - type: .other, - frame: offscreenThemeRowFrame, - children: [ - FixtureNode( - label: "Visible overlay", - type: .staticText, - frame: visibleRowFrame, - children: [] - ) - ] - ) - ] - ) - visit(fixture, cursor: .root, scrollAnchor: nil) - - XCTAssertEqual( - visibleLabels, - ["Table", "Profile Picture", "Profile semantics", "Profile detail", "Visible overlay"] - ) - XCTAssertEqual(hints.count, 1) - XCTAssertEqual(hints[0]?.above, false) - XCTAssertEqual(hints[0]?.below, true) - - let scopedRoot = flatSnapshotProjectionTransition( - frame: offscreenThemeRowFrame, - intersectsViewportAndScrollClip: false, - elementType: .cell, - hasChildren: true, - cursor: .root - ) - let scopedClampedChild = flatSnapshotProjectionTransition( - frame: clampedThemeFrame, - intersectsViewportAndScrollClip: true, - elementType: .staticText, - hasChildren: false, - cursor: scopedRoot.decision.descendants - ) - XCTAssertFalse(scopedClampedChild.decision.presentationVisible) - } - - func testFlatSnapshotFilterDecisionMatrixCoversOptions() { - func hint(interactiveOnly: Bool) -> CaptureHint { - CaptureHint( - projection: .regular, depth: nil, interactiveOnly: interactiveOnly, customActions: false) - } - let visibleContent = FlatSnapshotFilterNode( - isRoot: false, - visible: true - ) - let hiddenInteractive = FlatSnapshotFilterNode( - isRoot: false, - visible: false - ) - let decorative = FlatSnapshotFilterNode( - isRoot: false, - visible: true - ) - let hiddenRoot = FlatSnapshotFilterNode( - isRoot: true, - visible: false - ) - - XCTAssertTrue( - flatSnapshotFilterDecision( - visibleContent, - hint: hint(interactiveOnly: false), - visibilityPolicy: .interactiveOnly - ).include - ) - XCTAssertFalse( - flatSnapshotFilterDecision( - hiddenInteractive, - hint: hint(interactiveOnly: true), - visibilityPolicy: .interactiveOnly - ).include - ) - XCTAssertFalse( - flatSnapshotFilterDecision( - hiddenInteractive, - hint: hint(interactiveOnly: false), - visibilityPolicy: .viewportProjected - ).include - ) - XCTAssertTrue( - flatSnapshotFilterDecision( - hiddenInteractive, - hint: hint(interactiveOnly: false), - visibilityPolicy: .interactiveOnly - ).include - ) - XCTAssertTrue( - flatSnapshotFilterDecision( - hiddenRoot, - hint: hint(interactiveOnly: false), - visibilityPolicy: .viewportProjected - ).include - ) - XCTAssertTrue( - flatSnapshotFilterDecision( - decorative, - hint: hint(interactiveOnly: false), - visibilityPolicy: .interactiveOnly - ).include - ) - } - - func testPrivateAXInteractiveCandidatesPreserveBackendInputs() { - XCTAssertTrue( - privateAXInteractiveCandidate(rawElementType: Int(XCUIElement.ElementType.scrollView.rawValue)), - "private AX marks scroll containers as interactive candidates" - ) - } -#endif -} diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXElementTypes.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXElementTypes.swift new file mode 100644 index 0000000000..b00728df65 --- /dev/null +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXElementTypes.swift @@ -0,0 +1,28 @@ +import XCTest + +extension RunnerTests { + func privateAXInteractiveCandidate(rawElementType: Int) -> Bool { + guard let type = privateAXElementType(rawElementType: rawElementType) else { + return false + } + return interactiveTypes.contains(type) || Self.scrollContainerTypes.contains(type) + } + + func privateAXElementType(rawElementType: Int) -> XCUIElement.ElementType? { + guard let raw = UInt(exactly: rawElementType), + let type = XCUIElement.ElementType(rawValue: raw) + else { + return nil + } + return type + } + +#if AGENT_DEVICE_RUNNER_UNIT_TESTS + func testPrivateAXInteractiveCandidatesPreserveBackendInputs() { + XCTAssertTrue( + privateAXInteractiveCandidate(rawElementType: Int(XCUIElement.ElementType.scrollView.rawValue)), + "private AX marks scroll containers as interactive candidates" + ) + } +#endif +} diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXPresentation.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXPresentation.swift index bf4e7591e8..51295e180b 100644 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXPresentation.swift +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXPresentation.swift @@ -1,7 +1,7 @@ import XCTest -/// Reported facts for one private-AX element, read once so both projections describe a node the -/// same way. Only membership and clip-derived state differ between them. +/// Reported facts for one private-AX element, read once so every consumer describes a node the +/// same way. struct PrivateAXFields { let rect: CGRect let label: String @@ -15,121 +15,40 @@ struct PrivateAXFields { let actions: [String]? let children: [[String: Any]] - var hasSemanticContent: Bool { [label, identifier, value].contains { !$0.isEmpty } } var hasFrame: Bool { !rect.isNull && !rect.isEmpty } } extension RunnerTests { - private static let privateAXProjectionTolerance: CGFloat = 1 - - /// Acquisition entry point for the private AX backend. - /// - /// The regular projection folds the viewport and scroll clips and records scroll hints; the raw - /// projection is the acquired tree, so it drops nothing and keeps traversal depth (#1797 D4 — a - /// recovered `snapshot --raw` used to return viewport-pruned nodes labeled raw). + /// Private-AX acquisition: ONE serializer for both projections. Reported facts at traversal + /// depth -- membership, the clip fold, scroll hints, and collapsed depth are + /// `SnapshotPresentation`'s alone (#1797). The traversal-depth cut is the backend's one + /// narrowing, complete for raw (raw depth *is* traversal depth) and a declared residue for + /// regular; `hittable` is the pre-clip geometric fact the fold narrows per projection. func privateAXAcquisition(rawRoot: [String: Any], hint: CaptureHint, viewport: CGRect) -> [RawAXNode] { var nodes: [RawAXNode] = [] - switch hint.projection { - case .raw: - appendPrivateAXRawNode(rawRoot, to: &nodes, hint: hint, viewport: viewport, - depth: 0, parentIndex: nil) - return nodes - case .regular: - var hints: [Int: (above: Bool, below: Bool)] = [:] - appendPrivateAXNode(rawRoot, to: &nodes, hints: &hints, hint: hint, viewport: viewport, - depth: 0, parentIndex: nil, scrollContext: nil, - projectionCursor: .root) - return applyHiddenContentHints(hints, to: nodes) - } + appendPrivateAXNode(rawRoot, to: &nodes, hint: hint, viewport: viewport, + depth: 0, parentIndex: nil) + return nodes } - /// Raw projection: every serialized node, in traversal order, at traversal depth. Depth is the - /// one narrowing a raw acquisition can prove complete, because raw depth *is* traversal depth. - private func appendPrivateAXRawNode(_ raw: [String: Any], to nodes: inout [RawAXNode], + private func appendPrivateAXNode(_ raw: [String: Any], to nodes: inout [RawAXNode], hint: CaptureHint, viewport: CGRect, depth: Int, parentIndex: Int?) { if let limit = hint.depth, depth > limit { return } let fields = privateAXFields(raw) - // No clip fold outside the regular projection: hittability stays the reported geometric fact. - let onScreen = fields.hasFrame && isVisibleInViewport(fields.rect, viewport) + let onScreen = fields.hasFrame && Self.isVisibleInViewport(fields.rect, viewport) let index = nodes.count nodes.append( privateAXNode(fields, index: index, depth: depth, parentIndex: parentIndex, onScreen: onScreen) ) for child in fields.children { - appendPrivateAXRawNode(child, to: &nodes, hint: hint, viewport: viewport, + appendPrivateAXNode(child, to: &nodes, hint: hint, viewport: viewport, depth: depth + 1, parentIndex: index) } } - private func appendPrivateAXNode(_ raw: [String: Any], to nodes: inout [RawAXNode], - hints: inout [Int: (above: Bool, below: Bool)], hint: CaptureHint, viewport: CGRect, - depth: Int, parentIndex: Int?, - scrollContext: (index: Int, rect: CGRect)?, projectionCursor: FlatSnapshotProjectionCursor) - { - if let limit = hint.depth, depth > limit { return } - let fields = privateAXFields(raw) - let rect = fields.rect - let children = fields.children - let elementType = fields.elementType - let negligibleDecoration = parentIndex != nil - && !fields.hasSemanticContent - && (!fields.hasFrame - || rect.width <= Self.privateAXProjectionTolerance - || rect.height <= Self.privateAXProjectionTolerance) - let onScreen = fields.hasFrame - && isVisibleInRegularSnapshot( - rect, - viewport: viewport, - scrollContainerAnchor: scrollContext - ) - let projectionTransition = flatSnapshotProjectionTransition( - frame: rect, - intersectsViewportAndScrollClip: onScreen, - elementType: elementType, - hasChildren: !children.isEmpty, - cursor: projectionCursor - ) - let projection = projectionTransition.decision - let presentationVisible = projection.presentationVisible && !negligibleDecoration - let decision = flatSnapshotFilterDecision( - FlatSnapshotFilterNode(isRoot: parentIndex == nil, visible: presentationVisible), - hint: hint, visibilityPolicy: .viewportProjected) - let include = decision.include - - if let hiddenFrame = projectionTransition.hiddenContentFrame, let scrollContext { - rememberHiddenContentHint(for: hiddenFrame, relativeTo: scrollContext, hints: &hints) - } - - let currentIndex: Int? - if include { - currentIndex = nodes.count - nodes.append( - privateAXNode( - fields, index: nodes.count, depth: depth, parentIndex: parentIndex, onScreen: onScreen) - ) - } else { currentIndex = parentIndex } - - let nextScrollContext: (index: Int, rect: CGRect)? - if include, let elementType, let currentIndex { - nextScrollContext = scrollContainerAnchor( - for: elementType, - hasChildren: !children.isEmpty, - visible: onScreen, - frame: rect, - nodeIndex: currentIndex - ) ?? scrollContext - } else { nextScrollContext = scrollContext } - for child in children { - appendPrivateAXNode(child, to: &nodes, hints: &hints, hint: hint, viewport: viewport, - depth: depth + 1, parentIndex: currentIndex, - scrollContext: nextScrollContext, - projectionCursor: projection.descendants) - } - } - private func privateAXFields(_ raw: [String: Any]) -> PrivateAXFields { let rawType = privateAXPresentationInt(raw["type"]) ?? 0 return PrivateAXFields( @@ -138,7 +57,7 @@ extension RunnerTests { identifier: privateAXPresentationString(raw["identifier"]), value: privateAXPresentationString(raw["value"]), rawType: rawType, - elementType: flatSnapshotElementType(rawElementType: rawType), + elementType: privateAXElementType(rawElementType: rawType), enabled: privateAXPresentationBool(raw["enabled"]) ?? true, focused: privateAXPresentationBool(raw["focused"]) == true ? true : nil, selected: privateAXPresentationBool(raw["selected"]) == true ? true : nil, @@ -201,10 +120,28 @@ extension RunnerTests { "frame": frame(16, 900, 360, 44)]]]]] } + /// Acquire with the private-AX serializer, then present through the shared regular fold -- + /// the production route for this backend since the fold moved into presentation (#1797). + fileprivate func privateAXRegularPresentation( + rawRoot: [String: Any], + viewport: CGRect, + interactiveOnly: Bool = false + ) -> [PresentedNode] { + let hint = CaptureHint( + projection: .regular, depth: nil, interactiveOnly: interactiveOnly, customActions: false) + let acquired = privateAXAcquisition(rawRoot: rawRoot, hint: hint, viewport: viewport) + return SnapshotPresentation.presentRegular( + SnapshotAcquisition( + hint: hint, nodes: acquired, truncated: false, effectiveDepth: nil, viewport: viewport), + options: PresentationOptions( + interactiveOnly: interactiveOnly, depth: nil, scope: nil, raw: false), + policy: .cursorProjected + ).payload.nodes ?? [] + } + func testPrivateAXRegularPresentationProjectsToViewportAndKeepsScrollHint() { - let nodes = privateAXAcquisition(rawRoot: Self.privateAXScrolledFixture, - hint: CaptureHint(projection: .regular, depth: nil, interactiveOnly: false, - customActions: false), + let nodes = privateAXRegularPresentation( + rawRoot: Self.privateAXScrolledFixture, viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) XCTAssertEqual(nodes.compactMap(\.label), ["Element", "Profile picture"]) let scrollView = nodes.first { $0.type == "ScrollView" } @@ -214,14 +151,12 @@ extension RunnerTests { /// #1797 D4: the raw projection is the acquired tree. The offscreen row and the sub-pixel /// decoration the regular projection folds away are both present, at traversal depth, and every - /// regular node still appears — `regular ⊆ raw` on the same capture. + /// regular node still appears -- `regular ⊆ raw` on the same capture. func testPrivateAXRawProjectionKeepsEveryAcquiredNode() { let viewport = CGRect(x: 0, y: 0, width: 402, height: 874) let root = Self.privateAXScrolledFixture - let regular = privateAXAcquisition(rawRoot: root, - hint: CaptureHint(projection: .regular, depth: nil, interactiveOnly: true, - customActions: false), - viewport: viewport) + let regular = privateAXRegularPresentation(rawRoot: root, viewport: viewport, + interactiveOnly: true) let raw = privateAXAcquisition(rawRoot: root, hint: CaptureHint(projection: .raw, depth: nil, interactiveOnly: false, customActions: false), viewport: viewport) @@ -267,10 +202,8 @@ extension RunnerTests { ["type": Int(XCUIElement.ElementType.switch.rawValue), "label": "Theme", "frame": frame(340, 96, 46, 44)]]]]]]] - let nodes = privateAXAcquisition(rawRoot: root, - hint: CaptureHint(projection: .regular, depth: nil, interactiveOnly: false, - customActions: false), - viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) + let nodes = privateAXRegularPresentation( + rawRoot: root, viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) XCTAssertEqual(nodes.compactMap(\.label), ["Element"]) XCTAssertEqual(nodes.first { $0.type == "Table" }?.hiddenContentBelow, true) @@ -284,10 +217,9 @@ extension RunnerTests { "label": "Settings semantics", "frame": zero, "children": [[ "type": Int(XCUIElement.ElementType.button.rawValue), "label": "Theme", "frame": zero], ["type": Int(XCUIElement.ElementType.other.rawValue), "frame": zero]]]]] - let nodes = privateAXAcquisition(rawRoot: root, - hint: CaptureHint(projection: .regular, depth: nil, interactiveOnly: true, - customActions: false), - viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) + let nodes = privateAXRegularPresentation( + rawRoot: root, viewport: CGRect(x: 0, y: 0, width: 402, height: 874), + interactiveOnly: true) XCTAssertEqual(nodes.compactMap(\.label), ["Element", "Settings semantics", "Theme"]) XCTAssertEqual(nodes.filter { $0.index != 0 }.map(\.hittable), [false, false]) XCTAssertFalse(nodes.contains { $0.type == "Other" }) diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift index 428b4ef3d2..9513c3a8b2 100644 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift @@ -27,16 +27,12 @@ extension RunnerTests { let hittable: Bool let focused: Bool let selected: Bool - let visible: Bool } private struct SnapshotTraversalEntry { let snapshot: XCUIElementSnapshot let depth: Int - let visibleDepth: Int let parentIndex: Int? - let nearestScrollAnchor: (index: Int, rect: CGRect)? - let projectionCursor: FlatSnapshotProjectionCursor } struct SnapshotCaptureFailure: Error { @@ -147,8 +143,12 @@ extension RunnerTests { return result.elements } + // Acquisition serializes facts: every traversed node is emitted at raw traversal depth, and + // the regular projection's clip fold runs once inside `SnapshotPresentation` (#1797). The two + // walks this backend keeps are budget and augmentation, never membership: the traversal-depth + // cut (declared residue -- regular presentation emits collapsed depth) and the collapsed-tab + // expansion, which needs live element handles. var nodes: [RawAXNode] = [] - var hiddenContentHintsByNodeIndex: [Int: (above: Bool, below: Bool)] = [:] let rootEvaluation = evaluateSnapshot(context.rootSnapshot, in: context) nodes.append( makeSnapshotNode( @@ -170,87 +170,21 @@ extension RunnerTests { } var seen = Set() - let rootScrollAnchor = scrollContainerAnchor( - for: context.rootSnapshot, - visible: rootEvaluation.visible, - nodeIndex: 0 - ) - #if os(iOS) - let rootProjection = flatSnapshotProjectionTransition( - frame: context.rootSnapshot.frame, - intersectsViewportAndScrollClip: rootEvaluation.visible, - elementType: context.rootSnapshot.elementType, - hasChildren: !context.rootSnapshot.children.isEmpty, - cursor: .root - ) - let rootDescendantProjectionCursor = rootProjection.decision.descendants - #else - let rootDescendantProjectionCursor = FlatSnapshotProjectionCursor.root - #endif var stack: [SnapshotTraversalEntry] = context.rootSnapshot.children.map { SnapshotTraversalEntry( snapshot: $0, depth: 1, - visibleDepth: 1, - parentIndex: 0, - nearestScrollAnchor: rootScrollAnchor, - projectionCursor: rootDescendantProjectionCursor + parentIndex: 0 ) } while let entry = stack.popLast() { let snapshot = entry.snapshot let depth = entry.depth - let visibleDepth = entry.visibleDepth let parentIndex = entry.parentIndex - let nearestScrollAnchor = entry.nearestScrollAnchor - let projectionCursor = entry.projectionCursor if let limit = hint.depth, depth > limit { continue } let evaluation = evaluateSnapshot(snapshot, in: context) - let intersectsViewportAndScrollClip = isVisibleInRegularSnapshot( - snapshot.frame, - viewport: context.viewport, - scrollContainerAnchor: nearestScrollAnchor - ) - #if os(iOS) - let projectionTransition = flatSnapshotProjectionTransition( - frame: snapshot.frame, - intersectsViewportAndScrollClip: intersectsViewportAndScrollClip, - elementType: snapshot.elementType, - hasChildren: !snapshot.children.isEmpty, - cursor: projectionCursor - ) - let projection = projectionTransition.decision - #else - let projectionTransition = FlatSnapshotProjectionTransition( - decision: FlatSnapshotProjectionDecision( - presentationVisible: intersectsViewportAndScrollClip, - descendants: .root - ), - hiddenContentFrame: !intersectsViewportAndScrollClip ? snapshot.frame : nil - ) - let projection = projectionTransition.decision - #endif - if let hiddenFrame = projectionTransition.hiddenContentFrame, let nearestScrollAnchor - { - rememberHiddenContentHint( - for: hiddenFrame, - relativeTo: nearestScrollAnchor, - hints: &hiddenContentHintsByNodeIndex - ) - } - let include = shouldAcquireSnapshotNode( - snapshot: snapshot, - label: evaluation.label, - identifier: evaluation.identifier, - valueText: evaluation.valueText, - hint: hint, - hittable: evaluation.hittable, - visible: projection.presentationVisible, - regularSnapshot: true - ) - let key = Self.snapshotTraversalIdentity( elementType: snapshot.elementType, label: evaluation.label, @@ -262,64 +196,48 @@ extension RunnerTests { seen.insert(key) } - let currentIndex = include && !isDuplicate ? nodes.count : parentIndex + let currentIndex = !isDuplicate ? nodes.count : parentIndex if depth < context.maxDepth { - let nextVisibleDepth = include && !isDuplicate ? visibleDepth + 1 : visibleDepth - let nextScrollContainerAnchor: (index: Int, rect: CGRect)? - if include && !isDuplicate { - nextScrollContainerAnchor = - scrollContainerAnchor( - for: snapshot, - visible: intersectsViewportAndScrollClip, - nodeIndex: currentIndex - ) - ?? nearestScrollAnchor - } else { - nextScrollContainerAnchor = nearestScrollAnchor - } for child in snapshot.children.reversed() { stack.append( SnapshotTraversalEntry( snapshot: child, depth: depth + 1, - visibleDepth: nextVisibleDepth, - parentIndex: currentIndex, - nearestScrollAnchor: nextScrollContainerAnchor, - projectionCursor: projection.descendants + parentIndex: currentIndex ) ) } } - if !include || isDuplicate { continue } + if isDuplicate { continue } let index = nodes.count nodes.append( makeSnapshotNode( snapshot: snapshot, evaluation: evaluation, - depth: min(context.maxDepth, visibleDepth), + depth: depth, index: index, parentIndex: parentIndex ) ) - if visibleDepth < context.maxDepth { + if depth < context.maxDepth { appendCollapsedTabFallbackNodes( to: &nodes, containerSnapshot: snapshot, resolveElements: collapsedTabDescendants, - depth: visibleDepth + 1, + depth: depth + 1, parentIndex: index ) } - } return SnapshotAcquisition( hint: hint, - nodes: applyHiddenContentHints(hiddenContentHintsByNodeIndex, to: nodes), + nodes: nodes, truncated: false, - effectiveDepth: nil + effectiveDepth: nil, + viewport: context.viewport ) } @@ -430,30 +348,19 @@ extension RunnerTests { if let limit = hint.depth, depth > limit { return } let evaluation = evaluateSnapshot(snapshot, in: context) - let include = shouldAcquireSnapshotNode( - snapshot: snapshot, - label: evaluation.label, - identifier: evaluation.identifier, - valueText: evaluation.valueText, - hint: hint, - hittable: evaluation.hittable, - visible: evaluation.visible - ) - let currentIndex = include ? nodes.count : parentIndex - if include { - if nodes.count >= Self.rawSnapshotMaxNodes { - throw rawSnapshotTooLargeFailure(nodeCount: nodes.count + 1) - } - nodes.append( - makeSnapshotNode( - snapshot: snapshot, - evaluation: evaluation, - depth: depth, - index: nodes.count, - parentIndex: parentIndex - ) - ) + if nodes.count >= Self.rawSnapshotMaxNodes { + throw rawSnapshotTooLargeFailure(nodeCount: nodes.count + 1) } + let currentIndex = nodes.count + nodes.append( + makeSnapshotNode( + snapshot: snapshot, + evaluation: evaluation, + depth: depth, + index: currentIndex, + parentIndex: parentIndex + ) + ) let children = snapshot.children for child in children { @@ -462,7 +369,13 @@ extension RunnerTests { } try walk(context.rootSnapshot, depth: 0, parentIndex: nil) - return SnapshotAcquisition(hint: hint, nodes: nodes, truncated: false, effectiveDepth: nil) + return SnapshotAcquisition( + hint: hint, + nodes: nodes, + truncated: false, + effectiveDepth: nil, + viewport: context.viewport + ) } func querySweepSnapshotAcquisition( @@ -474,7 +387,13 @@ extension RunnerTests { interactiveRootNode(rect: .zero) ] if hint.depth == 0 { - return SnapshotAcquisition(hint: hint, nodes: nodes, truncated: false, effectiveDepth: nil) + return SnapshotAcquisition( + hint: hint, + nodes: nodes, + truncated: false, + effectiveDepth: nil, + viewport: .infinite + ) } // Bounded by both its own sweep budget and the umbrella capture-plan deadline, so a @@ -498,8 +417,7 @@ extension RunnerTests { element: element, index: 0, parentIndex: 0, - viewport: viewport, - hint: hint + viewport: viewport ) else { continue } @@ -545,7 +463,13 @@ extension RunnerTests { ) ) } - return SnapshotAcquisition(hint: hint, nodes: nodes, truncated: truncated, effectiveDepth: nil) + return SnapshotAcquisition( + hint: hint, + nodes: nodes, + truncated: truncated, + effectiveDepth: nil, + viewport: viewport + ) } func snapshotAccessibilityUnavailable(failure: SnapshotCaptureFailure) -> DataPayload { @@ -880,44 +804,6 @@ extension RunnerTests { // MARK: - Snapshot Filtering - /// Conservative acquisition gate. Regular capture keeps every node its visibility policy can - /// supply; `SnapshotPresentation` alone decides eligibility after the backend returns. - private func shouldAcquireSnapshotNode( - snapshot: XCUIElementSnapshot, - label: String, - identifier: String, - valueText: String?, - hint: CaptureHint, - hittable: Bool, - visible: Bool, - regularSnapshot: Bool = false - ) -> Bool { - let type = snapshot.elementType - let hasContent = !label.isEmpty || !identifier.isEmpty || (valueText != nil) - #if os(iOS) - if regularSnapshot && !visible && type != .application && type != .window { - return false - } - #endif - if regularSnapshot { - #if os(macOS) - if hint.interactiveOnly && !visible && type != .application { - return false - } - #endif - if type == .application || type == .window { return true } - return visible - } - if hint.interactiveOnly { - if isScrollableContainer(snapshot, visible: visible) { return true } - if interactiveTypes.contains(type) { return true } - if hittable && type != .other { return true } - if hasContent { return true } - return false - } - return true - } - private func computedSnapshotHittable( _ snapshot: XCUIElementSnapshot, viewport: CGRect, @@ -1119,8 +1005,7 @@ extension RunnerTests { valueText: valueText, hittable: computedSnapshotHittable(snapshot, viewport: context.viewport, laterNodes: laterNodes), focused: snapshotHasFocus(snapshot), - selected: snapshotIsSelected(snapshot), - visible: isVisibleInViewport(snapshot.frame, context.viewport) + selected: snapshotIsSelected(snapshot) ) } @@ -1235,21 +1120,11 @@ extension RunnerTests { return nil } - func isVisibleInViewport(_ rect: CGRect, _ viewport: CGRect) -> Bool { + static func isVisibleInViewport(_ rect: CGRect, _ viewport: CGRect) -> Bool { if rect.isNull || rect.isEmpty { return false } return rect.intersects(viewport) } - func isVisibleInRegularSnapshot( - _ rect: CGRect, - viewport: CGRect, - scrollContainerAnchor: (index: Int, rect: CGRect)? - ) -> Bool { - if !isVisibleInViewport(rect, viewport) { return false } - guard let scrollContainerAnchor else { return true } - return isVisibleInViewport(rect, scrollContainerAnchor.rect) - } - private func appendCollapsedTabFallbackNodes( to nodes: inout [RawAXNode], containerSnapshot: XCUIElementSnapshot, @@ -1267,82 +1142,6 @@ extension RunnerTests { nodes.append(contentsOf: fallbackNodes) } - private func scrollContainerAnchor( - for snapshot: XCUIElementSnapshot, - visible: Bool, - nodeIndex: Int? - ) -> (index: Int, rect: CGRect)? { - return scrollContainerAnchor( - for: snapshot.elementType, - hasChildren: !snapshot.children.isEmpty, - visible: visible, - frame: snapshot.frame, - nodeIndex: nodeIndex - ) - } - - func scrollContainerAnchor( - for elementType: XCUIElement.ElementType, - hasChildren: Bool, - visible: Bool, - frame: CGRect, - nodeIndex: Int? - ) -> (index: Int, rect: CGRect)? { - guard let nodeIndex else { return nil } - if !isScrollableContainer( - elementType: elementType, - hasChildren: hasChildren, - visible: visible - ) { return nil } - return (nodeIndex, frame) - } - - func rememberHiddenContentHint( - for frame: CGRect, - relativeTo scrollContainerAnchor: (index: Int, rect: CGRect), - hints: inout [Int: (above: Bool, below: Bool)] - ) { - if frame.isNull || frame.isEmpty { return } - var hint = hints[scrollContainerAnchor.index] ?? (above: false, below: false) - if frame.maxY <= scrollContainerAnchor.rect.minY { - hint.above = true - } else if frame.minY >= scrollContainerAnchor.rect.maxY { - hint.below = true - } else { - return - } - hints[scrollContainerAnchor.index] = hint - } - - func applyHiddenContentHints( - _ hints: [Int: (above: Bool, below: Bool)], - to nodes: [RawAXNode] - ) -> [RawAXNode] { - if hints.isEmpty { return nodes } - return nodes.map { node in - guard let hint = hints[node.index] else { return node } - let hiddenContentAbove: Bool? = (node.hiddenContentAbove == true || hint.above) ? true : nil - let hiddenContentBelow: Bool? = (node.hiddenContentBelow == true || hint.below) ? true : nil - return RawAXNode( - index: node.index, - type: node.type, - label: node.label, - identifier: node.identifier, - value: node.value, - rect: node.rect, - enabled: node.enabled, - focused: node.focused, - selected: node.selected, - hittable: node.hittable, - depth: node.depth, - parentIndex: node.parentIndex, - hiddenContentAbove: hiddenContentAbove, - hiddenContentBelow: hiddenContentBelow, - actions: node.actions - ) - } - } - private func collapsedTabFallbackNodes( for containerSnapshot: XCUIElementSnapshot, resolveElements: () -> [XCUIElement], @@ -1576,36 +1375,22 @@ extension RunnerTests { element: XCUIElement, index: Int, parentIndex: Int?, - viewport: CGRect, - hint: CaptureHint + viewport: CGRect ) -> RawAXNode? { var node: RawAXNode? let exceptionMessage = RunnerObjCExceptionCatcher.catchException({ if !element.exists { return } + // Declared residue: a flat element query has no hierarchy for geometryless semantics to + // attach to, so frameless elements are dropped at acquisition rather than presented. let frame = element.frame if frame.isNull || frame.isEmpty { return } - let visible = isVisibleInViewport(frame, viewport) - if hint.interactiveOnly && !visible { return } - #if os(macOS) - if !visible { return } - #endif + let visible = Self.isVisibleInViewport(frame, viewport) let label = element.label.trimmingCharacters(in: .whitespacesAndNewlines) let identifier = element.identifier.trimmingCharacters(in: .whitespacesAndNewlines) let valueText = snapshotValueText(element) let elementType = element.elementType let enabled = element.isEnabled let hittable = visible && enabled && element.isHittable - let filterNode = FlatSnapshotFilterNode( - isRoot: false, - visible: visible - ) - if !flatSnapshotFilterDecision( - filterNode, - hint: hint, - visibilityPolicy: .interactiveOnly - ).include { - return - } node = RawAXNode( index: index, @@ -1631,19 +1416,4 @@ extension RunnerTests { return node } - private func isScrollableContainer(_ snapshot: XCUIElementSnapshot, visible: Bool) -> Bool { - return isScrollableContainer( - elementType: snapshot.elementType, - hasChildren: !snapshot.children.isEmpty, - visible: visible - ) - } - - private func isScrollableContainer( - elementType: XCUIElement.ElementType, - hasChildren: Bool, - visible: Bool - ) -> Bool { - return visible && hasChildren && Self.scrollContainerTypes.contains(elementType) - } } diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentation.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentation.swift index fc6884a2d4..6f9538cbe4 100644 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentation.swift +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentation.swift @@ -20,6 +20,12 @@ struct RawAXNode { let hiddenContentAbove: Bool? let hiddenContentBelow: Bool? var actions: [String]? = nil + + var hasSemanticContent: Bool { + [label, identifier, value].contains { + !($0?.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty ?? true) + } + } } /// The acquisition-facing view of a snapshot request, derived once by @@ -69,6 +75,9 @@ struct SnapshotAcquisition { let truncated: Bool let effectiveDepth: Int? var customActions: SnapshotCustomActionCoverage? = nil + /// Viewport the regular projection's clip fold runs against. `.infinite` disables the fold -- + /// legitimate only for raw acquisitions and depth-0 probes, where no fold applies. + let viewport: CGRect } /// The only snapshot node shape accepted by response payload assembly. @@ -169,14 +178,26 @@ enum SnapshotPresentation { } } - /// Visible projection: eligibility (an interactive type or non-empty semantic content, below the - /// root carrier), scope, and the acquired scroll hints. The only interpreter of what a screen - /// currently shows. + /// Visible projection: the clip fold (viewport ∩ scroll clip, ancestor cursor, scroll hints), + /// eligibility (an interactive type or non-empty semantic content, below the root carrier), and + /// scope. The only interpreter of what a screen currently shows -- no backend folds its own + /// visibility (#1797). static func presentRegular( _ acquisition: SnapshotAcquisition, - options: PresentationOptions + options: PresentationOptions, + policy: SnapshotVisibilityFold.Policy = .platformDefault ) -> SnapshotBackendCapture { - project(acquisition, options: options, projection: .regular) + project( + SnapshotVisibilityFold.fold( + acquisition.nodes, + viewport: acquisition.viewport, + interactiveOnly: options.interactiveOnly, + policy: policy + ), + acquisition: acquisition, + options: options, + projection: .regular + ) } /// Diagnostic projection: the acquired tree, normalized. Scope and depth apply when explicitly @@ -186,7 +207,7 @@ enum SnapshotPresentation { _ acquisition: SnapshotAcquisition, options: PresentationOptions ) -> SnapshotBackendCapture { - project(acquisition, options: options, projection: .raw) + project(acquisition.nodes, acquisition: acquisition, options: options, projection: .raw) } /// Derives the one acquisition-facing view of a request, so no backend re-reads @@ -206,15 +227,16 @@ enum SnapshotPresentation { } private static func project( - _ acquisition: SnapshotAcquisition, + _ projectionNodes: [RawAXNode], + acquisition: SnapshotAcquisition, options: PresentationOptions, projection: CaptureHint.Projection ) -> SnapshotBackendCapture { - let scopedRawNodes = applyScope(to: acquisition.nodes, options: options, projection: projection) + let scopedRawNodes = applyScope(to: projectionNodes, options: options, projection: projection) let nodes = presentedNodes(from: scopedRawNodes, projection: projection) let qualityPayload: DataPayload? = SnapshotScopePolicy.isActive(options.scope) ? DataPayload( - nodes: presentedNodes(from: acquisition.nodes, projection: projection), + nodes: presentedNodes(from: projectionNodes, projection: projection), truncated: acquisition.truncated ) : nil @@ -331,13 +353,6 @@ enum SnapshotPresentation { // The top-level carrier owns viewport geometry and must survive even for query-sweep's // deliberately unlabeled synthetic Application node. if node.parentIndex == nil { return true } - return eligibleInteractiveTypes.contains(node.type) || hasSemanticContent(node) - } - - private static func hasSemanticContent(_ node: RawAXNode) -> Bool { - [node.label, node.identifier, node.value].contains { value in - guard let value else { return false } - return !value.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty - } + return eligibleInteractiveTypes.contains(node.type) || node.hasSemanticContent } } diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentationTests.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentationTests.swift index ae2a858931..0bae0f2247 100644 --- a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentationTests.swift +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentationTests.swift @@ -37,7 +37,8 @@ extension RunnerTests { candidates: 2, truncated: 0, blocked: false - ) + ), + viewport: .infinite ), options: PresentationOptions( interactiveOnly: true, @@ -108,7 +109,8 @@ extension RunnerTests { SnapshotAcquisition( hint: CaptureHint( projection: .regular, depth: nil, interactiveOnly: false, customActions: false), - nodes: acquired, truncated: false, effectiveDepth: nil), + nodes: acquired, truncated: false, effectiveDepth: nil, + viewport: CGRect(x: 0, y: 0, width: 1_000, height: 1_000)), options: PresentationOptions(interactiveOnly: false, depth: nil, scope: nil, raw: false) ).payload.nodes ) @@ -117,7 +119,8 @@ extension RunnerTests { SnapshotAcquisition( hint: CaptureHint( projection: .regular, depth: nil, interactiveOnly: true, customActions: false), - nodes: acquired, truncated: false, effectiveDepth: nil), + nodes: acquired, truncated: false, effectiveDepth: nil, + viewport: CGRect(x: 0, y: 0, width: 1_000, height: 1_000)), options: PresentationOptions(interactiveOnly: true, depth: nil, scope: nil, raw: false) ).payload.nodes ) @@ -139,7 +142,7 @@ extension RunnerTests { SnapshotAcquisition( hint: CaptureHint( projection: .raw, depth: nil, interactiveOnly: false, customActions: false), - nodes: acquired, truncated: false, effectiveDepth: nil), + nodes: acquired, truncated: false, effectiveDepth: nil, viewport: .infinite), options: PresentationOptions(interactiveOnly: true, depth: nil, scope: nil, raw: true) ).payload.nodes ) @@ -148,6 +151,37 @@ extension RunnerTests { XCTAssertEqual(raw.last?.parentIndex, 9) } + func testRegularPresentationRoutesThroughVisibilityFold() { + let nodes = [ + RawAXNode( + index: 0, type: "Application", label: "App", identifier: nil, value: nil, + rect: SnapshotRect(x: 0, y: 0, width: 100, height: 100), enabled: true, + focused: nil, selected: nil, hittable: false, depth: 0, parentIndex: nil, + hiddenContentAbove: nil, hiddenContentBelow: nil), + RawAXNode( + index: 1, type: "Button", label: "Outside viewport", identifier: nil, value: nil, + rect: SnapshotRect(x: 200, y: 200, width: 40, height: 40), enabled: true, + focused: nil, selected: nil, hittable: true, depth: 1, parentIndex: 0, + hiddenContentAbove: nil, hiddenContentBelow: nil), + ] + let acquisition = SnapshotAcquisition( + hint: CaptureHint( + projection: .regular, depth: nil, interactiveOnly: false, customActions: false), + nodes: nodes, + truncated: false, + effectiveDepth: nil, + viewport: CGRect(x: 0, y: 0, width: 100, height: 100) + ) + + let presented = SnapshotPresentation.presentRegular( + acquisition, + options: PresentationOptions( + interactiveOnly: false, depth: nil, scope: nil, raw: false) + ).payload.nodes + + XCTAssertEqual(presented?.compactMap(\.label), ["App"]) + } + func testSnapshotPresentationOwnsScopeAndRelativeDepth() throws { // Non-vacuity: disconnecting applyScope produces eight scope, depth, and raw-projection failures. func node( @@ -189,7 +223,8 @@ extension RunnerTests { node(6, type: "Button", label: "Outside sibling", depth: 1, parentIndex: 0), ], truncated: false, - effectiveDepth: nil + effectiveDepth: nil, + viewport: CGRect(x: 0, y: 0, width: 1_000, height: 1_000) ) let options = PresentationOptions( interactiveOnly: true, @@ -214,7 +249,8 @@ extension RunnerTests { projection: .raw, depth: nil, interactiveOnly: false, customActions: false), nodes: acquisition.nodes, truncated: false, - effectiveDepth: nil + effectiveDepth: nil, + viewport: .infinite ), options: PresentationOptions( interactiveOnly: true, @@ -270,10 +306,11 @@ extension RunnerTests { let regularAcquisition = SnapshotAcquisition( hint: SnapshotPresentation.captureHint(for: regularRequest), - nodes: nodes, truncated: false, effectiveDepth: nil) + nodes: nodes, truncated: false, effectiveDepth: nil, + viewport: CGRect(x: 0, y: 0, width: 100, height: 100)) let rawAcquisition = SnapshotAcquisition( hint: SnapshotPresentation.captureHint(for: rawRequest), - nodes: nodes, truncated: false, effectiveDepth: nil) + nodes: nodes, truncated: false, effectiveDepth: nil, viewport: .infinite) XCTAssertNil(SnapshotPresentation.present(regularAcquisition, options: rawRequest)) XCTAssertNil(SnapshotPresentation.present(rawAcquisition, options: regularRequest)) diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotVisibilityFold.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotVisibilityFold.swift new file mode 100644 index 0000000000..69dd658c40 --- /dev/null +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotVisibilityFold.swift @@ -0,0 +1,294 @@ +import Foundation + +/// The regular projection's single visibility interpreter for every snapshot backend. It applies +/// the platform clip policy, reparents survivors, and records hidden-content hints behind one pure +/// interface so acquisition backends cannot recreate only part of the projection contract. +enum SnapshotVisibilityFold { + enum Policy { + /// An out-of-clip Cell or scroll container owns its descendants' visibility. + case cursorProjected + /// Each node is intersected independently with the viewport. + case plainViewport + + static var platformDefault: Policy { + #if os(iOS) + return .cursorProjected + #else + return .plainViewport + #endif + } + } + + /// Wire-name vocabulary corresponding to XCTest's scroll-container element types. + static let scrollContainerTypeNames: Set = ["CollectionView", "ScrollView", "Table"] + + private enum Geometry { + case geometryless + case framed(intersectsClip: Bool) + } + + private enum DescendantVisibility { + case independent + case owned + } + + private struct ProjectionCursor { + static let root = ProjectionCursor(ancestorProjectedOut: false) + + private let ancestorProjectedOut: Bool + + var isProjectedOut: Bool { ancestorProjectedOut } + + func project( + geometry: Geometry, + descendantVisibility: DescendantVisibility + ) -> ProjectionDecision { + let nodeProjectedOut: Bool + switch geometry { + case .geometryless: + nodeProjectedOut = false + case .framed(let intersectsClip): + nodeProjectedOut = !intersectsClip + } + let presentationVisible = !ancestorProjectedOut && !nodeProjectedOut + let descendantsProjectedOut = ancestorProjectedOut + || (nodeProjectedOut && descendantVisibility == .owned) + return ProjectionDecision( + presentationVisible: presentationVisible, + descendants: ProjectionCursor(ancestorProjectedOut: descendantsProjectedOut) + ) + } + } + + private struct ProjectionDecision { + let presentationVisible: Bool + let descendants: ProjectionCursor + } + private struct ProjectionTransition { + let decision: ProjectionDecision + let hiddenContentFrame: CGRect? + } + private struct BranchState { + let cursor: ProjectionCursor + let anchor: (index: Int, rect: CGRect)? + let keptIndex: Int? + let keptDepth: Int + } + + private static let negligibleDecorationTolerance = 1.0 + private static let visibilityExemptCarrierTypes: Set = ["Application", "Window"] + + static func fold( + _ nodes: [RawAXNode], + viewport: CGRect, + interactiveOnly: Bool, + policy: Policy + ) -> [RawAXNode] { + var hasChildren = [Bool](repeating: false, count: nodes.count) + for node in nodes { + if let parentIndex = node.parentIndex, parentIndex >= 0, parentIndex < nodes.count { + hasChildren[parentIndex] = true + } + } + + var states = [BranchState?](repeating: nil, count: nodes.count) + var kept: [RawAXNode] = [] + var hints: [Int: (above: Bool, below: Bool)] = [:] + + for (offset, node) in nodes.enumerated() { + let parentState = node.parentIndex.flatMap { states[$0] } + let parentCursor = parentState?.cursor ?? .root + let parentAnchor = parentState?.anchor + let rect = CGRect( + x: node.rect.x, y: node.rect.y, width: node.rect.width, height: node.rect.height + ) + let intersects = intersectsClip(rect, viewport: viewport, scrollAnchor: parentAnchor) + let transition = projectionTransition( + frame: rect, + intersectsClip: intersects, + typeName: node.type, + hasChildren: hasChildren[offset], + cursor: parentCursor, + policy: policy + ) + + let negligibleDecoration = policy == .cursorProjected + && node.parentIndex != nil + && !node.hasSemanticContent + && (rect.isEmpty + || rect.width <= negligibleDecorationTolerance + || rect.height <= negligibleDecorationTolerance) + let visible = transition.decision.presentationVisible && !negligibleDecoration + let include = shouldInclude( + node, + visible: visible, + interactiveOnly: interactiveOnly, + policy: policy + ) + + if let hiddenFrame = transition.hiddenContentFrame, let parentAnchor { + rememberHiddenContentHint(for: hiddenFrame, relativeTo: parentAnchor, hints: &hints) + } + + var keptIndex = parentState?.keptIndex + var keptDepth = parentState?.keptDepth ?? -1 + if include { + let outIndex = kept.count + let outDepth = keptDepth + 1 + kept.append( + RawAXNode( + index: outIndex, + type: node.type, + label: node.label, + identifier: node.identifier, + value: node.value, + rect: node.rect, + enabled: node.enabled, + focused: node.focused, + selected: node.selected, + hittable: node.hittable && intersects, + depth: outDepth, + parentIndex: keptIndex, + hiddenContentAbove: node.hiddenContentAbove, + hiddenContentBelow: node.hiddenContentBelow, + actions: node.actions + ) + ) + keptIndex = outIndex + keptDepth = outDepth + } + + var anchor = parentAnchor + if include, + let newAnchor = scrollContainerAnchor( + forTypeName: node.type, + hasChildren: hasChildren[offset], + visible: intersects, + frame: rect, + nodeIndex: keptIndex + ) + { + anchor = newAnchor + } + states[offset] = BranchState( + cursor: transition.decision.descendants, + anchor: anchor, + keptIndex: keptIndex, + keptDepth: keptDepth + ) + } + return applyHiddenContentHints(hints, to: kept) + } + + private static func projectionTransition( + frame: CGRect, + intersectsClip: Bool, + typeName: String, + hasChildren: Bool, + cursor: ProjectionCursor, + policy: Policy + ) -> ProjectionTransition { + if policy == .plainViewport { + return ProjectionTransition( + decision: ProjectionDecision( + presentationVisible: intersectsClip, + descendants: .root + ), + hiddenContentFrame: !intersectsClip ? frame : nil + ) + } + + let hasFrame = !frame.isNull && !frame.isEmpty + let geometry: Geometry = hasFrame ? .framed(intersectsClip: intersectsClip) : .geometryless + let ownsDescendants = hasChildren + && (typeName == "Cell" || scrollContainerTypeNames.contains(typeName)) + return ProjectionTransition( + decision: cursor.project( + geometry: geometry, + descendantVisibility: ownsDescendants ? .owned : .independent + ), + hiddenContentFrame: !cursor.isProjectedOut && hasFrame && !intersectsClip ? frame : nil + ) + } + + private static func shouldInclude( + _ node: RawAXNode, + visible: Bool, + interactiveOnly: Bool, + policy: Policy + ) -> Bool { + if node.parentIndex == nil { return true } + if policy == .plainViewport && interactiveOnly && !visible && node.type != "Application" { + return false + } + return visibilityExemptCarrierTypes.contains(node.type) || visible + } + + private static func intersectsClip( + _ rect: CGRect, + viewport: CGRect, + scrollAnchor: (index: Int, rect: CGRect)? + ) -> Bool { + guard !rect.isNull, !rect.isEmpty, rect.intersects(viewport) else { return false } + guard let scrollAnchor else { return true } + return rect.intersects(scrollAnchor.rect) + } + + private static func scrollContainerAnchor( + forTypeName typeName: String, + hasChildren: Bool, + visible: Bool, + frame: CGRect, + nodeIndex: Int? + ) -> (index: Int, rect: CGRect)? { + guard let nodeIndex, + visible, + hasChildren, + scrollContainerTypeNames.contains(typeName) + else { return nil } + return (nodeIndex, frame) + } + + private static func rememberHiddenContentHint( + for frame: CGRect, + relativeTo scrollAnchor: (index: Int, rect: CGRect), + hints: inout [Int: (above: Bool, below: Bool)] + ) { + var hint = hints[scrollAnchor.index] ?? (above: false, below: false) + if frame.maxY <= scrollAnchor.rect.minY { + hint.above = true + } else if frame.minY >= scrollAnchor.rect.maxY { + hint.below = true + } else { + return + } + hints[scrollAnchor.index] = hint + } + + private static func applyHiddenContentHints( + _ hints: [Int: (above: Bool, below: Bool)], + to nodes: [RawAXNode] + ) -> [RawAXNode] { + if hints.isEmpty { return nodes } + return nodes.map { node in + guard let hint = hints[node.index] else { return node } + return RawAXNode( + index: node.index, + type: node.type, + label: node.label, + identifier: node.identifier, + value: node.value, + rect: node.rect, + enabled: node.enabled, + focused: node.focused, + selected: node.selected, + hittable: node.hittable, + depth: node.depth, + parentIndex: node.parentIndex, + hiddenContentAbove: node.hiddenContentAbove == true || hint.above ? true : nil, + hiddenContentBelow: node.hiddenContentBelow == true || hint.below ? true : nil, + actions: node.actions + ) + } + } +} diff --git a/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotVisibilityFoldTests.swift b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotVisibilityFoldTests.swift new file mode 100644 index 0000000000..aa12fce447 --- /dev/null +++ b/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotVisibilityFoldTests.swift @@ -0,0 +1,132 @@ +#if AGENT_DEVICE_RUNNER_UNIT_TESTS +import XCTest + +extension RunnerTests { + private static func foldNode( + _ index: Int, + type: String, + label: String? = nil, + rect: SnapshotRect, + hittable: Bool = false, + depth: Int, + parentIndex: Int? + ) -> RawAXNode { + RawAXNode( + index: index, type: type, label: label, identifier: nil, value: nil, rect: rect, + enabled: true, focused: nil, selected: nil, hittable: hittable, depth: depth, + parentIndex: parentIndex, hiddenContentAbove: nil, hiddenContentBelow: nil + ) + } + + private static func folded( + _ nodes: [RawAXNode], + viewport: CGRect, + interactiveOnly: Bool = false, + policy: SnapshotVisibilityFold.Policy = .cursorProjected + ) -> [RawAXNode] { + SnapshotVisibilityFold.fold( + nodes, viewport: viewport, interactiveOnly: interactiveOnly, policy: policy) + } + + func testRegularFoldClipsScrollOverflowReparentsAndBooksHints() { + let nodes = [ + Self.foldNode(0, type: "Application", label: "App", + rect: SnapshotRect(x: 0, y: 0, width: 402, height: 874), depth: 0, parentIndex: nil), + Self.foldNode(1, type: "ScrollView", + rect: SnapshotRect(x: 0, y: 96, width: 402, height: 700), depth: 1, parentIndex: 0), + Self.foldNode(2, type: "Cell", label: "Visible row", + rect: SnapshotRect(x: 0, y: 120, width: 402, height: 52), hittable: true, + depth: 2, parentIndex: 1), + Self.foldNode(3, type: "StaticText", label: "Detail", + rect: SnapshotRect(x: 16, y: 130, width: 200, height: 20), depth: 3, parentIndex: 2), + Self.foldNode(4, type: "Cell", label: "Offscreen row", + rect: SnapshotRect(x: 0, y: 900, width: 402, height: 52), hittable: true, + depth: 2, parentIndex: 1), + Self.foldNode(5, type: "StaticText", label: "Clamped child", + rect: SnapshotRect(x: 16, y: 96, width: 100, height: 20), depth: 3, parentIndex: 4), + // An offscreen generic wrapper does not own visibility; its visible overlay survives and is + // reparented to the scroll container. + Self.foldNode(6, type: "Other", + rect: SnapshotRect(x: 0, y: 900, width: 402, height: 52), depth: 2, parentIndex: 1), + Self.foldNode(7, type: "StaticText", label: "Visible overlay", + rect: SnapshotRect(x: 16, y: 200, width: 100, height: 20), depth: 3, parentIndex: 6), + ] + let folded = Self.folded(nodes, viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) + + XCTAssertEqual( + folded.map(\.type), ["Application", "ScrollView", "Cell", "StaticText", "StaticText"]) + XCTAssertEqual( + folded.compactMap(\.label), ["App", "Visible row", "Detail", "Visible overlay"]) + XCTAssertEqual(folded.map(\.depth), [0, 1, 2, 3, 2]) + XCTAssertEqual(folded.map(\.parentIndex), [nil, 0, 1, 2, 1]) + XCTAssertEqual(folded.first { $0.type == "ScrollView" }?.hiddenContentBelow, true) + } + + func testRegularFoldKeepsWindowCarriersButNeverHittableOutsideClip() { + let nodes = [ + Self.foldNode(0, type: "Application", label: "App", + rect: SnapshotRect(x: 0, y: 0, width: 402, height: 874), depth: 0, parentIndex: nil), + Self.foldNode(1, type: "Window", label: "Second screen", + rect: SnapshotRect(x: 402, y: 0, width: 402, height: 874), hittable: true, + depth: 1, parentIndex: 0), + Self.foldNode(2, type: "Button", label: "Gone", + rect: SnapshotRect(x: 500, y: 100, width: 100, height: 44), hittable: true, + depth: 2, parentIndex: 1), + ] + let folded = Self.folded(nodes, viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) + + XCTAssertEqual(folded.map(\.type), ["Application", "Window"]) + XCTAssertEqual(folded.last?.hittable, false) + XCTAssertFalse(folded.contains { $0.label == "Gone" }) + } + + func testRegularFoldDropsSubPixelContentlessDecorationOnEveryBackend() { + let nodes = [ + Self.foldNode(0, type: "Application", label: "App", + rect: SnapshotRect(x: 0, y: 0, width: 402, height: 874), depth: 0, parentIndex: nil), + Self.foldNode(1, type: "Button", + rect: SnapshotRect(x: 0, y: 100, width: 402, height: 1), hittable: true, + depth: 1, parentIndex: 0), + Self.foldNode(2, type: "StaticText", label: "Hairline caption", + rect: SnapshotRect(x: 0, y: 200, width: 402, height: 1), depth: 1, parentIndex: 0), + Self.foldNode(3, type: "StaticText", label: "Frameless semantics", + rect: SnapshotRect(x: 0, y: 0, width: 0, height: 0), hittable: true, + depth: 1, parentIndex: 0), + ] + let folded = Self.folded(nodes, viewport: CGRect(x: 0, y: 0, width: 402, height: 874)) + + XCTAssertEqual(folded.compactMap(\.label), ["App", "Hairline caption", "Frameless semantics"]) + XCTAssertFalse(folded.contains { $0.type == "Button" }) + XCTAssertEqual(folded.first { $0.label == "Frameless semantics" }?.hittable, false) + } + + func testPlainViewportPolicyFoldsWithoutAncestorCursor() { + let nodes = [ + Self.foldNode(0, type: "Application", label: "App", + rect: SnapshotRect(x: 0, y: 0, width: 800, height: 600), depth: 0, parentIndex: nil), + Self.foldNode(1, type: "Window", label: "Offscreen window", + rect: SnapshotRect(x: 900, y: 0, width: 800, height: 600), depth: 1, parentIndex: 0), + Self.foldNode(2, type: "Cell", label: "Offscreen row", + rect: SnapshotRect(x: 0, y: 900, width: 800, height: 52), depth: 1, parentIndex: 0), + Self.foldNode(3, type: "StaticText", label: "Clamped child", + rect: SnapshotRect(x: 16, y: 100, width: 100, height: 20), depth: 2, parentIndex: 2), + ] + let viewport = CGRect(x: 0, y: 0, width: 800, height: 600) + + let regular = Self.folded(nodes, viewport: viewport, policy: .plainViewport) + XCTAssertEqual(regular.compactMap(\.label), ["App", "Offscreen window", "Clamped child"]) + + let interactive = Self.folded( + nodes, viewport: viewport, interactiveOnly: true, policy: .plainViewport) + XCTAssertFalse(interactive.contains { $0.label == "Offscreen window" }) + } + + func testScrollContainerTypeNamesMatchElementTypeSet() { + XCTAssertEqual( + SnapshotVisibilityFold.scrollContainerTypeNames, + Set(Self.scrollContainerTypes.map(elementTypeName)) + ) + XCTAssertEqual(elementTypeName(.cell), "Cell") + } +} +#endif diff --git a/docs/adr/0004-ios-snapshot-backend-strategy.md b/docs/adr/0004-ios-snapshot-backend-strategy.md index 4d6a723022..706a9f053b 100644 --- a/docs/adr/0004-ios-snapshot-backend-strategy.md +++ b/docs/adr/0004-ios-snapshot-backend-strategy.md @@ -143,6 +143,21 @@ compares the requested projection with the hint the acquisition was captured und tier with a structured `IOS_SNAPSHOT_PROJECTION_MISMATCH` failure instead of presenting it under the requested label. +The fourth semantic layer moves the clip fold itself into presentation. Acquisition backends are +fact serializers: every traversed node is emitted at raw traversal depth with its reported frame, +and `SnapshotAcquisition` carries the viewport. `presentRegular` runs the one visibility +interpreter for every backend — viewport ∩ scroll-container clip, the ancestor projection cursor +(an out-of-clip Cell or scroll container hides descendants whose clamped frames would otherwise +leak back into the viewport), the sub-pixel decoration rule, hidden-content hints booked onto +scroll anchors, and reparenting of survivors with collapsed depth. The fold also narrows the +emitted `hittable` to the clip: nothing outside its clip, and nothing without geometry, is ever +hittable regardless of what a backend reported. Platform differences are a `SnapshotFoldPolicy` +input to the shared algorithm (iOS cursor-projected; macOS/tvOS plain viewport intersection), +never a backend exception. The remaining acquisition-side narrowings are declared: the +traversal-depth budget cut, the flat query sweep's frameless-element drop (a flat query has no +hierarchy for geometryless semantics to attach to), and the private-AX bridge's device-side node +cap. + Declared residue: a regular-projection `--depth` request still cuts the traversal at that depth, while regular presentation emits collapsed depth. A node whose presented depth would be within the limit can therefore be dropped when structural wrappers put it deeper in the raw tree. The cut is