Skip to content

feat(web): add hover command for hover-gated UI (#1783) - #1786

Merged
thymikee merged 4 commits into
mainfrom
claude/agent-device-issue-1783-9dcc8e
Aug 18, 2026
Merged

feat(web): add hover command for hover-gated UI (#1783)#1786
thymikee merged 4 commits into
mainfrom
claude/agent-device-issue-1783-9dcc8e

Conversation

@thymikee

@thymikee thymikee commented Aug 17, 2026

Copy link
Copy Markdown
Member

Closes #1783.

Summary

Adds a first-class hover <x y|@ref|selector> [--settle] command, admitted on --platform web only, so hover-gated UI (a message row's ... toolbar, menus that open on pointer enter) is reachable through agent-device the way it already was through the underlying agent-browser backend.

Shape — minimal by reuse, not by scope:

  • One new optional op pair, Interactor.hover? / AgentDeviceBackend.hover? (coordinate: agent-browser mouse move, via movePointer extracted from clickCoordinates) plus WebProvider.hoverRef / AgentDeviceBackend.hoverTarget (ref: agent-browser hover <ref-selector>). Only the web provider implements them; touch platforms need no stubs.
  • Rides the existing targeted-touch pipeline: same @ref/selector/coordinate targeting, occlusion/off-screen guards, staleness, --settle (settle-only, no --verify — hover reveals rather than activates), response builder (gesture: 'hover'), event timeline, MCP schema, Node client (interactions.hover), and script recording (hover @ref publishes as a portable selector line).
  • hover @ref takes the ADR 0011 native-ref path exactly like click @ref/fill @ref (web ref frames carry no rects, so a coordinate route could never resolve them). The shared preflight → handle-dispatch → exact-ref envelope is extracted into dispatchNativeRefInteraction and used by tap, fill, and hover so the three cannot drift; the guarantee matrix native-ref row lists hover.
  • Web-only by capability the same way viewport is (capability: { apple: {}, android: {}, linux: LINUX_NONE } + WEB_INTERACTION_COMMANDS): capabilities --platform web lists it; iOS/Android/Linux reject at admission with UNSUPPORTED_OPERATION and a --platform web hint. longpress stays the mobile hold-gesture verb.
  • Selector/coordinate hover uses the resolvedTarget pipeline (no hittable-ancestor promotion): the pointer only has to enter the matched node's box.

Help (overview, help web, help hover), commands.md, and CHANGELOG are updated so help/capabilities agree with what the CLI does. No skill files are touched.

Live validation

Real CLI + managed agent-browser against a fixture reproducing the issue (chat rows whose Delete button is display:none until :hover):

hover 'text="Second message"' --settle
  → Hovered text="Second message" (155, 172)
    settled after 757ms: +1 -0
    + @e4~s971890 [button] "Delete"
click 'text="Delete"'  → "Deleted second message" (screenshot-confirmed)

hover @e3              → Hovered @e3; next snapshot -i shows @e4 [button] "Delete"

Known, disclosed, pre-existing: hover @ref --settle on web fails the same way click @ref --settle does today (settle needs a pre-action baseline → runtime resolution → rect-less web frame). Docs/help say to pair --settle with a selector or coordinates on web; fixing that is a web ref-frame change that applies to click too, out of scope here.

Test plan

  • pnpm typecheck, lint, format:check, check:fallow, check:layering, check:command-docs, check:daemon-wire-compat, check:gate-manifest, check:bundle-owner-files
  • pnpm test:unit (876 files / 6744 tests) — pinned enumerations updated (descriptor parity, timeout policy, post-action observation, capabilities, e2e coverage manifests + Android summary pin, replay arbitraries, command-result map, apple output guard DRIVEN_COMMANDS)
  • test:integration:node 55, test:integration:provider 52 files / 157 (web desktop scenario now drives hover @ref → provider hoverRef), test:integration:progress:check (0 missing commands), test:smoke 38, interaction-contract 73; test:coverage changed-line gate PASS 85.19% (threshold 70%)
  • New tests (each with planted-red evidence in the PR comments): agent-browser provider hover → bare mouse move, hoverRefhover <ref>; web interactor delegation + provider-missing refusal; daemon hover @ref on a rect-less web frame takes the provider hoverRef route with no coordinate dispatch; daemon hover <selector> resolves and dispatches coordinate hover; capability refusal on iOS before dispatch; capabilities matrix web-only; recorded hover @ref → selector line; direct-dispatch handleHoverCommand
  • Live web run above

Add a first-class `hover <x y|@ref|selector> [--settle]` verb, admitted on
web only, that moves the pointer without pressing via the agent-browser
backend (mouse move). It rides the existing targeted-touch pipeline
(ref/selector/coordinate resolution, occlusion/off-screen guards, settle
observation, response builder, recording) through a new optional
Interactor/backend `hover` op that only the web provider implements.

Touch platforms have no hover state: capabilities advertise it on web
only and iOS/Android/Linux reject it at admission with a --platform web
hint; longpress stays the mobile hold-gesture verb.

Closes #1783
@github-actions

github-actions Bot commented Aug 17, 2026

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-08-18 09:33 UTC

@github-actions

github-actions Bot commented Aug 17, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 2.26 MB 2.27 MB +4.5 kB
JS gzip 744.0 kB 745.1 kB +1.1 kB
npm tarball 862.7 kB 863.0 kB +294 B
npm unpacked 3.01 MB 3.01 MB +1.7 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 27.4 ms 27.6 ms +0.1 ms
CLI --help 66.1 ms 66.2 ms +0.1 ms

Top changed chunks:

Chunk Raw diff Gzip diff
dist/src/registry.js +1.1 kB +272 B
dist/src/cli-help.js +563 B +213 B
dist/src/prepare-kind.js +991 B +181 B
dist/src/interaction2.js +468 B +160 B
dist/src/dispatch.js +606 B +136 B

@thymikee

Copy link
Copy Markdown
Member Author

Review findings — changes requested before merge:

  1. P1: hover @ref is advertised but cannot run against a real web snapshot. The new CLI/help/docs contract says hover accepts coordinates, @ref, or a selector, and the guarantee matrix classifies runtime-ref as enforced. However, web snapshot -i refs are minted from a ref-frame snapshot without rects; resolveSnapshotForRef authorizes that frame and deliberately does not replace it with a fresh capture. The new hover backend accepts only coordinates; there is no provider/native hoverRef route. Thus hover @ref reaches Ref … not found or has invalid bounds. The new daemon test fabricates a web ref with a rect, so it does not exercise the shipped agent-browser route. Either add real web ref-hover support at the snapshot/provider boundary with a production-faithful regression, or stop advertising/classifying @ref support for hover.

  2. P1: Android coverage pin is stale. Adding hover as a capability-denial at test/integration/android-emulator-e2e/coverage-manifest.ts:198 changes the derived summary to capabilityDenial: 3 and total: 54, while smoke-android-emulator-coverage.test.ts still expects 2/53. Android Smoke is red on exactly this assertion; the Integration Tests job is also red.

  3. P2: Do not update skills/agent-device/SKILL.md without explicit authorization. The repository AGENTS.md hard rule prohibits behavior/workflow edits under skills/**/SKILL.md unless the user explicitly asks. web: no hover verb - hover-gated UI unreachable despite agent-browser supporting hover #1783 calls for CLI/help/capability alignment, so please revert the skill change unless separately authorized.

Please also include the required planted-red/revert evidence for each new regression test in the PR validation. Remaining iOS Smoke and Coverage lanes were pending when reviewed.

… revert skill edit

Review follow-ups on #1786:
- hover @ref on web now dispatches through the provider's own element handle
  (agent-browser `hover <ref>`) via a new backend hoverTarget, mirroring
  click/fill's ADR 0011 native-ref path — web ref frames carry no rects, so
  the coordinate route could never resolve them. The shared preflight +
  exact-ref dispatch is extracted into dispatchNativeRefInteraction and used
  by tap/fill/hover; the guarantee matrix native-ref row now lists hover.
- Daemon regression test is production-faithful: rect-less web ref frame,
  scoped provider, asserts no coordinate dispatch. Selector→coordinate and
  provider hoverRef tests added.
- Android emulator coverage summary pin 2/53 → 3/54.
- skills/agent-device/SKILL.md reverted (out of scope, AGENTS.md rule).
- Docs/help disclose that --settle with @ref on web shares click's existing
  limitation; use a selector or coordinates for the settled diff.
@thymikee

Copy link
Copy Markdown
Member Author

Addressed in fdebd81:

1. hover @ref on web — fixed with a real ref route, not a contract downgrade. You're right that the web ref frame carries no rects, so the coordinate route could never resolve hover @ref (the same reason press @ref / click @ref --settle fail on web today). Hover now has the same ADR 0011 native-ref path as click/fill: WebProvider.hoverRef → agent-browser hover <ref-selector>, exposed as backend.hoverTarget, taken by hoverCommand under the same conditions as maybeTapRefTarget (ref target, backend capability, no replay guard, no settle baseline). The preflight-guards → handle-dispatch → exact-ref envelope was extracted into dispatchNativeRefInteraction and is now used by tap, fill, and hover so the three can't drift. The guarantee matrix native-ref row lists hover.

  • Regression is production-faithful now: a web session whose stored frame has no rects, a scoped WebProvider with hoverRef, asserting the provider route is taken and dispatchCommand is never called (interaction-touch.test.ts). Added a selector→coordinate daemon test and a provider test pinning hoverRef['hover', '@e5', …].
  • Live: hover @e3 on the fixture page now returns Hovered @e3 and the next snapshot -i shows the revealed [button] "Delete".
  • Disclosed limitation (not new): hover @ref --settle on web fails exactly like click @ref --settle does today (settle needs a pre-action baseline → runtime resolution → rect-less frame). Docs/help now say to pair --settle with a selector or coordinates on web. Fixing that gap is a web ref-frame change (fetch boxes) that applies to click too — out of scope here.

2. Android coverage pin — updated to capabilityDenial: 3, total: 54; pnpm test:smoke 38/38 green.

3. SKILL.md — reverted, sorry; only CLI/help/capabilities/docs are touched.

Planted-red evidence (each new regression test, production seam removed → red, restored → green; working tree diff verified unchanged after):

Test Planted break Result
hover @ref on web dispatches through the provider hoverRef route hoverTarget wiring removed from interaction-runtime.ts 1 failed
hover selector on web resolves … coordinate hover daemon backend hover dispatches 'press' instead of 'hover' 1 failed
hover is refused by capability on touch platforms + hover is admitted only on web registry admits apple: APPLE_SIM_AND_DEVICE 1 failed each
provider maps supported operations (hover = bare mouse move, hoverRef = hover <ref>) provider hover uses clickCoordinates, hoverRef uses click 1 failed
web interactor delegation + provider-missing refusal interactor hover routes to provider().click 2 failed
recorded hover @ref publishes as selector line writer's hover branch removed 1 failed
post-action observation pin hover: 'settle' removed fails at import: Missing post-action observation descriptor support for hover

Full sweep at fdebd81: typecheck / lint / format / fallow / layering / test:unit 875 files (6741) / test:smoke 38 / interaction-contract 73 — all green.

@thymikee

Copy link
Copy Markdown
Member Author

Follow-up review at fdebd814: the @ref native-hover route, Android summary pin, SKILL.md revert, and planted-red evidence are now addressed. One merge blocker remains: add hover to DRIVEN_COMMANDS in test/integration/provider-scenarios/apple-platform-output-guard.test.ts (preferred, so its unsupported Apple response is scanned for internal Apple-platform leakage), or to SKIPPED_COMMANDS with a concrete harness rationale. The exhaustive partition currently omits it, making Integration Tests and Coverage fail with Public command "hover" is neither driven nor skipped; Coverage then cannot produce lcov as a downstream result. Please also update the stale PR body: it still says hover @ref fails and that skills/agent-device/SKILL.md changed.

…dispatch

The provider-integration apple-leak guard partitions every public command
into driven/skipped; hover was neither, which failed Integration Tests and
took Coverage down with it. Drive it (it reaches the Apple capability
refusal, which is scanned like any other error response). Also cover the
direct-dispatch handleHoverCommand seam.
@thymikee

Copy link
Copy Markdown
Member Author

CI red lanes fixed in e95941d — one root cause: the provider-integration apple-leak guard (apple-platform-output-guard.test.ts) partitions every public command into driven/skipped and hover was neither. That failed Integration Tests directly and killed the Coverage run before lcov existed (its "no lcov report" was the downstream symptom). hover is now driven (10 10, reaching the Apple capability refusal, whose error response is scanned like the rest).

Also added dispatch-hover.test.ts for the direct-dispatch handleHoverCommand seam (pointer move + point in the result, unsupported-platform refusal, coordinate-requirement hint) — planted-red: swapping the handler to tap/a different message → 2 failed, restored → 3 passed.

Local: test:unit 876 files (6744), test:integration:provider 52 files (157), test:coverage 940 files green with changed-line gate PASS at 85.19% (threshold 70%), typecheck / lint / format / fallow green.

The integration-progress gate requires every public command to be referenced
by a provider-backed scenario. Add hover @ref to the web desktop flow: it
must reach the provider's hoverRef handle (never a coordinate) and be
recorded on the session without fabricated x/y, like click @ref.
@thymikee

Copy link
Copy Markdown
Member Author

Follow-up on the remaining blocker + the lane that turned red after it:

  • hover was added to DRIVEN_COMMANDS in apple-platform-output-guard.test.ts in e95941d (its Apple UNSUPPORTED_OPERATION response is scanned for leaks). That unblocked the guard, and Integration Tests then hit the next gate in the same job, integration-progress ("missing Provider-backed integration command coverage: hover"). 134115c addresses it with a real scenario rather than a token reference: the web desktop provider scenario now drives hover @e4 and asserts it reaches the provider's hoverRef handle (['web','hoverRef','@e4']), returns Hovered @e4 with no fabricated x/y, and is recorded on the session that way — mirroring the existing click @e4 step. Locally: test:integration:node 55, test:integration:provider 52/157, test:integration:progress:check 0 missing, plus typecheck/lint/format/fallow.
  • Bundle Size / deploy-preview / CodeQL reds on the previous run were GitHub 503s at their comment/deploy steps, not test failures; they should clear on rerun.
  • PR body updated: no longer says hover @ref fails or that SKILL.md changed.

@thymikee

Copy link
Copy Markdown
Member Author

Clean review at 134115c: the prior findings are resolved. The Apple output guard now drives hover, and the provider-backed web scenario exercises native hover @ref through hoverRef (without fabricated coordinates). Integration Tests and Coverage are green.

The only red check, deploy-preview, completed the gh-pages deployment successfully and then failed while posting its sticky comment with a GitHub 503. Please rerun that check; no ready label until it is green.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Aug 18, 2026
@thymikee
thymikee merged commit 8db3629 into main Aug 18, 2026
34 of 35 checks passed
@thymikee
thymikee deleted the claude/agent-device-issue-1783-9dcc8e branch August 18, 2026 09:32
thymikee added a commit that referenced this pull request Aug 18, 2026
`boot` and `shutdown` never consulted the host-global device claim store, so a
daemon in one state directory could terminate an emulator another daemon held a
verified-live claim on and report success (#1799).

Rather than adding a claim check to those two handlers, this makes the class
unrepresentable: `CommandDescriptor` gains a REQUIRED `deviceClaimPolicy` trait
(#1320's vocabulary), and the request-execution scope enforces it where the
request runtime bindings create a device binding — the one seam through which
any handler can obtain device operations, and already the place per-device
deduplication lives. A `transient-exclusive` command acquires a command-scoped
claim before operations reach the handler, refuses a foreign live claim with the
existing DEVICE_IN_USE/DEVICE_CLAIM_LIVE_OWNER error, and releases in the
scope's finally. Every other policy performs no claim-store I/O, so session-bound
commands keep #1320's non-goal intact.

`hover` also gains the `frameworkTier` it was missing: #1786 landed after
#1804's branch point, leaving main's own parity gate red.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

web: no hover verb - hover-gated UI unreachable despite agent-browser supporting hover

1 participant