use locator inputs for observe & extract - #2654
Conversation
|
720514d to
e5dff1f
Compare
There was a problem hiding this comment.
All reported issues were addressed across 47 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
There was a problem hiding this comment.
All reported issues were addressed across 8 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
fbf39ac to
6362f37
Compare
There was a problem hiding this comment.
All reported issues were addressed across 52 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
There was a problem hiding this comment.
All reported issues were addressed across 7 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
# why
- for consistency across `act`/`extract`/`observe`
# what changed
this PR extends locator-based scoping to
instruction-based/nondeterministic `act`
- adds `ignoreLocators` to the `act` protocol options
- passes `locator` and `ignoreLocators` through act snapshot capture so
the model only sees the intended page context
- includes act locator descriptors in cache keys
- updates TS, Python, & Go SDK public APIs to accept page-bound locator
wrapper objects for `act`
- validates that supplied act locators belong to the resolved target
page before sending RPC requests
## SDK Shape
TypeScript:
```ts
await stagehand.act("Click the checkout button", {
page,
locator: page.locator("main"),
ignoreLocators: [page.locator("nav"), page.locator(".promo-modal")],
});
```
Python:
```py
await stagehand.act(
"Click the checkout button",
page=page,
locator=page.locator("main"),
ignore_locators=[page.locator("nav"), page.locator(".promo-modal")],
)
```
Go:
```go
_, err := client.Act(ctx, stagehand.ActInstruction("Click the checkout button"), &stagehand.StagehandClientActOptions{
Page: page,
Locator: page.Locator("main"),
IgnoreLocators: []*stagehand.PageLocator{
page.Locator("nav"),
page.Locator(".promo-modal"),
},
})
```
## behavioural notes:
- observed actions still execute against their `Action.selector` data
directly. Locator scoping only affects
instruction-based/nondeterministic `act`
- `page` remains optional. If omitted, Stagehand resolves the active
page & validates locators against that page
- locator descriptors include `selector` & optional `nth` internally
- `ignoreLocators` removes matched nodes & descendants from the act
planning snapshot
- act cache data now includes `locator` and `ignoreLocators`, so scoped
planning does not reuse incompatible cached actions
- TS batch callbacks now extend the existing locator option
serialization path to `act`
- Go `StagehandClientActOptions` now uses explicit client fields instead
of embedding generated protocol `ActOptions`
# test plan
- added protocol tests to cover the new act wire shape for `locator` and
`ignoreLocators`
- extension act tests cover focused locator scope, ignored locator
subtrees, and locator-filtered planning prompts
- extension cache tests cover act, observe, and extract locator
descriptors in cache data
- TS SDK tests cover act locator serialization, batch callback locator
serialization, and cross-page validation
- Python SDK tests cover public act `Locator` inputs, descriptor
serialization, and cross-page validation
- Go SDK tests cover act `*PageLocator` conversion, cross-page
validation, and generated model compatibility for the SDK/internal
packages
- docs tests cover the updated v4 act reference and examples
<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Adds locator scoping to instruction-based `act()` with a new
`ignoreLocators` option so the model plans within a focused DOM and
excludes ignored regions. Scoped `act()` bypasses server-side caching
and sends locators to snapshot capture for both planning and
verification.
- **New Features**
- Add `ignoreLocators` to `act` options; pass `locator` and
`ignoreLocators` to snapshot capture for planning and the verification
step.
- Locator-scoped instruction-based `act()` bypasses server-side result
caching and reports `metadata.cache.status: DISABLED`.
- Protocol: add `ignore_locators` to `ActOptions` in the v4 schema.
- Locator scoping applies to instruction-based `act()` only; replays of
recorded `Action`s use their own selectors.
- **SDKs**
- `sdk-ts`: `act` accepts locator wrappers and serializes them in direct
calls and callback batches (page resolution + locator serialization for
`act`, `observe`, `extract`); cross-page validation added.
- `sdk-python`: `act` accepts `locator` and `ignore_locators` as
page-bound locators with cross-page validation.
- `sdk-go`: `StagehandClientActOptions` uses flattened fields (`Page`,
`Locator`, `IgnoreLocators`, `Timeout`, `Variables`, `Model`, `Cache`);
`act` serializes `PageLocator`s with cross-page validation.
- Docs updated with `act()` scoping examples, cache bypass behavior, and
Go option shape; speed and caching guides show scoped `act()` usage.
<sup>Written for commit 6e038da.
Summary will update on new commits.</sup>
<a
href="https://cubic.dev/pr/browserbase/stagehand/pull/2655?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
why
locator's. for consistency across our APIs, this PR changes them to acceptlocator'swhat changed
selector/ignoreSelectorsobserve and extract protocol options with locator descriptor fieldsnthpage.locator(...)/page.Locator(...)scopingreturned selectors remain plain output data.
Action.selectorand snapshot selector maps are not wrapped as live locators by this PRSDK shapes
TypeScript:
Python:
Go:
behaviour notes:
pageremains optional. If omitted, Stagehand resolves the active page and validates locators against that pageselectorand optionalnthinternallyobserveresults andact(Action)inputs still use selector strings as action datatest plan
locator&ignoreLocatorsnthhandlingLocatorwrapper inputs, descriptor serialization, & cross-page validationLocatorwrapper inputs, descriptor serialization, and cross-page validation*PageLocatorconversion, cross-page validation, and generated model compatibility for the SDK/internal packagesSummary by cubic
Switch observe, extract, and act to accept page-bound locator inputs instead of selector strings, standardizing scoping, enabling precise nth targeting, and letting you exclude subtrees. Locator-scoped calls bypass server caching; SDKs validate that locators belong to the target page.
New Features
locatorandignoreLocatorsto act/observe/extract; all support.nthfor disambiguation.text=locators are not yet supported for snapshot scoping.focusLocator/ignoreLocators) and honor.nth; added targeted tests.metadata.cache.statusasDISABLED.ActOptionsaddsignore_locators; observe/extract now uselocator/ignore_locators.page.locator("...")andxpath=...; evals Vitest config updated. Go SDK: flattened per-call fields for Act options and fixed nilIgnoreLocatorshandling.Migration
selectorwithlocator: page.locator("...")andignoreSelectorswithignoreLocators: [page.locator("..."), ...]in observe/extract..nth(index)to target a specific match; includexpath=when using XPath locators.StagehandClientActOptions(and already for Observe/Extract), e.g., useCache,Timeout,Model,Locator,IgnoreLocators,Variablesdirectly.Written for commit 85138cd. Summary will update on new commits.