Skip to content

test(conformance): run the WDIO service through the suite - #433

Merged
vishnuv688 merged 10 commits into
mainfrom
feat/300-wdio-scenario
Oct 19, 2026
Merged

vishnuv688 merged 10 commits into
mainfrom
feat/300-wdio-scenario

Conversation

@vishnuv688

@vishnuv688 vishnuv688 commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

Stacked on #432 (Nightwatch scenario); base is feat/300-nightwatch-scenario. Part of #300.

What

The WDIO service now runs through the conformance suite in both modes, from packages/conformance/scenarios/wdio/, under its own CLI and attached to the recording backend through the reuse handshake. All four adapters are green, with no new allowlist entries.

What the first runs found

Adapter bugs, fixed here in @wdio/devtools-service (changeset included):

  • The config frame was published from beforeSession on a capturer with no socket, so it was dropped every run; only a backend the launcher itself starts, which inherits the path from the environment, ever learned the file. It is now sent from before().
  • A failed command's row reached the live socket as error: {}: the raw Error was stored and its message is non-enumerable. The row now carries a serialized error with the browser's own message; webdriver's WebDriverError: <message> when running "<endpoint>" with method "<M>" decoration is stripped, since the other adapters carry the bare message.
  • setValue/addValue rows carried the selector only as a leading argument; they now also stamp CommandLog.selector like every other adapter.

Suite assumptions, fixed in packages/conformance:

  • The fake browser answers what WebdriverIO routes over BiDi once a session offers it (browsingContext.navigate, browsingContext.getTree, browsingContext.locateNodes, script.callFunction), from the same two-page model as the classic routes.
  • run-controls-are-served applies the dashboard's own defaults (silence advertises every control) through resolveRunCapabilities, now in @wdio/devtools-shared with the app delegating to it, and accepts the config frame as what serves them.
  • The checker refuses a failed row whose error has no message instead of crashing in the normaliser; a fill's text is read as the trailing argument.

Toolchain

  • The scenario runs on the workspace's WebdriverIO 9, the same pins as examples/wdio, with expect-webdriverio declared so pnpm's peer auto-install does not pick a pre-release. A WebdriverIO 10 run cannot coexist in this workspace: @wdio/utils resolves the runner, framework and services from its own location, and both pnpm's hidden hoist and the root node_modules sit on that path, so whichever major is hoisted or declared at the root wins for every project. Moving the repo to 10 is its own change; the wdio-peer job already exercises the service under webdriverio 10.
  • A scenario launch may name a working directory, since WebdriverIO's CLI falls back to it when resolving plugins.

Verification

  • pnpm build, pnpm lint, pnpm test are green (201 files, 2596 tests), including the conformance suite for all four adapters in both modes and the cross-adapter comparison.
  • Regression tests: the config frame (packages/service/tests/index.test.ts), the error on the wire (packages/service/tests/session.test.ts, packages/service/tests/command-error.test.ts), the selector stamp (packages/service/tests/command-selectors.test.ts), the fake's BiDi path (packages/conformance/tests/fakes.test.ts), the rule changes (packages/conformance/tests/checker.test.ts, packages/conformance/tests/agreement.test.ts), shared's resolution (packages/shared/tests/run-capabilities.test.ts).
  • Not covered by the scenario: console and network capture, trace zips, per-test slicing, screenshots and assertions.

@greptile-apps

greptile-apps Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge with PR #434’s dependency cleanup.

Summary

Adds the WDIO service to the conformance suite with and without BiDi.

  • The conformance suite runs the WDIO service through its own CLI.
  • The fake browser models WebdriverIO's BiDi page actions.
  • Service command rows keep the browser's errors and selectors.
  • The attached dashboard receives the WDIO config file for reruns.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Harness[Conformance harness] --> CLI[WDIO CLI]
  CLI --> Service[Devtools service]
  CLI --> Fake[Fake WebDriver and BiDi browser]
  Service --> Backend[Recording backend]
  Fake --> Truth[Browser actions and documents]
  Backend --> Checker[Rules and adapter comparison]
  Truth --> Checker
Loading

Reviews (5) · Last reviewed commit: "chore(conformance): run the WDIO scenari..." · Reviewed by Greptile

Comment thread pnpm-lock.yaml Outdated
The dashboard's per-runner defaults and the declared-over-defaults rule
lived in the app's sidebar, where nothing outside it could apply them.
The conformance suite needs the same rule to know which controls a
stream advertises, so the tables and the resolution move to shared and
the app delegates to them. The runner lookup moves with it: the typed
runner first, then the older options.framework.
…ard reads them

Three rules read the stream in a way only the first two adapters met.
Run controls: the dashboard offers every control a runner's defaults
allow unless metadata turns it off, so silence advertises all of them,
and the WDIO service serves its config through a separate config frame
rather than metadata.options.configFile; the rule now applies shared's
resolution and accepts either carrier, so a declared set is read over
those defaults and a control is absent only when declared false. Fills:
the WDIO service also leads a fill with its selector, which reading the
text as the trailing argument already handles; a test pins it.
Errors: a row whose error has no message is refused where the stream is
read, instead of crashing the normaliser.
Once a session offers a BiDi socket, WebdriverIO routes navigation,
context lookup, element search and script calls over it. The fake
acknowledged those with an empty result, so the browser never left
about:blank and the run collapsed. The endpoint now answers every
command from the same two-page model the classic routes use, stamps a
navigation as the same user action, and refuses /broken with the same
message, so an adapter is judged on what it captured rather than on
which protocol its client picked.
beforeSession published the config file on the placeholder capturer the
plugin starts with, which has no socket, so the frame was dropped on
every run. A backend the launcher itself starts inherits the path from
the environment, which hid the loss; a dashboard the service attaches
to, including the one a rerun reports into, never learned the file. The
frame now goes out from before(), on the capturer that connects, and the
publish lives beside the detector it uses.
The row stored the thrown Error as is. Its message is non-enumerable, so
the live socket received error: {} and the Errors tab had nothing to
show. The error is now serialized at the row, and webdriver's
decoration, "WebDriverError: <message> when running <endpoint> with
method <M>", is stripped back to the browser's own text, which is what
the other adapters carry. The serialized shape is shared's; core's copy
becomes a re-export.
setValue and addValue rows carried the selector only as a leading
argument, which the trace reads positionally, while the other adapters
put it on CommandLog.selector. The row now carries both, so a reader of
the shared field sees the same target from every adapter.
The service runs under its own CLI from scenarios/wdio, attached to the
recording backend through the reuse handshake, in both modes. The CLI
resolves its runner, framework and service from the working directory,
so a scenario launch may name one. WebdriverIO 10 could not coexist with
the version 9 examples under pnpm's hidden hoist, because @wdio/utils
resolves plugins from its own location before the project's; the
workspace no longer hoists @wdio packages, so each project resolves its
own. Pinned at 10.0.1: the registry view pnpm resolves from does not yet
list 10.0.2, and the 10.0.2 CLI pins dependencies it cannot resolve.
test:ui and the demo:wdio scripts run the root wdio binary against
configs that name the browser runner, the local runner, the mocha and
cucumber frameworks, the spec reporter and the devtools service. Those
were declared only in packages/app and examples/wdio and reached the CLI
through pnpm's hidden hoist, which the workspace no longer applies to
@wdio packages; CI's component tests failed with "Couldn't find plugin
browser runner". The root now declares what it loads, which is where
the CLI's fallback resolution looks.
Given a result, WebdriverIO reads a navigation id from it and then waits
for the network events of that navigation, which the fake browser never
emits; version 9 waits even when the id is null. With no result at all,
neither version waits, and the row the adapter captured is what gets
judged.
…rIO 9

A second WebdriverIO major cannot be isolated inside this workspace.
@wdio/utils resolves the runner, framework and services from its own
location before the project's, and both pnpm's hidden hoist and the
root node_modules sit on that path: excluding @wdio from the hoist broke
test:ui and the demos, which load their plugins through it from the root,
and declaring those plugins at the root handed the version 9 runner back
to the version 10 CLI. The scenario therefore runs on the pins the
examples use, with expect-webdriverio declared so peer auto-install does
not pull a pre-release that drags webdriverio 10 into the graph. The
wdio-peer job keeps exercising the service under webdriverio 10.
@vishnuv688
vishnuv688 force-pushed the feat/300-wdio-scenario branch from f3eb951 to 1693dcc Compare October 19, 2026 12:08
@vishnuv688
vishnuv688 changed the base branch from feat/300-nightwatch-scenario to main October 19, 2026 12:29
@vishnuv688
vishnuv688 merged commit 67ea1d0 into main Oct 19, 2026
9 checks passed
@vishnuv688
vishnuv688 deleted the feat/300-wdio-scenario branch October 19, 2026 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant