Add adversarial browser environment matrices#2063
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
wp-codebox/browser-environment-matrix/v1contract with canonical cartesian expansion, stable per-cell seeds/identity, required and optional capability negotiation, and sequential bounded executionFixes #2024
Contract and expansion semantics
Dimensions and values are sorted by stable ids before cartesian expansion, independent of declaration or object insertion order. Cell ids and seeds derive from the matrix seed, canonical requested environment, and selected dimension values. Conflicting assignments fail validation, while independent capability state merges deterministically.
The runner composes existing browser probe, action, adaptive, visual, accessibility, and multi-actor executors through an execution callback rather than duplicating those systems. Execution is sequential by default. Environment cell identity participates in default finding fingerprints and replay metadata.
Bounds and resources
Construction rejects matrices exceeding
maxCellsor whose reservedcells * maxCellDurationMsexceedsmaxDurationMs. Runtime execution separately enforces per-cell time, total wall time, interruption, and aggregatemaxArtifactBytes; all completed-cell evidence is retained when a later cell or resource limit fails.browserEnvironmentMatrixFailed()preserves fail-on-finding behavior while allowing explicitly advisory matrices.Artifacts and provenance
Each run reserves
browser-matrices/<matrix-id>/<run-id>/exclusively and rejects an existing namespace. Cells write beneathcells/<index>-<cell-id>/, andmatrix-report.jsonrecords cell identity, requested/effective environment, provider/browser/channel/version, capability fidelity, status/findings, timing, artifact refs, unsupported/inconclusive dimensions, and replay command/contract.Unsupported capability handling
Required unsupported capabilities fail closed without executing the cell. Optional unsupported capabilities retain safe execution and mark an otherwise passing cell inconclusive. Zoom and CPU/network debugging-protocol controls are labeled
emulated; unknown device/resource profiles areunsupported. OS/browser-chrome zoom, server clocks, and socket transport faults are not claimed.Tests
Passed:
npm run buildnpm run test:browser-environment-matrix(10 tests, including real-browser environment application and artifact-collision fixtures)npm run test:adversarial-runtimenpm run test:browser-scenariosnpm run test:browser-visual-compare-capture-reliabilitynpm run test:schema-paritynpm run test:recipe-validation-descriptorsnpm run test:public-api-contractnpm run test:production-boundary-enforcementRepository check baseline failures, unrelated and unchanged by this PR:
npm run checkreachescommand-registry-smokeand fails onwordpress.collect-workload-result outputShape should mention outputSchema id; tracked in command-registry-smoke fails on main: wordpress.collect-workload-result outputShape missing outputSchema id mention #1745, and both the registry and smoke files are unchanged.npm run test:docs-boundary-languagefails on existingagents/...examples indocs/agent-task-reusable-workflow.md; tracked in test(docs): align boundary-language gate with existing reusable-workflow paths #2061, andgit diff --exit-code origin/main -- docs/agent-task-reusable-workflow.mdpasses.