ci(release): advisory Playwright web-UI sweep on tag builds (QA gate T2) - #1030
Merged
Conversation
Spec 081 T2. The release QA gate now runs the Playwright Web UI sweep on every tag build (release.yml / prerelease.yml already call the reusable gate), against the Web UI served by the candidate binary with a live mcpfixture stdio upstream. Advisory by design, enforced twice: `continue-on-error: true` on the job (a failing job inside a reusable workflow would otherwise become the workflow_call conclusion the publishers gate on), and a non-blocking manifest entry `advisory/web-ui-sweep` (renamed from `reserved/web-ui-sweep`) whose failures land in `advisory_failures` via a new `release-gate run-suite --advisory` flag. Promotion to blocking is Blocking:true + dropping continue-on-error. The sweep itself is now committed (e2e/web-ui-sweep: servers list, server detail + security tab, tools page + search, activity log, settings; fails on uncaught page exceptions) and scripts/run-web-smoke.sh — previously dead, it ran a spec file that lives only in a gitignored dir — is the single launcher for both the manual run and the CI job.
Deploying mcpproxy-docs with
|
| Latest commit: |
4d35f00
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://07fbdd76.mcpproxy-docs.pages.dev |
| Branch Preview URL: | https://ci-release-qa-gate-playwrigh.mcpproxy-docs.pages.dev |
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
📦 Build ArtifactsWorkflow Run: View Run Available Artifacts
How to DownloadOption 1: GitHub Web UI (easiest)
Option 2: GitHub CLI gh run download 32687351114 --repo smart-mcp-proxy/mcpproxy-go
|
…launcher Cross-model review follow-ups on the T2 Playwright sweep. The gate runs as a reusable workflow, so its artifacts land in the publisher's own run — and release.yml/prerelease.yml build release assets by globbing every *.tar.gz/*.zip from ALL run artifacts. A Playwright HTML report keeps its traces as playwright-report/data/ <sha>.zip, retained on failure (and on a flaky-then-green retry), so a red advisory sweep on a tag would have attached unnamed trace zips to the public release. Both publishers now exclude the sweep artifact from that glob, guarded by TestPublishersDoNotShipSweepArtifacts. scripts/run-web-smoke.sh: - generate a throwaway API key instead of honouring MCPPROXY_API_KEY, which the sweep writes into every navigation URL and Playwright then stores in the HTML report and failure traces; pin it into the server's environment too, since the env var outranks the config file and an ambient key otherwise left the server demanding one key while the probe used another (46s of 401s, then a readiness timeout) - fail fast when the candidate exits before becoming ready instead of burning the full timeout, and give the readiness curl connect/max timeouts so a port held by a non-responding process cannot block the loop forever (occupied-port repro: 119s -> 5s) - preserve the server log from the cleanup trap so an early failure is diagnosable from the uploaded artifact, not just the CI console - reject fixture paths containing a quote/backslash/newline rather than silently emitting a malformed config heredoc - npm ci against a committed package-lock.json instead of resolving the mutable ^1.49.0 range during release qualification
Round-2 review follow-up. `npm ci` ran only when the playwright binary was missing, so a tree installed by an older revision of this script (unlocked `npm install`) kept serving an unreviewed version forever — the lockfile was enforced on fresh CI runners but not in existing worktrees. Compare the installed @playwright/test version against the lockfile instead of merely checking for presence: a mismatch reinstalls, a match still skips. That closes the hole without making every hand-run pay a registry round-trip, which running `npm ci` unconditionally would.
Round-3 review follow-up, closing two low-severity residuals in the install guard. Comparing only @playwright/test's version still missed a lockfile that moved a transitive dependency (playwright-core) without touching the top-level version. Hash the whole lockfile instead and stamp it into node_modules after a successful npm ci; any drift reinstalls. Verified against a transitive-only lockfile bump, which the version check would have skipped. The stamp is written only on success, so a failed npm ci leaves the previous, still-accurate stamp in place. Also pass the path through process.argv rather than interpolating it into the node -e source: a checkout under a path containing a quote produced invalid JavaScript and a silently empty result.
Cross-model review (opencode/gpt-5.6-sol) findings on the advisory Web UI sweep, both verified against the running sweep before fixing: * The throwaway instance's API key was `web-sweep-<epoch>-<pid>`. It guards the full REST admin API, which can register a stdio upstream — i.e. run an arbitrary command as whoever launched the sweep — and 127.0.0.1 is reachable by every local account, so a clock+PID key is brute-forceable for the minute the instance is up. Now 24 bytes from crypto.randomBytes. * The tools-search assertion (`count <= before` plus first-row-matches) passed vacuously when the search box did nothing: the unfiltered set is trivially <= itself and the fixture's first tool is already `echo`. It now requires every surviving row to match, which is also race-free — a strict `<` would have raced the background indexer, since `before` can be sampled mid-render. Verified non-vacuous by mutation: with the query emptied the check fails, where the old one passed. A third finding (readiness loop could latch onto a foreign server already on the port) was rejected as a false positive: the probe carries this run's unique key, and a foreign mcpproxy answers 401 — measured — while a second instance on a taken port exits 2 and is caught by the loop's liveness check.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Spec 081 T2 (roadmap
release-qa-gate-playwright): the release QA gate now runs the Playwright Web UI sweep on every tag build —release.yml/prerelease.ymlalready call the reusable gate — against the Web UI served by the candidate binary (embedded frontend, not a dev server) with a livemcpfixturestdio upstream.Advisory, enforced twice (it cannot block a release):
continue-on-error: trueon theweb-ui-sweepjob — a failing job inside a reusable workflow would otherwise become theworkflow_callconclusion the publishers'needs:gate on;advisory/web-ui-sweep(renamed fromreserved/web-ui-sweep), whose failures are recorded asadvisory-failand listed inadvisory_failuresvia a newrelease-gate run-suite --advisoryflag.Promotion to blocking (FR-016 end state) is
Blocking: true+ droppingcontinue-on-error, after 3 consecutive clean tags.Reuse over copy-paste: the job calls
scripts/run-web-smoke.sh, the same launcher a maintainer runs by hand. That script was dead — it ranweb-smoke.spec.tsfrom the gitignored.playwright-mcp/— so this PR points it at a committed sweep (e2e/web-ui-sweep: servers list, server detail + security tab, tools page + search, activity log, settings; uncaught page exceptions fail it) and adds the optional fixture upstream.Artifacts: Playwright HTML report + server log upload as
web-ui-sweep-playwright-report(14 days).Also:
verdictnowneeds:the sweep job so its fragment is merged, and docs (docs/development/release-gate.md,docs/development/web-ui-verification.md) +roadmap.yaml/ROADMAP.mdare updated.