ci: move parked device replay suites to a dispatch-only workflow (#1781 A1) - #1794
Conversation
Size Report
Startup median (7 runs, lower is better):
Top changed chunks: no changes in the largest emitted chunks. |
… A1)
Both full-tier device jobs have failed every scheduled run since 2026-07-24: the
Android suite inside full-tier scenarios that had never executed end to end, the
iOS suite on varying steps. They move to .github/workflows/replays-manual.yml,
which has no `schedule:`, so the schedule stops emitting a guaranteed failure while
the suites stay runnable on demand.
A job-level `if: github.event_name == 'workflow_dispatch'` would have looked the
same and lied: `workflowLanes()` decides `qualifying` per workflow FILE and never
reads job-level `if:`, so the manifest kept reporting replay-android, replay-ios,
and replay-ios-device as scheduled-lane owners — the silent-owner-loss failure the
manifest exists to catch. A separate file is what the file-level model already
reads correctly.
Those three checks now have no pull_request/schedule owner, so they are declared as
MANUAL_ONLY_OWNERS rather than folded into UNPROVABLE_OWNERS, whose claim ("it runs,
this loader cannot see it") is no longer true for replay-android. check:gate-manifest
drops from 48 to 46 wired checks and names the three on every run. Two tests pin it:
a dispatch-only lane is non-qualifying however many gates it declares, and every
manual-only declaration must name a registered check that no qualifying lane owns, so
a re-scheduled lane cannot keep a stale exemption.
a6e8c6c to
2198ff5
Compare
thymikee
changed the title
|
P1: MANUAL_ONLY_OWNERS is currently only a negative allowlist. The new test proves each entry is registered and has no qualifying owner, but it never proves a workflow_dispatch lane still declares and can run that check. Deleting a parked manual job, or its run-gate step, would therefore leave check:gate-manifest green and continue to print the check as manual-only: parked coverage has silently become deleted coverage. Please derive/attest each manual-only declaration from a dispatch-only lane and add a regression that deleting its declaration fails. Android needs an explicit attestation too: replay-android remains inside the opaque third-party action, so the existing loader cannot supply it by itself. The loss of scheduled replay coverage itself matches #1781 A1 (manual means off). The current Bundle Size and CodeQL failures are GitHub 503 infrastructure failures; the wording/count issue in the manifest output is non-blocking. |
|
CI status on this head: 25 pass, 2 fail. Both failures are |
Review P1: MANUAL_ONLY_OWNERS was a negative allowlist — it proved each entry named a registered check no qualifying lane owned, but nothing tied the entry to a lane that can still run it. Deleting a parked job, or its run-gate step, would have left the manifest green and still printing the check as manual-only: parked coverage silently becoming deleted coverage. Each entry now names its dispatch lane, and a new 'manual-only' audit assertion resolves that name against the derived model: the lane must exist, must still be dispatch-only, and must still declare the gate. replay-android carries an explicit `opaque` flag because its gate sits inside the third-party emulator action's `script:` (#1429), so the job's existence is the whole attestation the model can make — and the flag says so rather than letting an unreadable lane look like a declaring one. Four regressions pin both directions: deleting a declaration reports the check as unowned; deleting the parked job fails with 'no workflow defines'; re-scheduling the lane fails until the entry is dropped; and a parked lane that loses its run-gate step fails unless the entry is opaque.
|
P1 addressed in
Android gets the explicit attestation you asked for. Its entry is marked Four regressions pin both directions (
I checked the new ratchet is not vacuous by planting
On sequencing: #1793 now has a green end-to-end full-tier run, including |
|
Re-reviewed exact head 643bc9c. P1 remains: the new audit proves only that a declared owner has no PR/schedule-qualified lane; it does not prove the workflow is still |
…1781 A1) Review follow-up: the attestation checked `qualifying === false`, which is true of any lane that is not pull_request/schedule. Swapping `workflow_dispatch` for `push` in replays-manual.yml would have kept the audit green and the checks printed as manual-only, while the runs nobody starts by hand quietly started themselves on every push. The lane model now keeps the trigger names instead of collapsing them into that one bit, and the manual-only assertion requires `workflow_dispatch` and nothing else. Three planted regressions cover the gap the review named: a parked lane re-triggered by `push` fails, a parked lane with no trigger at all fails, and the loader test pins that trigger kinds survive into the model (a push lane reads `[push]`, the nightly reads `[schedule, workflow_dispatch]`).
|
Fixed in The lane model now preserves the trigger names instead of collapsing them ( Three regressions cover exactly the gap you named (
|
thymikee
added
the
ready-for-human
|
Re-reviewed exact head b94f93d. Clean readiness verdict: the lane model now preserves trigger kinds, the manual-only audit requires |
|
1 participant
Summary
#1781A1 fallback.Replay Nightlyhas failed every scheduled run since 2026-07-24. Two jobs are responsible and neither has an owner:full:lifecycle-system, a scenario that had never executed end to end. Diagnosis and a validated fix for it (and forfull:observability-artifacts) are in test(android): restore full-tier lifecycle and observability scenarios (#1781 A1) #1793; even with that, the last scenariofull:fixture-replaysstill fails on a replay fixture whose scroll canary is authored for a taller device than thepixel_7profile the lane pins.Both jobs move, byte-for-byte, into
.github/workflows/replays-manual.yml, which has noschedule:. The parser fuzz lane keeps its nightly schedule inreplays-nightly.ymland is untouched; the macOS job is already gone (#1787). Nothing about how the suites run changes — dispatch them with:Why a separate file rather than
if: github.event_name == 'workflow_dispatch'The first revision of this PR used a job-level
if:. It would have parked the jobs and lied about it:workflowLanes()(scripts/gate/workflows.ts) computesqualifying = 'pull_request' in on || 'schedule' in ononce per workflow file and never reads job-levelif:. Sopnpm check:gate-manifestkept printingwhile
replay-android,replay-ios, andreplay-ios-device— declared only in those two jobs — no longer ran on any schedule. That is exactly the failure the manifest exists to catch: "a check that silently loses its owner looks exactly like a green build" (docs/agents/testing.md). A separateworkflow_dispatch-only file is what the existing file-level model already reads correctly.What that exposed, and how it is declared
With the jobs moved, the audit reports the truth:
These are declared in
scripts/gate/declarations.tsas a newMANUAL_ONLY_OWNERSrecord rather than being folded intoUNPROVABLE_OWNERS, because the two facts are opposites:UNPROVABLE_OWNERSclaims "the suite runs; this loader cannot see it" — which stopped being true forreplay-androidthe moment its lane stopped running automatically, so its stale prose moved with it.UNPROVABLE_OWNERSis now empty, with a comment saying why.MANUAL_ONLY_OWNERSsays "nothing runs this until someone dispatches it", with the reason and the exit condition per entry.check:gate-manifestnow reports:Named, not counted — the ids read like a list of things nothing runs, where "3 manual-only" would read like a tally.
Each entry names the dispatch lane that still runs it, and a
manual-onlyaudit assertion resolves that name against the derived model: the lane must exist, must still be dispatch-only, and must still declare the gate. Without that the record would be a plain allowlist — deleting a parked job would leave the manifest green and still printing the check as "manual-only", turning parked coverage into deleted coverage.replay-androidcarries an explicitopaqueflag, because its gate sits inside the third-party emulator action'sscript:(#1429): the job's existence is the whole attestation the model can make, and the flag says so instead of letting an unreadable lane look like a declaring one.Six regressions pin the path (
pnpm check:gate-manifest:test, 34/34):workflow_dispatch-only lane reads its gate but is non-qualifying;no workflow defines;run-gatestep fails — unless the entry is opaque, which is asserted too, so the flag cannot wave a readable lane through.Both ratchets were checked for vacuity by planting
layering(a genuinely owned check) in the record and watching the suite go red.docs/agents/testing.mdrecords the attestation and whyreplay-androidis opaque.Validation
pnpm check:toolingpasses (format, lint, typecheck, layering, depgraph, gate-manifest + its tests, production-exports, tmpdir-leaks, mcp-metadata, build, bundle-owner-files, package). The moved job bodies are byte-identical to the originals — only the job ids changed (nightly-android/nightly-ios→manual-android/manual-ios); job names, steps, pins, env, and artifact names are unchanged.Notes for whoever picks these up
full:lifecycle-system/step: wait for Automation lab(run31990896404) — the same scenario family as the Android bug in test(android): restore full-tier lifecycle and observability scenarios (#1781 A1) #1793, where a permission reset kills the app and the relaunch does not land on the expected route. That class is worth checking first. The iOS scenario lives intest/integration/ios-simulator-e2e/live-full-scenarios.ts, separate from the Android files test(android): restore full-tier lifecycle and observability scenarios (#1781 A1) #1793 touches.Exit condition
Move the two jobs back into
replays-nightly.ymland delete theirMANUAL_ONLY_OWNERSentries once a dispatch run is green. The audit test fails if the entries outlive the parking, so this cannot quietly become permanent.