chore(vscode-e2e): group scenarios into named shards#9423
Open
ccastrotrejo wants to merge 4 commits into
Open
Conversation
Consolidate the vscode-e2e matrix from 18 per-scenario shards into 7 human-readable group shards (activation-and-create, workspace-creation, designer-lifecycle, runtime-actions, designer-interactions, suite-and-bundle, project-conversion). LA_E2E_SCENARIO now accepts a comma-separated list; each scenario still runs in its own fresh VS Code session via prepareFreshSession(), so grouping preserves per-scenario isolation while amortizing the fixed CI setup cost (pnpm install, xvfb, artifact hydration) across ~7 runners instead of 18. Failures stay attributable to a shard and fail loudly (aggregate = max exit). Updates the Squad knowledge entry to the grouped shape and the branch-protection check-name caveat. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1925dce8-22c3-4c33-b8c2-14d99a4d75f7
Contributor
🤖 AI PR Validation ReportPR Review ResultsThank you for your submission! Here's detailed feedback on your PR title and body compliance:✅ PR Title
✅ Commit Type
✅ Risk Level
✅ What & Why
✅ Impact of Change
✅ Test Plan
✅ Contributors
✅ Screenshots/Videos
Summary Table
✅ All checks pass. This PR is compliant with the team template and cleared to merge.Powered by: Copilot CLI (claude-opus-4.8) | Last updated: Fri, 17 Jul 2026 22:34:52 GMT |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the VS Code ExTester E2E CI pipeline to reduce runner/setup overhead by grouping individual scenario shards into a smaller set of named, human-readable shards, while keeping scenario-level isolation via fresh VS Code sessions per scenario.
Changes:
- Collapse the
vscode-e2eworkflow matrix from per-scenario jobs into 7 named shard jobs that pass a comma-separatedLA_E2E_SCENARIOlist. - Update the E2E launcher (
run-e2e.ts) soLA_E2E_SCENARIOcan select multiple scenario IDs (validated + deduped) and run them sequentially. - Refresh the squad knowledge doc to reflect the grouped-shard CI shape and branch-protection/check-name implications.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
.github/workflows/vscode-e2e.yml |
Replaces per-scenario matrix entries with 7 named shard entries; updates job/artifact naming to key off matrix.shard. |
apps/vs-code-designer/src/test/ui/run-e2e.ts |
Extends LA_E2E_SCENARIO parsing to support comma-separated lists and runs the selected scenarios sequentially. |
.squad/knowledge/vscode-e2e-testing.md |
Updates durable CI learnings to the new grouped-shard model and documents the check-name/branch-protection caveat. |
ExTester UI scenarios had no retry: a single transient xvfb/focus/cold-start race failed the whole shard. Add scenario-level retry in runScenarioPhases(), gated by LA_E2E_SCENARIO_RETRIES (default 0 locally; CI sets 1). Each retry is a full prepareFreshSession() (kills VS Code/chromedriver/func), so it also clears stale-window/leftover-process flakes. Non-masking: a scenario that fails every attempt still fails the shard (exit = max); a retry-to-pass is surfaced via a ::warning:: flake annotation so the race can still be root-caused. The whole attempt (session prep, preflight, run, p41a bundle verify) is retryable; deterministic failures re-throw and still fail. Job timeouts widened (vscode-e2e 35m, setup-fixtures 30m, compat 25m) so a retry isn't killed mid-run. Documented in the Squad knowledge base. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1925dce8-22c3-4c33-b8c2-14d99a4d75f7
ccastrotrejo
changed the title
ccastrotrejo
added
risk:low
… runs During a grouped shard, LA_E2E_SCENARIO held the full comma-separated list while each scenario ran, breaking test/helper code that reads it as the active scenario id via strict equality (e.g. multipleDesigners.test.ts). Re-point LA_E2E_SCENARIO at the current scenario id for the duration of each scenario (reusing the existing env-override capture/restore), so per-scenario semantics are preserved. Selection already parses the list before runScenarioPhases, so this does not affect which scenarios run. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1925dce8-22c3-4c33-b8c2-14d99a4d75f7
…t-group-vscode-e2e-shards
ccastrotrejo
enabled auto-merge (squash)
July 17, 2026 23:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Commit Type
Risk Level
What & Why
Consolidates the
vscode-e2ematrix from 18 per-scenario shards → 7 human-readable group shards, cutting redundant CI setup while preserving per-scenario isolation and fail-loud gating.Grouping is safe because isolation lives at the VS Code session level, not the shard level:
runScenarioPhases()runs each scenario throughprepareFreshSession()(kills lingering VS Code/chromedriver/dotnet) + a freshExTesterinstance. Grouped scenarios run sequentially in isolated sessions, not in one shared flaky VS Code window.Changes
.github/workflows/vscode-e2e.yml— matrix collapsed to 7 named shards; each passes a comma-separatedLA_E2E_SCENARIOlist. Job name and screenshot artifacts key onmatrix.shard.apps/vs-code-designer/src/test/ui/run-e2e.ts—LA_E2E_SCENARIOnow accepts a comma-separated list (validated and deduped); single-id usage is unchanged..squad/knowledge/vscode-e2e-testing.md— learning updated from strict per-scenario shards to grouped shards, including the branch-protection check-name caveat.activation-and-createworkspace-creationdesigner-lifecycleruntime-actionsdesigner-interactionssuite-and-bundleproject-conversionFlakiness: scenario-level retry (non-masking)
ExTester UI scenarios had no scenario-level retry, so one transient xvfb/focus/cold-start race failed the whole shard. This PR adds opt-in retry in
runScenarioPhases()viaLA_E2E_SCENARIO_RETRIES(default0; CI sets1). Each retry performs a fullprepareFreshSession()and deterministic failures still fail the shard after all attempts. Retry-to-pass is emitted as a warning annotation, not hidden withcontinue-on-error.Impact of Change
LA_E2E_SCENARIOaccepts comma-separated scenario lists, andLA_E2E_SCENARIO_RETRIESenables opt-in per-scenario retry.vscode-e2e-summaryrollup rather than individual renamed shard checks.Test Plan
tsup/type-compile and grouped shard execution run in CI because this worktree has nonode_modules.Contributors
@ccastrotrejo
Screenshots/Videos
N/A — no visual/UI changes.