Re-running individual failed test jobs fails because shared infrastructure has been torn down

[MariusStorhaug]

Maintainers and contributors run the Process-PSModule.yml workflow on every pull request. The workflow consists of a BeforeAll-ModuleLocal job that provisions shared GitHub infrastructure (organization and user repositories named Test-{OS}-{TokenType}-{RunID}), a fan-out matrix of Test-ModuleLocal jobs (one per test file × OS × auth case), and an AfterAll-ModuleLocal job that tears everything down. The setup and teardown were introduced in #541 so that the matrix could share repositories instead of each leaf job creating its own.

Request

When a small number of leaf test jobs in the Test-ModuleLocal matrix fail because of intermittent issues (rate limits, transient API errors, GitHub-side flakiness), the only available recovery path on the GitHub Actions UI is Re-run failed jobs or Re-run all jobs. Neither path produces a working rerun for this workflow:

What happens

1. The matrix completes with a mix of successful and failed leaf jobs (the screenshot on the originating discussion shows Test-Releases, Test-Repositories, and Test-Variables red while ~25 sibling jobs are green).
2. AfterAll-ModuleLocal is configured with if: ... && always() and runs as soon as Test-ModuleLocal finishes, regardless of the matrix outcome. It deletes every Test-{OS}-{TokenType}-{RunID}* repository for the run.
3. The user clicks Re-run failed jobs.
4. GitHub Actions reruns only the failed leaf jobs in Test-ModuleLocal. It does not rerun BeforeAll-ModuleLocal because that job succeeded the first time, and skipped/successful needs are not re-executed by Re-run failed jobs.
5. The rerun leaf jobs immediately fail in their per-context BeforeAll, where they call Get-GitHubRepository -Name "Test-{OS}-{TokenType}-{RunID}" against a repository that no longer exists.

What is expected

A user who wants to retry transiently failing leaf jobs can do so without having to rerun every successful sibling job. The shared infrastructure that the leaf jobs depend on must be present (or re-created) before the rerun executes, so the targeted retry succeeds when the underlying flake has cleared.

Reproduction steps

1. Open a pull request against main so that Process-PSModule.yml runs end-to-end.
2. Wait for the workflow to finish with at least one failed leaf job in the Test-ModuleLocal matrix and the AfterAll-ModuleLocal job completed (green).
3. From the workflow run page, click Re-run failed jobs.
4. Observe that every reran leaf job fails in its BeforeAll block while resolving the shared repository — the repository was deleted by AfterAll-ModuleLocal after step 2.

Regression

This behavior was introduced by #541. Before that change, every leaf test job created and deleted its own repositories using [guid]::NewGuid(), so reruns were self-contained — each leaf job rebuilt the infrastructure it needed. Consolidating to shared run-scoped infrastructure removed that property.

Workaround

The only working recovery today is Re-run all jobs, which re-executes every successful leaf job. For a workflow where a single matrix leg can take over two hours (Test-Apps (macOS) ran 1h 40m in the linked screenshot), this wastes substantial CI time and consumes additional GitHub API quota that the shared-repo design in #541 was specifically introduced to conserve.

Acceptance criteria

• A user can trigger a retry of a specific failed leaf job (or a subset of failed leaf jobs) and have it succeed when the underlying transient cause has cleared, without rerunning every successful sibling.
• The shared infrastructure the leaf jobs depend on is guaranteed to exist when those jobs execute, regardless of whether they are part of the original run or a partial rerun.
• Successful leaf jobs from the original run are not re-executed unnecessarily on a partial retry.
• Concurrent workflow runs continue to be isolated from each other (the run-scoped naming guarantee from #541 is preserved).
• API call budget for a partial retry is bounded — a retry of one leaf job does not cost more than the leaf job itself plus whatever minimum setup is unavoidable.

[MariusStorhaug]

Additional analysis and suggestions for discussion

Posting this separately from the issue body so the issue stays focused on the problem statement. Suggestions below are starting points for discussion, not a recommended solution.

Constraints derived from reading the harness

• GITHUB_RUN_ID is stable across reruns; GITHUB_RUN_ATTEMPT increments per attempt.
• tests/BeforeAll.ps1 is already idempotent for a given GITHUB_RUN_ID — it pre-cleans Test-{OS}-{TokenType}-{RunID}* before creating, so invoking it multiple times for the same run is safe.
• Matrix leaf jobs only read the shared repo (Get-GitHubRepository), with one exception — Repositories.Tests.ps1 still owns its own create/delete lifecycle.
• Workflow orchestration lives in PSModule/Process-PSModule, not in this repo. Some options require changes there; some can be done entirely within PSModule/GitHub.
• GitHub Actions semantics: Re-run failed jobs does not re-execute successful needs. Re-run all jobs does, but defeats the purpose of partial retries. There is no built-in "rerun this leaf and its setup chain" option.

Options to discuss

#	Option	Where it lives	Trade-off
1	Defer AfterAll-ModuleLocal — run only on final attempt; rely on a scheduled sweeper for stale Test-*-{old-RunID} repos	Process-PSModule + sweeper	Cleanest. Shared repos survive between attempts → partial reruns Just Work. Requires upstream change and a sweeper.
2	Make leaf jobs self-heal — move "ensure shared repo exists" into each test file's per-context BeforeAll (idempotent get-or-create)	This repo only	No upstream dependency. Costs extra API calls on first run (every leaf checks). Partial reruns work without orchestration changes.
3	Force BeforeAll-ModuleLocal to re-execute on every attempt	Process-PSModule	GitHub Actions still won't re-run a successful job under Re-run failed jobs, so this only helps if combined with a different rerun mechanism.
4	Custom workflow_dispatch retry workflow taking a run ID + test file names, which provisions infra under the same GITHUB_RUN_ID and reruns the named files via pwsh tests/...	This repo	Bypasses GitHub Actions rerun semantics entirely. Most flexible, most new surface area.
5	Scope infra by GITHUB_RUN_ATTEMPT and rebuild every attempt	This repo + Process-PSModule	Same orchestration problem as (3) — only helps if BeforeAll actually re-runs on partial retries.
6	Hybrid (1) + (2) — shared infra is the fast path; leaf jobs verify and recreate if missing	This repo (+ optional upstream)	Most resilient. Modest extra complexity. Defends against any cause of missing infra, not just rerun.

Initial lean

Options (2) and (6) are the only ones implementable entirely inside PSModule/GitHub. Option (1) is the cleanest architecturally if Process-PSModule is open to it. The shared-repo design from #541 was specifically to conserve API quota, so any option that adds per-leaf API calls (2 standalone) needs to quantify the additional cost against that budget.

Open questions for the discussion:

• Is changing PSModule/Process-PSModule in scope, or should the fix live entirely in this repo?
• How important is preserving the API budget gain from #541 versus retry ergonomics?
• Should Repositories.Tests.ps1 (which still owns its own lifecycle) follow whatever pattern is chosen, for consistency?

[MariusStorhaug]

Two prerequisite issues have been filed that must land before the persistent shared-repo model can be implemented safely:

• #591 — Replace hardcoded external dependency on ryanoasis/nerd-fonts in Releases.Tests.ps1. Currently one test calls Get-GitHubRelease -Owner ryanoasis -Repository nerd-fonts to exercise the pipeline path. This external resource can disappear or change at any time, is unrelated to anything this module owns, and consumes additional API quota. It should use assets already uploaded to the shared test repository within the same test context.

• #592 — Scope release tag names with $id and add AfterAll cleanup of releases and git tags in Releases.Tests.ps1. Tags are currently hardcoded (v1.0–v1.4) and never cleaned up. On a persistent shared repository, sequential runs collide on the first New-GitHubRelease -Tag 'v1.0' call; concurrent runs race on the same tag. Both issues are latent today and guaranteed failures once the persistent-repo model is in place.

Both issues are purely in tests/Releases.Tests.ps1 and have no upstream or module-code dependencies.

[MariusStorhaug]

A related issue has been filed: #593 — Seed test fixture files (including a workflow_dispatch-enabled workflow YAML) into the shared test repository to enable testing of the full Workflow function surface and to re-home artifact tests away from their current dependency on the live PSModule/GitHub repository and GITHUB_RUN_ID.

This is independent of the persistent shared-repo work in this issue but benefits from it: once the shared repository is persistent and consistently seeded, workflow and artifact tests can be run reliably both in CI and locally without the current bootstrapping dependencies.