fix(ci): serialize staging-coder.ddev.com integration tests without cross-cancelling other runs - #188
Merged
Merged
Conversation
…l triggers integration-test.yml, drupal-integration-test.yml, and drupal-contrib-integration-test.yml each provision real Coder workspaces (nested Docker/Sysbox builds) on the single shared staging-coder.ddev.com box, but their concurrency groups were scoped per-ref (github.ref / github.head_ref). A push to main and any open PR -- or two different PRs -- land in different groups, so GitHub Actions runs them at the same time, and the box's finite CPU/RAM/Docker capacity gets starved. Observed symptom: flaky mid-build "SSH connection ended unexpectedly" during ddev start, followed by cleanup failing with "a workspace build is already active" -- resource contention, not a real test failure. Switch all three to one shared literal concurrency group so any run that provisions real staging workspaces queues behind another, regardless of which branch, PR, or push triggered it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…x via job-level concurrency 696eeac fixed cross-ref resource contention on staging-coder.ddev.com by collapsing all three integration-test workflows into one literal concurrency group with cancel-in-progress: false -- but that also disabled the pre-existing behavior where a new commit on an open PR cancels that PR's own stale run, and it let a PR push queue behind, but never cancel, an in-progress run on main or another PR. It also missed that drupal-plain-gh/drupal-issue-fork-gh and contrib-plain-gh/contrib-issue-fork-gh run in parallel within a single workflow run with no dependency between them, so they could still starve the box even after that fix. Split concurrency into two tiers instead: - Workflow-level: ref-scoped, cancel-in-progress: true -- a new commit on a PR (or push to main) cancels only that ref's own stale run, never another ref's. - Job-level, on every job that provisions a real workspace on staging-coder.ddev.com: literal shared group across all three workflows, cancel-in-progress: false -- serializes real box usage globally without ever preempting someone else's in-flight build. Job-level groups match within a run as well as across runs, so this also serializes the previously-unaddressed parallel GH-hosted job pairs against each other. Also add timeout-minutes: 30 to those jobs -- with cancel-in-progress false on a shared group, a hung run would otherwise block every future trigger indefinitely. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
The job-level concurrency group added in 54ff4f4 was the wrong tool: a single PR push fires five box-provisioning jobs across the three integration-test workflows at nearly the same instant, and GitHub Actions concurrency groups only keep one job running plus one pending -- every other simultaneous contender gets silently CANCELLED, not queued. On PR #188 this dropped 3 of 5 templates outright (0 steps run, no red flag beyond a generic "cancelled" status) while only the first and the last-arriving job actually executed. Drop the job-level concurrency: group entirely. Each box-touching job now calls the new scripts/ci-wait-for-staging-box.sh right before creating its workspace: it polls the Coder server's actual ci-bot workspace count and waits its turn, so every job eventually runs instead of some being dropped. Bump timeout-minutes to 75 on those jobs to cover worst-case queued time (up to ~4 other jobs ahead, each several minutes) plus the job's own runtime. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
🤖 Generated with Claude Code