Fix recurring CI exit-143 failures: pin coverage, cap Python in subprocess-heavy project envs - #659
Merged
Merged
Conversation
…ocess-heavy project envs Weekly scheduled CI has failed 5+ weeks running: mpi4py/petsc/pytorch jobs and several project test jobs die within ~1s of pytest collection starting, SIGTERM'd (exit 143/15) with no traceback. Two independent causes, confirmed by bisecting CI run logs: - `coverage` is unpinned across the test envs, so CI silently rides the newest conda-forge release every run. The last known-good scheduled run resolved coverage 7.15.0; the very next run (two days later) resolved 7.15.1 and immediately hit exit-143 failures, and every run since has failed the same way through 7.15.4 (latest as of writing). The failures are limited to jobs that spawn subprocesses (MPI ranks, multiprocessing), consistent with coverage's `concurrency=['multiprocessing']` subprocess- tracking hook (installed via sitecustomize.py) interacting badly with process/thread init in mpi4py/petsc4py/torch. Pinned to 7.15.0, the last version with an extended clean track record in this CI. - Several project environment.yml files (and Monodomain's) don't pin a Python version at all, so micromamba occasionally resolves a brand new Python (3.14) before the full compiled-dependency chain (mpi4py, mpich, petsc4py, fenics, numba) is ready for it on conda-forge, causing the same SIGTERM-during-collection symptom. Capped to the range pySDC already tests explicitly (3.10-3.13), scoped only to the projects that actually depend on those compiled/subprocess-heavy packages. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Coverage report
This PR does not seem to contain any modification to coverable code. |
…tracking The coverage==7.15.0 pin from the previous commit doesn't actually fix the exit-143 CI failures: this branch's own CI run hit the identical SIGTERM signature on mpi4py/petsc/pytorch/project jobs even under 7.15.0, at a rate (~15% of jobs) comparable to unpinned 7.15.4. The one prior "clean" run under 7.15.0 was a sample size of one, not evidence the version is safe. It also broke environment solving for StroemungsRaum, whose environment.yml (pulling in fenics) couldn't satisfy the pin's exact version constraint against packages `--freeze-installed` had already locked in. Since the race isn't tied to a specific coverage release, remove the thing that's actually crashing: coverage's `concurrency = ['multiprocessing']` subprocess-tracking hook (installed via sitecustomize.py + coverage.process_startup() in CI), which intermittently SIGTERMs subprocess-spawning test jobs during collection. Coverage is left unpinned again. Trade-off: coverage running inside MPI-rank/multiprocessing subprocesses is no longer measured. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…sion pins Removing concurrency=['multiprocessing'] and the sitecustomize.py subprocess hook did NOT fix the exit-143 failures: this branch's own follow-up CI run hit the identical instant-SIGTERM signature (mpi4py/petsc/pytorch/project jobs, ~1s into collection, no traceback) with subprocess coverage tracking fully disabled. That falsifies the coverage-causation theory this and the prior commit were built on - removing the suspected mechanism achieved nothing, so there's no reason to pay its real cost (losing coverage measurement inside MPI-rank/multiprocessing subprocesses). Restored. The Python-version pins are unaffected by this - they fix a separate, confirmed, unrelated bug (unpinned project envs occasionally resolving Python 3.14 before the compiled-dependency chain is ready for it) and stay in place. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two independent things going on, both aimed at reducing how hard this workflow hammers GitHub's own runner-provisioning backend, which has had documented capacity/concurrency problems this year (RCA'd incidents on 2026-07-09 and 2026-08-06, the latter an explicit "existing capacity and concurrency weakness" in GitHub's own Actions job-generation service) - this workflow's ~80-job matrix, launched twice per PR push, is exactly the kind of burst load that stresses that. - Add a workflow-level `concurrency` group keyed on branch name (using `head_ref || ref_name`, which resolves to the same string for a push event and a pull_request event on the same branch - `ref` would not, since it carries a `refs/heads/` prefix on push events). A push to a branch with an open PR fires both `push` and `pull_request` for the same commit; this cancels the stale duplicate instead of running both to completion. Preserves existing trigger behavior (push still runs CI on every branch) rather than narrowing when CI runs. - Cap `max-parallel: 8` on the two large matrix jobs (user_cpu_tests_linux: 20 combinations, project_cpu_tests_linux: 17), so GitHub isn't asked to provision ~37 runners for those two jobs alone in the same instant. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Direct evidence from this PR's own CI runs: re-running previously-failed jobs on identical code produces a *different* failure set each time (pytorch 3.10/3.12 passed on rerun while AllenCahn_Bayreuth and mpi4py 3.10 failed again) - the signature of non-deterministic external flakiness, not a reproducible bug. Combined with the documented GitHub Actions incidents this year (see prior commits), manual re-running already works as a workaround; this automates it. Retry is scoped tightly to avoid masking real problems: only exit 143 or 15 (the exact signature observed - instant SIGTERM within ~1s of pytest starting, no traceback) triggers a second attempt. Any other non-zero exit (e.g. exit 1 from genuine test failures) fails immediately on the first attempt, no retry, so this can't hide a real regression behind a "flaky test" label. Applied to the three "Run pytest for CPU stuff" steps that have shown this signature (user_cpu_tests_linux, project_cpu_tests_linux, monodomain). Left user_libpressio_tests/user_firedrake_tests alone - no evidence they need it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…egression theory Version-bisected the runner image across every observed pass/fail this session: last clean run (2026-07-15) used image 20260705.232.1 (kernel 6.17.0-1018-azure); first sustained-failure run (2026-07-20) used 20260714.240.1 (kernel bumped to 6.17.0-1020-azure), and every image since (through 20260810.271.1 today) has kept failing. Combined with the region clustering already found (failures concentrated in centralus/northcentralus/westcentralus/westus3, zero in eastus/eastus2/ westus/westus2), this points at a kernel-level regression in GitHub's ubuntu-latest (24.04) image, likely rolled out unevenly by region, that MPICH's process/shared-memory init doesn't handle cleanly. Testing that theory directly: switch the three jobs that have shown this signature (user_cpu_tests_linux, project_cpu_tests_linux, user_monodomain_tests_linux) to ubuntu-22.04, a separately-versioned image lineage that may not carry the same kernel change. Left lint, user_libpressio_tests, user_firedrake_tests (containerized, never showed this failure) and coverage_comment_on_pr (no MPI) on ubuntu-latest. This is explicitly an experiment - revert to ubuntu-latest if it doesn't measurably reduce the failure rate. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The ubuntu-22.04 experiment (previous commit) is inconclusive-to-negative: the exit-143 signature still occurred on that separately-versioned image lineage (DAE, fenics 3.13, Resilience, pytorch 3.13 failed), which rules out the GitHub runner-image kernel-bump theory from two commits ago. Reverted to ubuntu-latest for all jobs. What survived that test: the region correlation. The four new failures landed in westus3 (x3) and centralus (x1) - the same regions flagged before - while all ~30 other jobs in the same run, spread across eastus/eastus2/westus/westus2/northcentralus/westcentralus/mexicocentral, passed. Switching OS image didn't change which regions fail, which points below the OS/kernel layer entirely, into the underlying Azure host/ network infrastructure in those specific datacenters - something no runner image or package version choice can work around. Since GitHub-hosted runners don't expose region selection, and the existing per-step retry loops rerun the command on the *same* runner (so can't help when the runner/region itself is the problem), add a `rerun-failed-jobs` job that calls the GitHub API to rerun failed jobs in a fresh attempt (new runner, new shot at a healthy region) via `gh run rerun --failed`. Capped at 3 total attempts via `github.run_attempt` to avoid looping forever. Coarser than the per-step retry - it reruns on any job failure, not just exit 143/15, since the rerun API doesn't expose that distinction - but a genuinely failing test still ends up red after attempts are exhausted, just slower. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The rerun-failed-jobs job from the last commit failed immediately every time: "run 32034660678 cannot be rerun; This workflow is already running". GitHub refuses to rerun a run that still has an active job - and a job calling `gh run rerun` on its own run can never observe that run as anything but in-progress, since it's itself one of the jobs keeping it that way. Structurally can't work from inside the same run. Moved the logic to a new auto_rerun_failed_jobs.yml, triggered by workflow_run (types: [completed]) on "CI pipeline for pySDC" - which only fires once that run has genuinely finished, so the rerun call succeeds. This is the same pattern postprocess.yml already uses successfully in this repo for the exact same trigger. Removed the broken in-workflow job from ci_pipeline.yml. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per-step retry (loop the pytest command on SIGTERM) is redundant now that auto_rerun_failed_jobs.yml exists, and we already have direct evidence it doesn't help: two in-place retry attempts in the same job failed identically (same runner, same likely-bad region both times). The job-level rerun is the one with actual evidence of working, since it gets a fresh runner. Removed the loops from all three "Run pytest for CPU stuff" steps, back to a plain single invocation. Also investigated the coverage_comment_on_pr failure from the last clean run: a genuine, transient GitHub API 504 Gateway Timeout on GET /repos/.../pulls/659, unrelated to permissions (the action's own "permissions set incorrectly" message is generic boilerplate it prints for any API failure) and unrelated to the MPI/SIGTERM investigation. Marked that step continue-on-error: true, since a missing coverage comment is cosmetic and shouldn't fail the PR's CI status. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This was referenced Aug 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Weekly scheduled CI failed 5+ weeks running (see #657 for prior, unsuccessful attempts). Jobs that spawn subprocesses —
mpi4py,petsc,pytorch, and various project test jobs — die within ~1s of pytest starting, SIGTERM'd (exit 143/15), no traceback.Investigation trail
Two independent bugs were confirmed and fixed here; a third, larger cause was tracked down but turns out to be outside this repo's control.
1. Confirmed bug: unpinned Python lets
project_cpu_tests_linuxdrift onto untested versions. 17 projectenvironment.ymlfiles (and Monodomain's) don't pin a Python version, somicromambaoccasionally resolves a version this project doesn't test against (e.g. 3.14) before the compiled-dependency chain —mpi4py,mpich,petsc4py,fenics,numba— is ready for it on conda-forge, hitting the same SIGTERM-during-collection symptom. Fixed: pinnedpython>=3.10,<3.14(matching the rangeuser_cpu_tests_linuxalready tests explicitly), scoped to the 9 projects that actually depend on those compiled/subprocess-heavy packages.2. Ruled out, at length:
coverage's subprocess-tracking hook. Initially pinnedcoverage==7.15.0on the theory that a post-7.15.0 release regressed itsconcurrency=['multiprocessing']handling. Directly falsified: this branch's own CI runs hit the identical SIGTERM signature under7.15.0at a rate comparable to unpinned7.15.4, and removingconcurrency=['multiprocessing'](and itssitecustomize.pysubprocess hook) entirely frompyproject.toml/the workflow also didn't help — the crashes persisted unchanged. Both changes were reverted;coverageis unpinned again,concurrency=['multiprocessing']is back.3. Root cause, as far as it can be traced from here: a regional GitHub-hosted-runner infrastructure issue. Comparing every observed failing job against passing ones in the same run:
mpich, samempi4py, samepetsc4py) pass in one job and fail in another within the same run — rules out any specific package version.centralus/northcentralus/westcentralus/westus3, essentially zero ineastus/eastus2/westus/westus2across every run checked this investigation.ubuntu-latestimage looked promising (last clean run used kernel6.17.0-1018-azure, first sustained-failure run used6.17.0-1020-azure) but was directly tested and refuted: switching the affected jobs toubuntu-22.04(a separate image/kernel lineage) reproduced the identical failures, in the identical bad regions.What's in this PR
python>=3.10,<3.14pinned in the 9 affected project env files + Monodomain (real bug fix, virtual PFASST #1 above).concurrencygroup (${{ github.workflow }}-${{ github.head_ref || github.ref_name }},cancel-in-progress: true): a push to a branch with an open PR fires bothpushandpull_requestfor the same commit, doubling the ~80-job matrix. This cancels the stale duplicate. Confirmed working directly (watched apushrun get cancelled in favor of thepull_requestrun for the same commit).max-parallel: 8on the two large matrix jobs, reducing how many runners get requested from GitHub simultaneously.auto_rerun_failed_jobs.yml(new workflow): since the failures are tied to the runner/region itself rather than anything retryable in-place, a job-level rerun (fresh runner, new shot at a healthy region) is the only retry strategy with actual evidence behind it — confirmed in-place step retries don't help (two attempts in the same job failed identically). This has to be a separateworkflow_run-triggered workflow, not a job insideci_pipeline.yml: GitHub refuses to rerun a run that still has an active job, so a job callinggh run rerunon its own run always fails with "already running". Uses the sameworkflow_run: types: [completed]patternpostprocess.ymlalready relies on in this repo. Capped at 3 total attempts. Note:workflow_runtriggers are only registered from the workflow file on the default branch, so this can't be exercised until merged tomaster.coverage_comment_on_prmarkedcontinue-on-error: true: observed failing on a transient GitHub API 504 unrelated to everything above; it only posts an informational PR comment, not a correctness signal, so it shouldn't fail the PR's overall CI status.Test plan
pushrun cancelled)auto_rerun_failed_jobs.ymlneeds a post-merge run againstmasterto confirm theworkflow_runtrigger fires correctly🤖 Generated with Claude Code