Bump GitHub Actions to Node 24-compatible major versions - #660
Closed
pancetta wants to merge 10 commits into
Closed
Conversation
…ocess-heavy project envs Weekly scheduled CI has failed 5+ weeks running: mpi4py/petsc/pytorch jobs and several project test jobs die within ~1s of pytest collection starting, SIGTERM'd (exit 143/15) with no traceback. Two independent causes, confirmed by bisecting CI run logs: - `coverage` is unpinned across the test envs, so CI silently rides the newest conda-forge release every run. The last known-good scheduled run resolved coverage 7.15.0; the very next run (two days later) resolved 7.15.1 and immediately hit exit-143 failures, and every run since has failed the same way through 7.15.4 (latest as of writing). The failures are limited to jobs that spawn subprocesses (MPI ranks, multiprocessing), consistent with coverage's `concurrency=['multiprocessing']` subprocess- tracking hook (installed via sitecustomize.py) interacting badly with process/thread init in mpi4py/petsc4py/torch. Pinned to 7.15.0, the last version with an extended clean track record in this CI. - Several project environment.yml files (and Monodomain's) don't pin a Python version at all, so micromamba occasionally resolves a brand new Python (3.14) before the full compiled-dependency chain (mpi4py, mpich, petsc4py, fenics, numba) is ready for it on conda-forge, causing the same SIGTERM-during-collection symptom. Capped to the range pySDC already tests explicitly (3.10-3.13), scoped only to the projects that actually depend on those compiled/subprocess-heavy packages. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…tracking The coverage==7.15.0 pin from the previous commit doesn't actually fix the exit-143 CI failures: this branch's own CI run hit the identical SIGTERM signature on mpi4py/petsc/pytorch/project jobs even under 7.15.0, at a rate (~15% of jobs) comparable to unpinned 7.15.4. The one prior "clean" run under 7.15.0 was a sample size of one, not evidence the version is safe. It also broke environment solving for StroemungsRaum, whose environment.yml (pulling in fenics) couldn't satisfy the pin's exact version constraint against packages `--freeze-installed` had already locked in. Since the race isn't tied to a specific coverage release, remove the thing that's actually crashing: coverage's `concurrency = ['multiprocessing']` subprocess-tracking hook (installed via sitecustomize.py + coverage.process_startup() in CI), which intermittently SIGTERMs subprocess-spawning test jobs during collection. Coverage is left unpinned again. Trade-off: coverage running inside MPI-rank/multiprocessing subprocesses is no longer measured. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…sion pins Removing concurrency=['multiprocessing'] and the sitecustomize.py subprocess hook did NOT fix the exit-143 failures: this branch's own follow-up CI run hit the identical instant-SIGTERM signature (mpi4py/petsc/pytorch/project jobs, ~1s into collection, no traceback) with subprocess coverage tracking fully disabled. That falsifies the coverage-causation theory this and the prior commit were built on - removing the suspected mechanism achieved nothing, so there's no reason to pay its real cost (losing coverage measurement inside MPI-rank/multiprocessing subprocesses). Restored. The Python-version pins are unaffected by this - they fix a separate, confirmed, unrelated bug (unpinned project envs occasionally resolving Python 3.14 before the compiled-dependency chain is ready for it) and stay in place. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two independent things going on, both aimed at reducing how hard this workflow hammers GitHub's own runner-provisioning backend, which has had documented capacity/concurrency problems this year (RCA'd incidents on 2026-07-09 and 2026-08-06, the latter an explicit "existing capacity and concurrency weakness" in GitHub's own Actions job-generation service) - this workflow's ~80-job matrix, launched twice per PR push, is exactly the kind of burst load that stresses that. - Add a workflow-level `concurrency` group keyed on branch name (using `head_ref || ref_name`, which resolves to the same string for a push event and a pull_request event on the same branch - `ref` would not, since it carries a `refs/heads/` prefix on push events). A push to a branch with an open PR fires both `push` and `pull_request` for the same commit; this cancels the stale duplicate instead of running both to completion. Preserves existing trigger behavior (push still runs CI on every branch) rather than narrowing when CI runs. - Cap `max-parallel: 8` on the two large matrix jobs (user_cpu_tests_linux: 20 combinations, project_cpu_tests_linux: 17), so GitHub isn't asked to provision ~37 runners for those two jobs alone in the same instant. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Direct evidence from this PR's own CI runs: re-running previously-failed jobs on identical code produces a *different* failure set each time (pytorch 3.10/3.12 passed on rerun while AllenCahn_Bayreuth and mpi4py 3.10 failed again) - the signature of non-deterministic external flakiness, not a reproducible bug. Combined with the documented GitHub Actions incidents this year (see prior commits), manual re-running already works as a workaround; this automates it. Retry is scoped tightly to avoid masking real problems: only exit 143 or 15 (the exact signature observed - instant SIGTERM within ~1s of pytest starting, no traceback) triggers a second attempt. Any other non-zero exit (e.g. exit 1 from genuine test failures) fails immediately on the first attempt, no retry, so this can't hide a real regression behind a "flaky test" label. Applied to the three "Run pytest for CPU stuff" steps that have shown this signature (user_cpu_tests_linux, project_cpu_tests_linux, monodomain). Left user_libpressio_tests/user_firedrake_tests alone - no evidence they need it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…egression theory Version-bisected the runner image across every observed pass/fail this session: last clean run (2026-07-15) used image 20260705.232.1 (kernel 6.17.0-1018-azure); first sustained-failure run (2026-07-20) used 20260714.240.1 (kernel bumped to 6.17.0-1020-azure), and every image since (through 20260810.271.1 today) has kept failing. Combined with the region clustering already found (failures concentrated in centralus/northcentralus/westcentralus/westus3, zero in eastus/eastus2/ westus/westus2), this points at a kernel-level regression in GitHub's ubuntu-latest (24.04) image, likely rolled out unevenly by region, that MPICH's process/shared-memory init doesn't handle cleanly. Testing that theory directly: switch the three jobs that have shown this signature (user_cpu_tests_linux, project_cpu_tests_linux, user_monodomain_tests_linux) to ubuntu-22.04, a separately-versioned image lineage that may not carry the same kernel change. Left lint, user_libpressio_tests, user_firedrake_tests (containerized, never showed this failure) and coverage_comment_on_pr (no MPI) on ubuntu-latest. This is explicitly an experiment - revert to ubuntu-latest if it doesn't measurably reduce the failure rate. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The ubuntu-22.04 experiment (previous commit) is inconclusive-to-negative: the exit-143 signature still occurred on that separately-versioned image lineage (DAE, fenics 3.13, Resilience, pytorch 3.13 failed), which rules out the GitHub runner-image kernel-bump theory from two commits ago. Reverted to ubuntu-latest for all jobs. What survived that test: the region correlation. The four new failures landed in westus3 (x3) and centralus (x1) - the same regions flagged before - while all ~30 other jobs in the same run, spread across eastus/eastus2/westus/westus2/northcentralus/westcentralus/mexicocentral, passed. Switching OS image didn't change which regions fail, which points below the OS/kernel layer entirely, into the underlying Azure host/ network infrastructure in those specific datacenters - something no runner image or package version choice can work around. Since GitHub-hosted runners don't expose region selection, and the existing per-step retry loops rerun the command on the *same* runner (so can't help when the runner/region itself is the problem), add a `rerun-failed-jobs` job that calls the GitHub API to rerun failed jobs in a fresh attempt (new runner, new shot at a healthy region) via `gh run rerun --failed`. Capped at 3 total attempts via `github.run_attempt` to avoid looping forever. Coarser than the per-step retry - it reruns on any job failure, not just exit 143/15, since the rerun API doesn't expose that distinction - but a genuinely failing test still ends up red after attempts are exhausted, just slower. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The rerun-failed-jobs job from the last commit failed immediately every time: "run 32034660678 cannot be rerun; This workflow is already running". GitHub refuses to rerun a run that still has an active job - and a job calling `gh run rerun` on its own run can never observe that run as anything but in-progress, since it's itself one of the jobs keeping it that way. Structurally can't work from inside the same run. Moved the logic to a new auto_rerun_failed_jobs.yml, triggered by workflow_run (types: [completed]) on "CI pipeline for pySDC" - which only fires once that run has genuinely finished, so the rerun call succeeds. This is the same pattern postprocess.yml already uses successfully in this repo for the exact same trigger. Removed the broken in-workflow job from ci_pipeline.yml. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per-step retry (loop the pytest command on SIGTERM) is redundant now that auto_rerun_failed_jobs.yml exists, and we already have direct evidence it doesn't help: two in-place retry attempts in the same job failed identically (same runner, same likely-bad region both times). The job-level rerun is the one with actual evidence of working, since it gets a fresh runner. Removed the loops from all three "Run pytest for CPU stuff" steps, back to a plain single invocation. Also investigated the coverage_comment_on_pr failure from the last clean run: a genuine, transient GitHub API 504 Gateway Timeout on GET /repos/.../pulls/659, unrelated to permissions (the action's own "permissions set incorrectly" message is generic boilerplate it prints for any API failure) and unrelated to the MPI/SIGTERM investigation. Marked that step continue-on-error: true, since a missing coverage comment is cosmetic and shouldn't fail the PR's CI status. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every CI run has been printing "Node.js 20 is deprecated" warnings for actions still declaring runs.using: node20 in their action.yml, forced onto Node 24 by the runner. Verified each action's actual runs.using value at both the currently-pinned and latest major version before bumping, rather than assuming: - actions/checkout: v1 (lint job only, a legacy pre-Node "plugin" action, not itself node20 but badly outdated) and v4 (node20) -> v7 (node24) - actions/upload-artifact: v4 (node20) -> v7 (node24) - actions/download-artifact: v4 (node20) -> v8 (node24) - mamba-org/setup-micromamba: v1 (node20) -> v3 (node24); confirmed the two inputs actually used here (environment-file, create-args) are unchanged in v3 Left codecov/codecov-action@v5, py-cov-action/python-coverage-comment- action@v3, and JamesIves/github-pages-deploy-action@v4 alone: confirmed these are composite/docker/already-node24 respectively, not implicated in the warning at all. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Coverage report
This PR does not seem to contain any modification to coverable code. |
1 task
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every CI run prints "Node.js 20 is deprecated" warnings for actions still declaring
runs.using: node20in theiraction.yml, which GitHub now force-runs on Node 24 anyway. Same issue previously seen and fixed in the RSE_course_JuRSE project.Verified each action's actual
runs.usingvalue at both the currently-pinned and candidate new major version before bumping (not just assumed from version numbers):actions/checkout:v1(lint job only — a legacy pre-Node "plugin" action, not itself the source of the node20 warning, but badly outdated) andv4(node20) →v7(node24)actions/upload-artifact:v4(node20) →v7(node24)actions/download-artifact:v4(node20) →v8(node24)mamba-org/setup-micromamba:v1(node20) →v3(node24) — confirmed the two inputs actually used here (environment-file,create-args) are unchanged inv3Left alone (confirmed not implicated in the warning at all):
codecov/codecov-action@v5— composite action, no Node runtimepy-cov-action/python-coverage-comment-action@v3— Docker action, no Node runtimeJamesIves/github-pages-deploy-action@v4— alreadynode24Test plan
🤖 Generated with Claude Code