From 48a15e642a3367dfc543df056b691165e2d48406 Mon Sep 17 00:00:00 2001 From: DoubleGate Date: Wed, 19 Aug 2026 16:09:34 -0400 Subject: [PATCH 1/3] fix(ci): the apt bounds were calibrated against a claim, and the step over-installed The aarch64 provision failed on three consecutive PRs (#412, #415, #416) in one hour. The log says exactly where, and it is not a mirror outage: attempt 1 update killed at 180s, precisely the timeout attempt 2 update killed at 180s again attempt 3 update succeeded; install killed at 300s MID-DOWNLOAD, the log ending inside `Get:20 gcc-13-aarch64-linux-gnu [21.1 MB]` No apt error appears anywhere in it. The provision was succeeding slowly, and the wrapper converted that into a hard failure three times over -- the shape this release keeps finding, with the sign reversed: a gate that fires on a healthy run rather than one that passes a broken one. It blocked three PRs while reading as infrastructure decay. The bound came from a premise this script asserted and never measured: "a healthy `update` on these runners is a few seconds", which made 180s look like an order of magnitude of headroom. Rewritten to state the arithmetic instead, since that phrasing is precisely what went unchecked. Two independent changes. `apt-get update` now runs ONLY after a direct install has failed. The runner image ships a populated index, so a refresh does not belong on the happy path -- it is the recovery step for the one failure it fixes, an index stale enough that the requested version has moved. Attempt 1 skips it, removing the 180s that killed two of the three attempts before they reached the package at all. Worst case is now 300 + 15 + 600 + 30 + 600 = 1245s, inside the job's 25-minute budget with room for the check. The step also asked for the wrong package, and the comment above it said so without drawing the conclusion: it named `libc6-dev-arm64-cross` as what actually lands the headers, and stated the cross linker is unused because this gate is `cargo check` only. So it installed a whole cross toolchain to obtain a dependency it had already identified -- 20+ packages, one of them 21.1 MB, for a set of headers. bindgen runs the HOST clang against `--sysroot` and never invokes the cross compiler. Both ARM targets now install the header package directly, plus `--no-install-recommends`, which is the same argument as the timeout: every byte downloaded is time spent inside a bound. The sufficiency of the narrower package is VERIFIED BY THIS JOB, not asserted -- if the headers are not where the export step points clang, bindgen fails loudly in `cargo check` on the very run that installs it. The loop's branch structure is proven with a stubbed dry run: attempt 1 issues `install` alone, attempts 2 and 3 issue `update` then `install`. shellcheck and actionlint both clean. --- .github/scripts/apt-install-retry.sh | 65 ++++++++++++++++++++++++---- .github/workflows/ci.yml | 34 ++++++++++----- 2 files changed, 79 insertions(+), 20 deletions(-) diff --git a/.github/scripts/apt-install-retry.sh b/.github/scripts/apt-install-retry.sh index 4ccc1504..13052365 100755 --- a/.github/scripts/apt-install-retry.sh +++ b/.github/scripts/apt-install-retry.sh @@ -32,12 +32,41 @@ if [ -z "${APT_PACKAGE:-}" ]; then exit 1 fi -# Bounds chosen from observed behaviour, not from taste: a healthy `update` on -# these runners is a few seconds and a healthy `install` well under a minute, so -# these are roughly an order of magnitude of headroom. Long enough that a merely -# slow mirror still succeeds; short enough that three full attempts fit inside -# the 25-minute job budget with room for the build that follows. -readonly UPDATE_TIMEOUT=180 +# Bounds RE-CALIBRATED against a real failure, because the first set was +# calibrated against a claim. +# +# The comment that stood here asserted that "a healthy `update` on these runners +# is a few seconds", which made 180s look like an order of magnitude of headroom. +# It was never measured. On 2026-08-19 this step failed on three consecutive PRs +# (#412, #415, #416) and the log says exactly where: +# +# attempt 1 update killed at 180s, precisely the timeout +# attempt 2 update killed at 180s again +# attempt 3 update succeeded; install killed at 300s MID-DOWNLOAD, the log +# ending inside `Get:20 gcc-13-aarch64-linux-gnu [21.1 MB]` +# +# No apt error anywhere in it. The provision was succeeding slowly and the +# wrapper converted that into a hard failure, three times over. A bound derived +# from an unmeasured premise is a gate that fires on healthy runs, which is worse +# than a loose one: it blocked three PRs while looking like infrastructure decay. +# +# Two changes, and the second is what makes the first affordable. +# +# `apt-get update` now runs ONLY after a direct install has failed. The runner +# image ships a populated package index, so an index refresh does not belong on +# the happy path at all -- it is the recovery step for the one case that needs it, +# an index stale enough that the requested version has moved and install 404s. +# Attempt 1 skips it, which removes the 180s that killed two of the three +# attempts above before they ever reached the package. +# +# Worst case, stated as arithmetic rather than as "roughly an order of +# magnitude" -- that phrasing is what went unchecked last time: +# +# 300 + 15 + (300 + 300) + 30 + (300 + 300) = 1245s +# +# Under 21 minutes, inside the job's 25 with room for the `cargo check` that +# follows it. +readonly UPDATE_TIMEOUT=300 readonly INSTALL_TIMEOUT=300 readonly ATTEMPTS=3 @@ -60,9 +89,29 @@ readonly ATTEMPTS=3 # run at all, which would break the wrapper rather than degrade it. `env` is a # plain command and needs no such privilege. (Review on #409; both reviewers # raised it independently.) +apt_install() { + sudo env DEBIAN_FRONTEND=noninteractive timeout "$INSTALL_TIMEOUT" \ + apt-get install -yq --no-install-recommends "$APT_PACKAGE" +} + +apt_update() { + sudo env DEBIAN_FRONTEND=noninteractive timeout "$UPDATE_TIMEOUT" apt-get update -qq +} + +# `--no-install-recommends` is not a size micro-optimisation, it is the same +# argument as the bounds above: every byte downloaded is time spent inside a +# timeout. The recommends pulled here are packages the caller has already written +# down as unused -- the workflow step's own comment says the cross linker is +# unused because this gate is `cargo check` only. for attempt in $(seq 1 "$ATTEMPTS"); do - if sudo env DEBIAN_FRONTEND=noninteractive timeout "$UPDATE_TIMEOUT" apt-get update -qq && - sudo env DEBIAN_FRONTEND=noninteractive timeout "$INSTALL_TIMEOUT" apt-get install -yq "$APT_PACKAGE"; then + # Attempt 1 goes straight at the package. Later attempts refresh the index + # first, since a stale index is the one failure a refresh actually fixes. + if [ "$attempt" -eq 1 ]; then + if apt_install; then + echo "Installed ${APT_PACKAGE} on attempt ${attempt} (no index refresh needed)." + exit 0 + fi + elif apt_update && apt_install; then echo "Installed ${APT_PACKAGE} on attempt ${attempt}." exit 0 fi diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e269415f..6b9751d7 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -560,30 +560,40 @@ jobs: run: | echo "BINDGEN_EXTRA_CLANG_ARGS_aarch64_linux_android=--sysroot=$ANDROID_NDK_LATEST_HOME/toolchains/llvm/prebuilt/linux-x86_64/sysroot" >> "$GITHUB_ENV" # Provision the aarch64 glibc headers for bindgen (see the matrix note). - # `gcc-aarch64-linux-gnu` pulls `libc6-dev-arm64-cross`, which lands the - # aarch64 glibc headers under `/usr/aarch64-linux-gnu/include`; `--sysroot` - # sends clang there instead of the host's x86_64-only `/usr/include`, and - # the explicit `-isystem` guarantees the include dir is on the search path - # regardless of how the sysroot lays out `usr/include`. The cross linker in - # that package is unused — this gate is `cargo check` only. + # `libc6-dev-arm64-cross` lands the aarch64 glibc headers under + # `/usr/aarch64-linux-gnu/include`; `--sysroot` sends clang there instead of + # the host's x86_64-only `/usr/include`, and the explicit `-isystem` + # guarantees the include dir is on the search path regardless of how the + # sysroot lays out `usr/include`. + # + # This asked for `gcc-aarch64-linux-gnu` until v2.3.9, and the comment that + # stood here said why that was wrong without drawing the conclusion: it + # named `libc6-dev-arm64-cross` as the package that actually lands the + # headers, and stated that the cross linker is unused because this gate is + # `cargo check` only. So the step installed a whole cross toolchain to + # obtain a dependency it had already identified. The failure log makes the + # cost concrete — 20+ packages, of which `Get:20` alone is + # `gcc-13-aarch64-linux-gnu` at 21.1 MB, and it was mid-download when the + # install timeout fired. bindgen runs the HOST clang against the sysroot and + # never invokes the cross compiler. - name: Provision the aarch64 glibc headers for bindgen if: matrix.target == 'aarch64-unknown-linux-gnu' env: - APT_PACKAGE: gcc-aarch64-linux-gnu + APT_PACKAGE: libc6-dev-arm64-cross run: .github/scripts/apt-install-retry.sh - name: Export the aarch64 bindgen sysroot if: matrix.target == 'aarch64-unknown-linux-gnu' run: | echo "BINDGEN_EXTRA_CLANG_ARGS_aarch64_unknown_linux_gnu=--sysroot=/usr/aarch64-linux-gnu -isystem /usr/aarch64-linux-gnu/include" >> "$GITHUB_ENV" # Provision the armhf glibc headers for bindgen, exactly as for aarch64 - # above. `gcc-arm-linux-gnueabihf` pulls `libc6-dev-armhf-cross`, landing - # the 32-bit ARM glibc headers under `/usr/arm-linux-gnueabihf/include`. - # The cross linker in that package is unused here — this gate is - # `cargo check` only; the buildbot remains the authority on linking. + # above and narrowed for the same reason. `libc6-dev-armhf-cross` lands the + # 32-bit ARM glibc headers under `/usr/arm-linux-gnueabihf/include`; the + # cross compiler and linker are unused here, since this gate is + # `cargo check` only and the buildbot remains the authority on linking. - name: Provision the armhf glibc headers for bindgen if: matrix.target == 'armv7-unknown-linux-gnueabihf' env: - APT_PACKAGE: gcc-arm-linux-gnueabihf + APT_PACKAGE: libc6-dev-armhf-cross run: .github/scripts/apt-install-retry.sh - name: Export the armhf bindgen sysroot if: matrix.target == 'armv7-unknown-linux-gnueabihf' From 3775e78c0c4e97dc147080094a7de9059c261754 Mon Sep 17 00:00:00 2001 From: DoubleGate Date: Wed, 19 Aug 2026 16:23:50 -0400 Subject: [PATCH 2/3] fix(ci): re-calibrate again, this time against a Fetched line This change's own first CI run passed, and the log says why the previous bounds could not have: Fetched 4201 kB in 4min 45s (14.7 kB/s) Fourteen point seven kilobytes per second. The mirror is degraded by roughly three orders of magnitude, which is why every bound derived from "healthy" behaviour was wrong -- and why the old 40 MB package set was hopeless rather than unlucky: at that rate it needed about 45 minutes, past the whole job budget. Even at 4.2 MB it only just passed, and not the way the first draft of this comment assumed. The real sequence, read from the log rather than inferred from the exit code: attempt 1 install downloaded all 4201 kB (285s), then was killed at 300s during dpkg unpack -- the download finished, the install did not attempt 2 update, then install: NO re-download, because the archives were already in /var/cache/apt/archives. Succeeded. So the run was rescued by apt's archive cache persisting across attempts. That is a genuinely useful property -- each attempt makes progress instead of starting over -- but it was undesigned and undocumented, which puts it in the same class as the bound it rescued: behaviour nobody wrote down, working by accident, and indistinguishable from a design until it stops. It is now written down and no longer load-bearing. INSTALL_TIMEOUT is sized so ONE attempt completes at the worst speed actually observed: 285s of download plus dpkg, so 600s is about 2x that. ATTEMPTS drops to 2 to keep the worst case inside the job's 25 minutes -- 600 + 15 + (180 + 600) = 1395s plus ~45s of surrounding steps. The third attempt is no loss: attempt 2 already retries with a refreshed index AND a warm download cache, covering both the stale index and the slow mirror, so a third would only repeat it. --- .github/scripts/apt-install-retry.sh | 56 +++++++++++++++++++++------- 1 file changed, 42 insertions(+), 14 deletions(-) diff --git a/.github/scripts/apt-install-retry.sh b/.github/scripts/apt-install-retry.sh index 13052365..76fc5e10 100755 --- a/.github/scripts/apt-install-retry.sh +++ b/.github/scripts/apt-install-retry.sh @@ -50,25 +50,53 @@ fi # from an unmeasured premise is a gate that fires on healthy runs, which is worse # than a loose one: it blocked three PRs while looking like infrastructure decay. # -# Two changes, and the second is what makes the first affordable. -# # `apt-get update` now runs ONLY after a direct install has failed. The runner # image ships a populated package index, so an index refresh does not belong on -# the happy path at all -- it is the recovery step for the one case that needs it, -# an index stale enough that the requested version has moved and install 404s. -# Attempt 1 skips it, which removes the 180s that killed two of the three -# attempts above before they ever reached the package. +# the happy path -- it is the recovery step for the one case that needs it, an +# index stale enough that the requested version has moved and install 404s. +# +# THE SECOND CALIBRATION, from this change's own first CI run. Do not tune these +# numbers again without a `Fetched` line to point at. +# +# The narrowed package (see the workflow step) made the provision succeed, and +# the log gives the reason the old one could not: +# +# Fetched 4201 kB in 4min 45s (14.7 kB/s) +# +# Fourteen point seven kilobytes per second. The mirror is degraded by roughly +# three orders of magnitude, which is why every bound derived from "healthy" +# behaviour was wrong -- and why the previous 40 MB package set was hopeless: +# at that rate it needed about 45 minutes, past the job budget entirely. +# +# Even at 4.2 MB it only just passed, and NOT the way the first draft of this +# comment assumed. The real sequence was: +# +# attempt 1 install downloaded all 4201 kB (285s), then was killed at 300s +# during dpkg unpack -- the download finished, the install did not +# attempt 2 update, then install: NO re-download, because the archives were +# already in /var/cache/apt/archives. Succeeded. +# +# So the run was rescued by apt's archive cache persisting across attempts. That +# is a real and useful property -- each attempt makes progress rather than +# starting over -- but it was undesigned and undocumented, which makes it the +# same defect class as the bound it rescued: behaviour nobody wrote down. +# Written down now, and no longer relied upon: INSTALL_TIMEOUT is sized so ONE +# attempt completes at the worst speed actually observed. +# +# 285s download at 14.7 kB/s + dpkg unpack, so 600s is ~2x the worst observed. # -# Worst case, stated as arithmetic rather than as "roughly an order of -# magnitude" -- that phrasing is what went unchecked last time: +# ATTEMPTS drops to 2 to keep the worst case inside the job's 25-minute budget, +# stated as arithmetic rather than as "roughly an order of magnitude" -- that +# phrasing is what went unchecked the first time: # -# 300 + 15 + (300 + 300) + 30 + (300 + 300) = 1245s +# 600 + 15 + (180 + 600) = 1395s, plus ~45s of surrounding steps # -# Under 21 minutes, inside the job's 25 with room for the `cargo check` that -# follows it. -readonly UPDATE_TIMEOUT=300 -readonly INSTALL_TIMEOUT=300 -readonly ATTEMPTS=3 +# The third attempt is not a loss worth arguing for: attempt 2 already retries +# with a refreshed index AND a warm download cache, which covers both the stale +# index and the slow mirror. A third would only repeat attempt 2. +readonly UPDATE_TIMEOUT=180 +readonly INSTALL_TIMEOUT=600 +readonly ATTEMPTS=2 # Elevation on the OUTSIDE, `timeout` on the inside. Review on #408 caught the # ordering and it is not cosmetic: with `timeout` outermost the SIGTERM goes to From 44fb2daa89db35f43ef7b8190c0e77b178efc3d2 Mon Sep 17 00:00:00 2001 From: DoubleGate Date: Wed, 19 Aug 2026 16:42:52 -0400 Subject: [PATCH 3/3] fix(ci): `--` before the package operand Both reviewers raised it independently on #417. `APT_PACKAGE` comes from a workflow `env:` block and never from event data, so this is not closing a live injection path -- but it is one token that makes the guarantee structural rather than dependent on every future caller remembering where the value came from. A value beginning with a hyphen is now an operand, not an option. --- .github/scripts/apt-install-retry.sh | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/.github/scripts/apt-install-retry.sh b/.github/scripts/apt-install-retry.sh index 76fc5e10..23b4ccee 100755 --- a/.github/scripts/apt-install-retry.sh +++ b/.github/scripts/apt-install-retry.sh @@ -117,9 +117,15 @@ readonly ATTEMPTS=2 # run at all, which would break the wrapper rather than degrade it. `env` is a # plain command and needs no such privilege. (Review on #409; both reviewers # raised it independently.) +# `--` before the package name so a value beginning with a hyphen is treated as +# an operand rather than as an option. `APT_PACKAGE` comes from a workflow `env:` +# block and never from event data, so this is not closing a live injection path; +# it is one token that makes the guarantee structural instead of dependent on +# every future caller remembering where the value came from. (Both reviewers +# raised it on #417.) apt_install() { sudo env DEBIAN_FRONTEND=noninteractive timeout "$INSTALL_TIMEOUT" \ - apt-get install -yq --no-install-recommends "$APT_PACKAGE" + apt-get install -yq --no-install-recommends -- "$APT_PACKAGE" } apt_update() {