From 75ae855006fc367a037d16276975a48fa205557f Mon Sep 17 00:00:00 2001 From: Riccardo Persello Date: Mon, 27 Jul 2026 10:23:28 +0200 Subject: [PATCH 01/18] Support arm64 builds alongside amd64 Select architecture-specific downloads from BuildKit's TARGETARCH so the image builds natively on arm64 (e.g. Apple Silicon via OrbStack) as well as amd64: - GitHub Actions runner: linux-x64 / linux-arm64 - Pkl: pkl-linux-amd64 / pkl-linux-aarch64 TARGETARCH is declared without a default, since a default shadows the value the builder injects and would silently fetch amd64 binaries into an arm64 image. Steps fall back to `dpkg --print-architecture` when it is unset so non-BuildKit builds still resolve the host architecture. Downloads now use `curl -f` so a 404 fails the build instead of writing an HTML error page in place of the binary. The Rust, espup and uv toolchains already resolve their own host architecture, and entrypoint.sh has no architecture assumptions. Verified by building --platform linux/arm64 and running the image: pkl 0.30.1 (native), runner 2.336.0, rustc/cargo-nextest on aarch64-unknown-linux-gnu, and the esp toolchain with xtensa-esp-elf-gcc 15.2.0. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01CM6943zQZosnY2QugpiQxf --- Dockerfile | 33 ++++++++++++++++++++++++++------- README.md | 17 +++++++++++++++++ 2 files changed, 43 insertions(+), 7 deletions(-) diff --git a/Dockerfile b/Dockerfile index 4c9a399..2a05743 100644 --- a/Dockerfile +++ b/Dockerfile @@ -3,6 +3,13 @@ FROM ubuntu:24.04 # Prevent interactive prompts during package installation ENV DEBIAN_FRONTEND=noninteractive +# Populated by BuildKit with "amd64" or "arm64". Must be declared WITHOUT a +# default: a default shadows the value the builder injects, which would silently +# fetch the wrong architecture's binaries. Steps below fall back to +# `dpkg --print-architecture` (same amd64/arm64 vocabulary) when it is unset, +# so non-BuildKit builds still resolve the host architecture correctly. +ARG TARGETARCH + # ============================================================================ # Base system dependencies (GitHub Actions Runner) # ============================================================================ @@ -55,7 +62,13 @@ RUN curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash # ============================================================================ # Install Pkl (Apple's configuration language - used by canvas) # ============================================================================ -RUN curl -L -o /usr/local/bin/pkl https://github.com/apple/pkl/releases/download/0.30.1/pkl-linux-amd64 && \ +RUN ARCH="${TARGETARCH:-$(dpkg --print-architecture)}" && \ + case "$ARCH" in \ + amd64) PKL_ARCH=amd64 ;; \ + arm64) PKL_ARCH=aarch64 ;; \ + *) echo "Unsupported architecture: $ARCH" >&2; exit 1 ;; \ + esac && \ + curl -fL -o /usr/local/bin/pkl "https://github.com/apple/pkl/releases/download/0.30.1/pkl-linux-${PKL_ARCH}" && \ chmod +x /usr/local/bin/pkl # ============================================================================ @@ -75,13 +88,19 @@ ENV PATH="/root/.local/bin:${PATH}" RUN mkdir -p /actions-runner WORKDIR /actions-runner -RUN LATEST_TAG=$(curl -s https://api.github.com/repos/actions/runner/releases/latest | jq -r .tag_name) && \ +RUN ARCH="${TARGETARCH:-$(dpkg --print-architecture)}" && \ + case "$ARCH" in \ + amd64) RUNNER_ARCH=x64 ;; \ + arm64) RUNNER_ARCH=arm64 ;; \ + *) echo "Unsupported architecture: $ARCH" >&2; exit 1 ;; \ + esac && \ + LATEST_TAG=$(curl -s https://api.github.com/repos/actions/runner/releases/latest | jq -r .tag_name) && \ RUNNER_VERSION=${LATEST_TAG#v} && \ - echo "Downloading Runner Version: ${RUNNER_VERSION}" && \ - curl -L -o actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz \ - "https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz" && \ - tar xzf actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz && \ - rm actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz + echo "Downloading Runner Version: ${RUNNER_VERSION} (${RUNNER_ARCH})" && \ + curl -fL -o runner.tar.gz \ + "https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-${RUNNER_ARCH}-${RUNNER_VERSION}.tar.gz" && \ + tar xzf runner.tar.gz && \ + rm runner.tar.gz # ============================================================================ # Setup SSH for private repository access (submodules) diff --git a/README.md b/README.md index ad8a82d..46bc2c0 100644 --- a/README.md +++ b/README.md @@ -29,6 +29,23 @@ This runner includes all tools required for the firmware CI pipeline: - **SSH** - Pre-configured with GitHub's host keys for private submodule access - Standard build essentials (`build-essential`, `pkg-config`, `libssl-dev`) +## Architectures + +The image builds for both `linux/amd64` and `linux/arm64` (e.g. Apple Silicon via +OrbStack/Docker Desktop). Architecture-specific downloads (GitHub Actions runner, +Pkl) are selected from BuildKit's `TARGETARCH`; the Rust, ESP (`espup`) and Python +toolchains resolve their own host architecture. + +Docker Compose and `docker build` produce a native image by default. To build +explicitly for one architecture: + +```bash +docker buildx build --platform linux/arm64 -t github-runner . +``` + +> Note: if `TARGETARCH` is unset (a build without BuildKit), the Dockerfile falls +> back to `dpkg --print-architecture`, i.e. the base image's own architecture. + ## Usage ### Environment Variables From adb9482da66bb5f398450a171e68e50bb682e16d Mon Sep 17 00:00:00 2001 From: Riccardo Persello Date: Mon, 27 Jul 2026 10:26:21 +0200 Subject: [PATCH 02/18] Fix maturin being unusable by the runner user uv installed itself and maturin under /root/.local, and the image copied that tree into /home/runner/.local. The copy brought along maturin's launcher symlink, which points at an absolute path inside /root/.local/share/uv/tools. /root is mode 0700, so the unprivileged runner user that actually executes jobs could not traverse it: $ gosu runner maturin --version error: exec: "maturin": executable file not found in $PATH It worked as root, which is why this went unnoticed. Both architectures were affected. Install uv into /usr/local/bin and its tools into /opt/uv (via UV_INSTALL_DIR / UV_TOOL_BIN_DIR / UV_TOOL_DIR) so they sit on the shared PATH and are readable by every user. This also removes the need to copy the tree into the runner's home at all. Verified in the arm64 image: `gosu runner maturin --version` reports 1.14.1 and `maturin list-python` resolves CPython 3.12. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01CM6943zQZosnY2QugpiQxf --- Dockerfile | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/Dockerfile b/Dockerfile index 2a05743..b1234e8 100644 --- a/Dockerfile +++ b/Dockerfile @@ -74,13 +74,15 @@ RUN ARCH="${TARGETARCH:-$(dpkg --print-architecture)}" && \ # ============================================================================ # Install uv (fast Python package manager) and maturin (Rust-Python build tool) # ============================================================================ -RUN curl -LsSf https://astral.sh/uv/install.sh | sh && \ - # Add uv to PATH - . $HOME/.local/bin/env && \ - # Install maturin globally via uv - uv tool install maturin +# Installed into shared, world-readable locations rather than under /root, which +# is mode 0700: a tool symlinked out of /root is unusable by the unprivileged +# runner user that actually executes jobs. +ENV UV_TOOL_DIR=/opt/uv/tools -ENV PATH="/root/.local/bin:${PATH}" +RUN curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR=/usr/local/bin sh && \ + # Install maturin globally, with its launcher on the shared PATH + UV_TOOL_BIN_DIR=/usr/local/bin uv tool install maturin && \ + chmod -R a+rX /opt/uv # ============================================================================ # Create runner directory and download GitHub Actions Runner @@ -124,9 +126,8 @@ RUN useradd -m runner && \ cp -r /root/.rustup/* /home/runner/.rustup/ 2>/dev/null || true && \ # Copy export-esp.sh to runner home cp /root/export-esp.sh /home/runner/export-esp.sh 2>/dev/null || true && \ - # Copy uv and tools to runner user - mkdir -p /home/runner/.local && \ - cp -r /root/.local/* /home/runner/.local/ 2>/dev/null || true && \ + # uv and its tools (maturin) live in /usr/local/bin and /opt/uv, which are + # already on the shared PATH and readable by this user — nothing to copy. # Copy SSH config to runner user mkdir -p /home/runner/.ssh && \ cp /root/.ssh/known_hosts /home/runner/.ssh/ && \ From 75c4e46280689b5a9ada210031326546201de438 Mon Sep 17 00:00:00 2001 From: Riccardo Persello Date: Mon, 27 Jul 2026 11:10:09 +0200 Subject: [PATCH 03/18] Add shared cargo cache and per-replica resource limits A runner executes one job at a time, so parallelism is purely the replica count. Default it to 4 and bound what each replica may consume. Sized for a 16-core / 64 GB host: 4 replicas x 4 CPUs x 10 GB, leaving headroom for the host OS. CARGO_BUILD_JOBS is pinned to RUNNER_CPUS because cargo otherwise sizes its thread pool from the host core count, so each replica would spawn ~16 threads and N replicas would oversubscribe the machine N-fold; the cpus limit alone only throttles the result rather than preventing the thrashing. All three knobs are overridable via RUNNER_COUNT / RUNNER_CPUS / RUNNER_MEMORY. Replicas now share a cargo-registry volume instead of each re-downloading the full dependency set. Only the registry is shared, not the whole CARGO_HOME: cargo locks that directory so concurrent access is safe, whereas a shared target/ dir would race. entrypoint.sh repairs ownership of the registry volume when it comes back root-owned, which happens for a volume not seeded from the image and would otherwise silently break every build. Also drops the runner-data volume, which was declared but never mounted. Verified on the arm64 image: compose applies the limits outside swarm (NanoCpus=4000000000, Memory=10737418240) across 4 replicas; the runner user can write to the registry volume both when seeded from the image and after the root-owned repair path; and a crate fetched in one container is served to a second via `cargo fetch --offline`, confirming real sharing. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01CM6943zQZosnY2QugpiQxf --- README.md | 37 +++++++++++++++++++++++++++++++++++-- docker-compose.yml | 22 ++++++++++++++++++++-- entrypoint.sh | 8 ++++++++ 3 files changed, 63 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 46bc2c0..e93f9b7 100644 --- a/README.md +++ b/README.md @@ -57,6 +57,35 @@ docker buildx build --platform linux/arm64 -t github-runner . | `RUNNER_TOKEN` | One of `GITHUB_PAT` / `RUNNER_TOKEN` | Static runner registration token from GitHub. Expires ~1 hour after creation, so restarts after that will fail unless refreshed. Ignored if `GITHUB_PAT` is set. | | `RUNNER_NAME` | No | Base name for the runner (default: `runner`) | | `RUNNER_LABELS` | No | Comma-separated labels for the runner | +| `RUNNER_COUNT` | No | Number of runner replicas (default: `4`) | +| `RUNNER_CPUS` | No | CPUs per replica; also caps `CARGO_BUILD_JOBS` (default: `4`) | +| `RUNNER_MEMORY` | No | Memory per replica (default: `10g`) | + +### Parallel Jobs + +A GitHub Actions runner executes **one job at a time** — there is no concurrency +setting inside the runner. Total parallelism is therefore just `RUNNER_COUNT`. + +The defaults (4 replicas x 4 CPUs x 10 GB) target a 16-core / 64 GB host. Each +replica gets a hard CPU and memory limit, and `CARGO_BUILD_JOBS` is pinned to +`RUNNER_CPUS` — without that, cargo sizes its thread pool from the *host* core +count and every replica would spawn ~16 threads, oversubscribing the machine. + +Raising `RUNNER_COUNT` past the core count trades per-job latency for throughput: +8 replicas x 2 CPUs runs twice as many jobs, but each Rust build is much slower. +Prefer more replicas only if your jobs are mostly light (fmt, clippy, tests) +rather than full firmware builds. + +```bash +RUNNER_COUNT=8 RUNNER_CPUS=2 RUNNER_MEMORY=6g docker compose up -d --build +``` + +### Caching + +Replicas share a `cargo-registry` volume, so crates are downloaded once rather +than once per replica. Only the registry is shared — cargo locks it, making +concurrent access safe, whereas a shared `target/` directory would race. +Build artifacts are **not** shared or persisted across `docker compose down`. ### Running with Docker Compose @@ -65,6 +94,10 @@ docker buildx build --platform linux/arm64 -t github-runner . export URL=https://github.com/jkuracing export GITHUB_PAT= -# Start the runner -docker compose up -d +# Start the runners +docker compose up -d --build ``` + +> Always pass `--build`. Plain `docker compose up -d` only builds when the image +> is missing, so it will happily keep running a stale image after the Dockerfile +> or `entrypoint.sh` changes. diff --git a/docker-compose.yml b/docker-compose.yml index 235563e..8201dd6 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -9,8 +9,26 @@ services: RUNNER_TOKEN: ${RUNNER_TOKEN} RUNNER_NAME: ${RUNNER_NAME} RUNNER_LABELS: ${RUNNER_LABELS} + # Match cargo's internal parallelism to this replica's CPU allotment. + # cargo defaults to one codegen unit per *host* core, so without this each + # replica would spawn ~16 threads and N replicas would oversubscribe the + # machine N-fold. The cpus limit below only throttles the result; capping + # the thread count is what actually avoids the thrashing. + CARGO_BUILD_JOBS: ${RUNNER_CPUS:-4} + volumes: + # Shared crate download cache. Only the registry is shared, not the whole + # CARGO_HOME: cargo locks this directory, so concurrent replicas are safe, + # whereas a shared target/ dir would race. Without this every replica + # re-downloads the full dependency set on a cold start. + - cargo-registry:/home/runner/.cargo/registry deploy: - replicas: ${RUNNER_COUNT:-1} + replicas: ${RUNNER_COUNT:-4} + resources: + limits: + # Sized for a 16-core / 64 GB host, leaving headroom for macOS itself. + # 4 x 4 CPUs saturates the machine without oversubscribing it. + cpus: ${RUNNER_CPUS:-4} + memory: ${RUNNER_MEMORY:-10g} volumes: - runner-data: + cargo-registry: diff --git a/entrypoint.sh b/entrypoint.sh index b74e239..96d66b7 100644 --- a/entrypoint.sh +++ b/entrypoint.sh @@ -18,6 +18,14 @@ FULL_RUNNER_NAME="${RUNNER_NAME}-${HOSTNAME}" echo "Fixing permissions for /actions-runner..." chown -R runner:runner /actions-runner +# The shared cargo registry is a named volume. Docker seeds it from the image +# with the right ownership, but a volume created before that directory existed +# (or by another image) comes back root-owned and silently breaks every build. +if [[ -d /home/runner/.cargo/registry ]] && [[ "$(stat -c %U /home/runner/.cargo/registry)" != "runner" ]]; then + echo "Fixing permissions for the shared cargo registry..." + chown -R runner:runner /home/runner/.cargo/registry +fi + # Fetches a short-lived token ($1: "registration-token" or "remove-token") from the # GitHub API, using GITHUB_PAT. Prints the token on stdout, returns non-zero on failure. fetch_runner_token() { From 6f0cafd253e11105238bf77c92412fcd020e7c02 Mon Sep 17 00:00:00 2001 From: Riccardo Persello Date: Mon, 27 Jul 2026 11:12:50 +0200 Subject: [PATCH 04/18] Default RUNNER_LABELS to fw-builder Every job in the firmware repo's firmware_ci.yml targets `runs-on: labels: [fw-builder]`. With no labels set, a runner registers successfully and then sits idle forever, since no job ever matches it. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01CM6943zQZosnY2QugpiQxf --- docker-compose.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docker-compose.yml b/docker-compose.yml index 8201dd6..72bcb53 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -8,7 +8,9 @@ services: GITHUB_PAT: ${GITHUB_PAT} RUNNER_TOKEN: ${RUNNER_TOKEN} RUNNER_NAME: ${RUNNER_NAME} - RUNNER_LABELS: ${RUNNER_LABELS} + # firmware_ci.yml targets `runs-on: labels: [fw-builder]`, so a runner + # without this label is registered but never assigned any job. + RUNNER_LABELS: ${RUNNER_LABELS:-fw-builder} # Match cargo's internal parallelism to this replica's CPU allotment. # cargo defaults to one codegen unit per *host* core, so without this each # replica would spawn ~16 threads and N replicas would oversubscribe the From ab95f76a2c394775216e6c5ec577beb192a319bf Mon Sep 17 00:00:00 2001 From: Riccardo Persello Date: Mon, 27 Jul 2026 11:36:42 +0200 Subject: [PATCH 05/18] Scale to 8 replicas at 2 CPUs / 6 GB each MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Observed CI state shows runs queuing while host CPU sits idle, so job throughput is runner-starved rather than compute-bound. A single run only reaches 5 concurrent jobs, but firmware_ci.yml keys its concurrency group per branch, so several runs execute simultaneously and jobs queue globally — more replicas do get used. Memory is the binding constraint: 8 x 6 GB = 48 GB of the ~58 GB the OrbStack VM exposes, leaving host headroom. CPUs are oversubscribed 1:1 (8 x 2 = 16) since jobs spend much of their wall time on network and link steps rather than pegged compute. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01CM6943zQZosnY2QugpiQxf --- README.md | 22 +++++++++++++--------- docker-compose.yml | 15 +++++++++------ 2 files changed, 22 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index e93f9b7..2b78cd8 100644 --- a/README.md +++ b/README.md @@ -57,27 +57,31 @@ docker buildx build --platform linux/arm64 -t github-runner . | `RUNNER_TOKEN` | One of `GITHUB_PAT` / `RUNNER_TOKEN` | Static runner registration token from GitHub. Expires ~1 hour after creation, so restarts after that will fail unless refreshed. Ignored if `GITHUB_PAT` is set. | | `RUNNER_NAME` | No | Base name for the runner (default: `runner`) | | `RUNNER_LABELS` | No | Comma-separated labels for the runner | -| `RUNNER_COUNT` | No | Number of runner replicas (default: `4`) | -| `RUNNER_CPUS` | No | CPUs per replica; also caps `CARGO_BUILD_JOBS` (default: `4`) | -| `RUNNER_MEMORY` | No | Memory per replica (default: `10g`) | +| `RUNNER_COUNT` | No | Number of runner replicas (default: `8`) | +| `RUNNER_CPUS` | No | CPUs per replica; also caps `CARGO_BUILD_JOBS` (default: `2`) | +| `RUNNER_MEMORY` | No | Memory per replica (default: `6g`) | ### Parallel Jobs A GitHub Actions runner executes **one job at a time** — there is no concurrency setting inside the runner. Total parallelism is therefore just `RUNNER_COUNT`. -The defaults (4 replicas x 4 CPUs x 10 GB) target a 16-core / 64 GB host. Each +The defaults (8 replicas x 2 CPUs x 6 GB) target a 16-core / 64 GB host. Each replica gets a hard CPU and memory limit, and `CARGO_BUILD_JOBS` is pinned to `RUNNER_CPUS` — without that, cargo sizes its thread pool from the *host* core count and every replica would spawn ~16 threads, oversubscribing the machine. -Raising `RUNNER_COUNT` past the core count trades per-job latency for throughput: -8 replicas x 2 CPUs runs twice as many jobs, but each Rust build is much slower. -Prefer more replicas only if your jobs are mostly light (fmt, clippy, tests) -rather than full firmware builds. +**Memory, not CPU, is what limits the replica count.** 8 x 6 GB = 48 GB of the +~58 GB the OrbStack VM exposes. Raising `RUNNER_COUNT` without lowering +`RUNNER_MEMORY` will overcommit and get builds OOM-killed. + +A single CI run only reaches 5 concurrent jobs (four checks in parallel, then +three builds behind `needs`). The reason more replicas still help is that +`concurrency` in `firmware_ci.yml` is keyed per *branch*, so several runs +execute at once and jobs queue globally. ```bash -RUNNER_COUNT=8 RUNNER_CPUS=2 RUNNER_MEMORY=6g docker compose up -d --build +RUNNER_COUNT=4 RUNNER_CPUS=4 RUNNER_MEMORY=10g docker compose up -d --build ``` ### Caching diff --git a/docker-compose.yml b/docker-compose.yml index 72bcb53..a0dbad9 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -16,7 +16,7 @@ services: # replica would spawn ~16 threads and N replicas would oversubscribe the # machine N-fold. The cpus limit below only throttles the result; capping # the thread count is what actually avoids the thrashing. - CARGO_BUILD_JOBS: ${RUNNER_CPUS:-4} + CARGO_BUILD_JOBS: ${RUNNER_CPUS:-2} volumes: # Shared crate download cache. Only the registry is shared, not the whole # CARGO_HOME: cargo locks this directory, so concurrent replicas are safe, @@ -24,13 +24,16 @@ services: # re-downloads the full dependency set on a cold start. - cargo-registry:/home/runner/.cargo/registry deploy: - replicas: ${RUNNER_COUNT:-4} + replicas: ${RUNNER_COUNT:-8} resources: limits: - # Sized for a 16-core / 64 GB host, leaving headroom for macOS itself. - # 4 x 4 CPUs saturates the machine without oversubscribing it. - cpus: ${RUNNER_CPUS:-4} - memory: ${RUNNER_MEMORY:-10g} + # Sized for a 16-core / 64 GB host. Memory is the binding constraint, + # not CPU: 8 x 6g = 48 GB of the ~58 GB the VM exposes, leaving + # headroom for the host. CPUs are deliberately oversubscribed 1:1 + # (8 x 2 = 16) because jobs spend much of their wall time on network + # and link steps rather than pegged compute. + cpus: ${RUNNER_CPUS:-2} + memory: ${RUNNER_MEMORY:-6g} volumes: cargo-registry: From bce3379764f437a6eaf9470363efc1afafce902b Mon Sep 17 00:00:00 2001 From: Riccardo Persello Date: Mon, 27 Jul 2026 11:49:06 +0200 Subject: [PATCH 06/18] Persist sccache per replica across container recreation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Containers are not long-lived, so $HOME/.cache/sccache in the writable layer is discarded on every recreate and each new container recompiles from cold. setup-rust-dual in the firmware repo configures a 25 GB sccache there and describes it as living on persistent runner storage, so that cache is worth keeping across recreates. It cannot be one shared volume. sccache maintains its LRU index in memory per server process, so several containers pointed at one cache directory evict against each other and corrupt it. Since every replica of a scaled service shares one set of volumes, `deploy.replicas` cannot express per-replica storage — the replicas are now eight explicit services built from a YAML anchor, each with its own sccache volume. The crate registry stays shared, which is safe because cargo locks it. _work is still not persisted: the stale submodule target/ it would preserve is exactly what was breaking builds. The Dockerfile pre-creates the sccache directory so its volume is seeded with runner ownership, and the entrypoint's ownership repair now covers both volume paths — a volume that is non-empty and root-owned is not re-seeded by Docker and would otherwise be unwritable by the runner user. Verified: compose resolves 8 services each with a distinct sccache volume and a shared cargo-registry; the runner user can write the sccache volume when seeded from the image; and with a deliberately root-owned non-empty volume, an unrepaired write fails with EACCES while the entrypoint loop restores ownership and the write succeeds. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01CM6943zQZosnY2QugpiQxf --- Dockerfile | 4 ++ README.md | 42 +++++++++++++----- docker-compose.yml | 107 ++++++++++++++++++++++++++++++--------------- entrypoint.sh | 16 ++++--- 4 files changed, 115 insertions(+), 54 deletions(-) diff --git a/Dockerfile b/Dockerfile index b1234e8..624058d 100644 --- a/Dockerfile +++ b/Dockerfile @@ -128,6 +128,10 @@ RUN useradd -m runner && \ cp /root/export-esp.sh /home/runner/export-esp.sh 2>/dev/null || true && \ # uv and its tools (maturin) live in /usr/local/bin and /opt/uv, which are # already on the shared PATH and readable by this user — nothing to copy. + # Pre-create the sccache directory so its named volume is seeded with runner + # ownership. A volume mounted over a path that does not exist in the image is + # created root-owned, which the unprivileged runner cannot write to. + mkdir -p /home/runner/.cache/sccache && \ # Copy SSH config to runner user mkdir -p /home/runner/.ssh && \ cp /root/.ssh/known_hosts /home/runner/.ssh/ && \ diff --git a/README.md b/README.md index 2b78cd8..0469300 100644 --- a/README.md +++ b/README.md @@ -57,7 +57,6 @@ docker buildx build --platform linux/arm64 -t github-runner . | `RUNNER_TOKEN` | One of `GITHUB_PAT` / `RUNNER_TOKEN` | Static runner registration token from GitHub. Expires ~1 hour after creation, so restarts after that will fail unless refreshed. Ignored if `GITHUB_PAT` is set. | | `RUNNER_NAME` | No | Base name for the runner (default: `runner`) | | `RUNNER_LABELS` | No | Comma-separated labels for the runner | -| `RUNNER_COUNT` | No | Number of runner replicas (default: `8`) | | `RUNNER_CPUS` | No | CPUs per replica; also caps `CARGO_BUILD_JOBS` (default: `2`) | | `RUNNER_MEMORY` | No | Memory per replica (default: `6g`) | @@ -66,30 +65,49 @@ docker buildx build --platform linux/arm64 -t github-runner . A GitHub Actions runner executes **one job at a time** — there is no concurrency setting inside the runner. Total parallelism is therefore just `RUNNER_COUNT`. -The defaults (8 replicas x 2 CPUs x 6 GB) target a 16-core / 64 GB host. Each -replica gets a hard CPU and memory limit, and `CARGO_BUILD_JOBS` is pinned to -`RUNNER_CPUS` — without that, cargo sizes its thread pool from the *host* core -count and every replica would spawn ~16 threads, oversubscribing the machine. +Eight replicas (`runner-1` .. `runner-8`) are declared explicitly in +`docker-compose.yml`, at 2 CPUs and 6 GB each, sized for a 16-core / 64 GB host. +`CARGO_BUILD_JOBS` is pinned to `RUNNER_CPUS` — without that, cargo sizes its +thread pool from the *host* core count and every replica would spawn ~16 +threads, oversubscribing the machine. **Memory, not CPU, is what limits the replica count.** 8 x 6 GB = 48 GB of the -~58 GB the OrbStack VM exposes. Raising `RUNNER_COUNT` without lowering -`RUNNER_MEMORY` will overcommit and get builds OOM-killed. +~58 GB the OrbStack VM exposes. Adding replicas without lowering `RUNNER_MEMORY` +will overcommit and get builds OOM-killed. A single CI run only reaches 5 concurrent jobs (four checks in parallel, then three builds behind `needs`). The reason more replicas still help is that `concurrency` in `firmware_ci.yml` is keyed per *branch*, so several runs execute at once and jobs queue globally. +To run fewer runners, name the services; to run bigger ones, raise the limits: + ```bash -RUNNER_COUNT=4 RUNNER_CPUS=4 RUNNER_MEMORY=10g docker compose up -d --build +docker compose up -d --build runner-1 runner-2 runner-3 +RUNNER_CPUS=4 RUNNER_MEMORY=10g docker compose up -d --build ``` +> Replicas are separate services rather than `deploy.replicas` because a scaled +> service shares one set of volumes, and sccache cannot safely share a cache +> directory between concurrent server processes (see below). + ### Caching -Replicas share a `cargo-registry` volume, so crates are downloaded once rather -than once per replica. Only the registry is shared — cargo locks it, making -concurrent access safe, whereas a shared `target/` directory would race. -Build artifacts are **not** shared or persisted across `docker compose down`. +Two caches survive container recreation: + +- **`cargo-registry`** — shared by all replicas. Crates are downloaded once + rather than once per runner. Sharing is safe because cargo locks the registry. +- **`sccache-N`** — one volume *per replica*. `setup-rust-dual` in the firmware + repo points sccache at `$HOME/.cache/sccache`, and sccache keeps an in-memory + LRU index per server process, so several containers sharing one cache + directory would evict against each other and corrupt it. + +The runner's `_work` directory is deliberately **not** persisted. The firmware +workflow checks out with `clean: false` to reuse `target/`, but a stale +submodule `target/` surviving `git submodule deinit` is what produced +`could not parse/generate dep info ... No such file or directory` build +failures. sccache is content-hashed and immune to that staleness, so it is the +right layer to persist; `_work` is not. ### Running with Docker Compose diff --git a/docker-compose.yml b/docker-compose.yml index a0dbad9..b824987 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,39 +1,76 @@ +# Replicas are declared explicitly rather than via `deploy.replicas` because +# every replica of a scaled service shares one set of volumes. sccache keeps an +# in-memory LRU index per server process, so pointing several containers at one +# cache directory lets them evict against each other and corrupt it. Each runner +# therefore needs its OWN sccache volume, which requires its own service. +# +# The crate registry is different: cargo locks it, so a single shared volume is +# safe and avoids N copies of the same downloads. +# +# `docker compose up -d --build` starts all 8. To run fewer, name them: +# docker compose up -d --build runner-1 runner-2 runner-3 +x-runner: &runner + build: . + restart: on-failure:5 + stop_grace_period: 5m + environment: + URL: ${URL} + GITHUB_PAT: ${GITHUB_PAT} + RUNNER_TOKEN: ${RUNNER_TOKEN} + RUNNER_NAME: ${RUNNER_NAME} + # firmware_ci.yml targets `runs-on: labels: [fw-builder]`, so a runner + # without this label is registered but never assigned any job. + RUNNER_LABELS: ${RUNNER_LABELS:-fw-builder} + # Match cargo's internal parallelism to this replica's CPU allotment. + # cargo defaults to one codegen unit per *host* core, so without this each + # replica would spawn ~16 threads and 8 replicas would oversubscribe the + # machine 8-fold. The cpus limit below only throttles the result; capping + # the thread count is what actually avoids the thrashing. + CARGO_BUILD_JOBS: ${RUNNER_CPUS:-2} + deploy: + resources: + limits: + # Sized for a 16-core / 64 GB host. Memory is the binding constraint, + # not CPU: 8 x 6g = 48 GB of the ~58 GB the VM exposes, leaving + # headroom for the host. CPUs are deliberately oversubscribed 1:1 + # (8 x 2 = 16) because jobs spend much of their wall time on network + # and link steps rather than pegged compute. + cpus: ${RUNNER_CPUS:-2} + memory: ${RUNNER_MEMORY:-6g} + services: - github-runner: - build: . - restart: on-failure:5 - stop_grace_period: 5m - environment: - URL: ${URL} - GITHUB_PAT: ${GITHUB_PAT} - RUNNER_TOKEN: ${RUNNER_TOKEN} - RUNNER_NAME: ${RUNNER_NAME} - # firmware_ci.yml targets `runs-on: labels: [fw-builder]`, so a runner - # without this label is registered but never assigned any job. - RUNNER_LABELS: ${RUNNER_LABELS:-fw-builder} - # Match cargo's internal parallelism to this replica's CPU allotment. - # cargo defaults to one codegen unit per *host* core, so without this each - # replica would spawn ~16 threads and N replicas would oversubscribe the - # machine N-fold. The cpus limit below only throttles the result; capping - # the thread count is what actually avoids the thrashing. - CARGO_BUILD_JOBS: ${RUNNER_CPUS:-2} - volumes: - # Shared crate download cache. Only the registry is shared, not the whole - # CARGO_HOME: cargo locks this directory, so concurrent replicas are safe, - # whereas a shared target/ dir would race. Without this every replica - # re-downloads the full dependency set on a cold start. - - cargo-registry:/home/runner/.cargo/registry - deploy: - replicas: ${RUNNER_COUNT:-8} - resources: - limits: - # Sized for a 16-core / 64 GB host. Memory is the binding constraint, - # not CPU: 8 x 6g = 48 GB of the ~58 GB the VM exposes, leaving - # headroom for the host. CPUs are deliberately oversubscribed 1:1 - # (8 x 2 = 16) because jobs spend much of their wall time on network - # and link steps rather than pegged compute. - cpus: ${RUNNER_CPUS:-2} - memory: ${RUNNER_MEMORY:-6g} + runner-1: + <<: *runner + volumes: [cargo-registry:/home/runner/.cargo/registry, sccache-1:/home/runner/.cache/sccache] + runner-2: + <<: *runner + volumes: [cargo-registry:/home/runner/.cargo/registry, sccache-2:/home/runner/.cache/sccache] + runner-3: + <<: *runner + volumes: [cargo-registry:/home/runner/.cargo/registry, sccache-3:/home/runner/.cache/sccache] + runner-4: + <<: *runner + volumes: [cargo-registry:/home/runner/.cargo/registry, sccache-4:/home/runner/.cache/sccache] + runner-5: + <<: *runner + volumes: [cargo-registry:/home/runner/.cargo/registry, sccache-5:/home/runner/.cache/sccache] + runner-6: + <<: *runner + volumes: [cargo-registry:/home/runner/.cargo/registry, sccache-6:/home/runner/.cache/sccache] + runner-7: + <<: *runner + volumes: [cargo-registry:/home/runner/.cargo/registry, sccache-7:/home/runner/.cache/sccache] + runner-8: + <<: *runner + volumes: [cargo-registry:/home/runner/.cargo/registry, sccache-8:/home/runner/.cache/sccache] volumes: cargo-registry: + sccache-1: + sccache-2: + sccache-3: + sccache-4: + sccache-5: + sccache-6: + sccache-7: + sccache-8: diff --git a/entrypoint.sh b/entrypoint.sh index 96d66b7..653b5b0 100644 --- a/entrypoint.sh +++ b/entrypoint.sh @@ -18,13 +18,15 @@ FULL_RUNNER_NAME="${RUNNER_NAME}-${HOSTNAME}" echo "Fixing permissions for /actions-runner..." chown -R runner:runner /actions-runner -# The shared cargo registry is a named volume. Docker seeds it from the image -# with the right ownership, but a volume created before that directory existed -# (or by another image) comes back root-owned and silently breaks every build. -if [[ -d /home/runner/.cargo/registry ]] && [[ "$(stat -c %U /home/runner/.cargo/registry)" != "runner" ]]; then - echo "Fixing permissions for the shared cargo registry..." - chown -R runner:runner /home/runner/.cargo/registry -fi +# These are named volumes. Docker seeds them from the image with the right +# ownership, but a volume created before the directory existed in the image (or +# by another image) comes back root-owned and silently breaks every build. +for vol_dir in /home/runner/.cargo/registry /home/runner/.cache/sccache; do + if [[ -d "$vol_dir" ]] && [[ "$(stat -c %U "$vol_dir")" != "runner" ]]; then + echo "Fixing permissions for ${vol_dir}..." + chown -R runner:runner "$vol_dir" + fi +done # Fetches a short-lived token ($1: "registration-token" or "remove-token") from the # GitHub API, using GITHUB_PAT. Prints the token on stdout, returns non-zero on failure. From 862ab62c1163a8856cf3d1013b620c1205e13aab Mon Sep 17 00:00:00 2001 From: Riccardo Persello Date: Mon, 27 Jul 2026 12:12:18 +0200 Subject: [PATCH 07/18] Give each runner its own cargo registry volume Sharing one cargo-registry volume across the 8 runners introduced a new CI failure absent from every run before it: error: could not compile `crc32fast` (lib) Caused by: could not execute process `.../bin/rustc --crate-name crc32fast .../registry/src/index.crates.io-*/crc32fast-1.5.0/src/lib.rs` Caused by: No such file or directory (os error 2) `could not execute process` appears 0 times across runs predating the shared volume and immediately after it, with the vanished path inside the shared registry. Unpacked sources under registry/src are removed mid-compile when another container's cargo garbage-collects the global cache, so the rustc spawn fails on a working directory that no longer exists. Cargo's package-cache lock does not cover a build for its whole duration, and it cannot arbitrate between separate containers. Give each runner its own registry volume, matching sccache. This costs N copies of the crate downloads and removes the only remaining shared mutable state between concurrently building runners. Note this is distinct from the pre-existing `could not parse/generate dep info` failures, which point at a submodule's target/ rather than the registry and are addressed separately in the firmware repo. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01CM6943zQZosnY2QugpiQxf --- README.md | 21 +++++++++++++-------- docker-compose.yml | 47 +++++++++++++++++++++++++++++++--------------- 2 files changed, 45 insertions(+), 23 deletions(-) diff --git a/README.md b/README.md index 0469300..318003d 100644 --- a/README.md +++ b/README.md @@ -93,14 +93,19 @@ RUNNER_CPUS=4 RUNNER_MEMORY=10g docker compose up -d --build ### Caching -Two caches survive container recreation: - -- **`cargo-registry`** — shared by all replicas. Crates are downloaded once - rather than once per runner. Sharing is safe because cargo locks the registry. -- **`sccache-N`** — one volume *per replica*. `setup-rust-dual` in the firmware - repo points sccache at `$HOME/.cache/sccache`, and sccache keeps an in-memory - LRU index per server process, so several containers sharing one cache - directory would evict against each other and corrupt it. +Two caches survive container recreation, both **per replica**: + +- **`cargo-registry-N`** — the crate download cache. +- **`sccache-N`** — the compiler cache. `setup-rust-dual` in the firmware repo + points sccache at `$HOME/.cache/sccache`. + +Neither may be shared between replicas. sccache keeps its LRU index in memory +per server process, so containers sharing one directory evict against each +other. The registry was shared in an earlier revision and broke CI: unpacked +sources under `registry/src` disappear mid-compile when another container's +cargo garbage-collects the global cache, producing +`could not execute process ... No such file or directory`. The cost of not +sharing is N copies of the same crate downloads, which is the right trade. The runner's `_work` directory is deliberately **not** persisted. The firmware workflow checks out with `clean: false` to reuse `target/`, but a stale diff --git a/docker-compose.yml b/docker-compose.yml index b824987..83d7a05 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,11 +1,21 @@ # Replicas are declared explicitly rather than via `deploy.replicas` because -# every replica of a scaled service shares one set of volumes. sccache keeps an -# in-memory LRU index per server process, so pointing several containers at one -# cache directory lets them evict against each other and corrupt it. Each runner -# therefore needs its OWN sccache volume, which requires its own service. +# every replica of a scaled service shares one set of volumes, and NOTHING here +# is safe to share between concurrently building runners: # -# The crate registry is different: cargo locks it, so a single shared volume is -# safe and avoids N copies of the same downloads. +# - sccache keeps its LRU index in memory per server process, so several +# containers on one cache directory evict against each other. +# - The cargo registry was shared here initially and caused real CI failures: +# error: could not compile `crc32fast` (lib) +# Caused by: could not execute process `.../bin/rustc --crate-name +# crc32fast .../registry/src/index.crates.io-*/crc32fast-1.5.0/src/lib.rs` +# Caused by: No such file or directory (os error 2) +# Unpacked sources under registry/src vanish mid-compile when another +# container's cargo garbage-collects the global cache, so the spawn fails on +# a working directory that no longer exists. Cargo's package-cache lock does +# not protect a build for its whole duration across separate containers. +# +# Each runner therefore gets its own registry AND sccache volume, which is only +# expressible as its own service. The cost is N copies of the crate downloads. # # `docker compose up -d --build` starts all 8. To run fewer, name them: # docker compose up -d --build runner-1 runner-2 runner-3 @@ -41,31 +51,38 @@ x-runner: &runner services: runner-1: <<: *runner - volumes: [cargo-registry:/home/runner/.cargo/registry, sccache-1:/home/runner/.cache/sccache] + volumes: [cargo-registry-1:/home/runner/.cargo/registry, sccache-1:/home/runner/.cache/sccache] runner-2: <<: *runner - volumes: [cargo-registry:/home/runner/.cargo/registry, sccache-2:/home/runner/.cache/sccache] + volumes: [cargo-registry-2:/home/runner/.cargo/registry, sccache-2:/home/runner/.cache/sccache] runner-3: <<: *runner - volumes: [cargo-registry:/home/runner/.cargo/registry, sccache-3:/home/runner/.cache/sccache] + volumes: [cargo-registry-3:/home/runner/.cargo/registry, sccache-3:/home/runner/.cache/sccache] runner-4: <<: *runner - volumes: [cargo-registry:/home/runner/.cargo/registry, sccache-4:/home/runner/.cache/sccache] + volumes: [cargo-registry-4:/home/runner/.cargo/registry, sccache-4:/home/runner/.cache/sccache] runner-5: <<: *runner - volumes: [cargo-registry:/home/runner/.cargo/registry, sccache-5:/home/runner/.cache/sccache] + volumes: [cargo-registry-5:/home/runner/.cargo/registry, sccache-5:/home/runner/.cache/sccache] runner-6: <<: *runner - volumes: [cargo-registry:/home/runner/.cargo/registry, sccache-6:/home/runner/.cache/sccache] + volumes: [cargo-registry-6:/home/runner/.cargo/registry, sccache-6:/home/runner/.cache/sccache] runner-7: <<: *runner - volumes: [cargo-registry:/home/runner/.cargo/registry, sccache-7:/home/runner/.cache/sccache] + volumes: [cargo-registry-7:/home/runner/.cargo/registry, sccache-7:/home/runner/.cache/sccache] runner-8: <<: *runner - volumes: [cargo-registry:/home/runner/.cargo/registry, sccache-8:/home/runner/.cache/sccache] + volumes: [cargo-registry-8:/home/runner/.cargo/registry, sccache-8:/home/runner/.cache/sccache] volumes: - cargo-registry: + cargo-registry-1: + cargo-registry-2: + cargo-registry-3: + cargo-registry-4: + cargo-registry-5: + cargo-registry-6: + cargo-registry-7: + cargo-registry-8: sccache-1: sccache-2: sccache-3: From 3f0f1e680c9b93c7754f754077b5a5a9d4413abd Mon Sep 17 00:00:00 2001 From: Riccardo Persello Date: Sat, 8 Aug 2026 12:58:53 +0200 Subject: [PATCH 08/18] Stop the runner self-update from killing the whole fleet All eight replicas had been offline for about ten days, which also meant firmware CI could not run at all. The failure was silent: nothing was left running to report it. The runner self-updates in place, and a post-update runner drops a `.runner_migrated` marker beside its config. `config.sh` treats that marker ALONE as proof the runner is already configured -- confirmed by creating only `.runner_migrated` and passing a deliberately bogus token, which fails with "Cannot configure the runner because it is already configured" without even attempting to authenticate. The cleanup here stopped at `.credentials_rsaparams`, so every replica that had auto-updated exited 1 on its next restart. Deleting the marker is correct rather than expedient: this entrypoint always reconfigures from a freshly minted registration token, so no migrated state is worth preserving across a restart. `restart: on-failure:5` turned that per-restart failure into a permanent one -- five retries were spent in seconds, after which Docker left the containers dead. A runner fleet should heal rather than latch off, so it becomes `unless-stopped`. The entrypoint mints one token per start, so even a genuinely broken image loops visibly in the logs instead of failing silently. Default RUNNER_TOKEN and RUNNER_NAME to empty as well. Both are optional when GITHUB_PAT is set, but leaving them unset made `docker compose` print two warnings per service -- sixteen lines that buried the real error underneath. Regression test: plant `.runner_migrated` in a live container, restart it, and confirm the count of "Listening for Jobs" lines increases. Do not test this by grepping `docker logs | tail -N` for that string without counting: a container that booted fine and then broke still has the line in its history, which reports a crash-looping runner as healthy. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_018cbZiFrvjiXMRHLq1L9PDf --- docker-compose.yml | 17 ++++++++++++++--- entrypoint.sh | 20 ++++++++++++++++++-- 2 files changed, 32 insertions(+), 5 deletions(-) diff --git a/docker-compose.yml b/docker-compose.yml index 83d7a05..2152f78 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -21,13 +21,24 @@ # docker compose up -d --build runner-1 runner-2 runner-3 x-runner: &runner build: . - restart: on-failure:5 + # `on-failure:5` used to be the policy here and it cost the fleet ten days of + # downtime: a bug in entrypoint.sh's config cleanup made every replica exit 1 + # on restart, the five retries were spent in seconds, and Docker then left all + # eight containers dead with no surviving process to notice. A runner fleet + # should heal rather than latch off, and the entrypoint mints a fresh + # registration token per start, so a genuinely broken image loops visibly in + # the logs instead of failing silently. + restart: unless-stopped stop_grace_period: 5m environment: URL: ${URL} GITHUB_PAT: ${GITHUB_PAT} - RUNNER_TOKEN: ${RUNNER_TOKEN} - RUNNER_NAME: ${RUNNER_NAME} + # Defaulted to empty: entrypoint.sh prefers GITHUB_PAT and only falls back + # to a static token, but an unset variable makes `docker compose` print a + # warning per service per invocation -- 16 lines of noise that bury the + # real errors underneath. + RUNNER_TOKEN: ${RUNNER_TOKEN:-} + RUNNER_NAME: ${RUNNER_NAME:-} # firmware_ci.yml targets `runs-on: labels: [fw-builder]`, so a runner # without this label is registered but never assigned any job. RUNNER_LABELS: ${RUNNER_LABELS:-fw-builder} diff --git a/entrypoint.sh b/entrypoint.sh index 653b5b0..88c35b2 100644 --- a/entrypoint.sh +++ b/entrypoint.sh @@ -81,8 +81,24 @@ if [[ -f /home/runner/export-esp.sh ]]; then fi echo "Removing any existing runner configuration..." -# Clean up previous runs (crucial for ephemeral runners) -rm -f .runner .credentials .credentials_rsaparams +# Clean up previous runs (crucial for ephemeral runners). +# +# `.runner_migrated` MUST be in this list. The runner self-updates in place, and +# a post-update runner drops that marker beside its config. `config.sh` treats +# the marker ALONE as proof the runner is already configured -- verified by +# creating only `.runner_migrated` and passing a deliberately bogus token: it +# fails with "Cannot configure the runner because it is already configured" +# without even attempting to authenticate. +# +# Because the old list stopped at `.credentials_rsaparams`, every replica that +# had auto-updated crash-looped on its next restart until `restart: +# on-failure:5` exhausted its retries, which silently took the entire fleet +# offline about ten days after it was last rebuilt. Deleting the marker is +# correct rather than merely expedient: this entrypoint always reconfigures from +# a freshly minted registration token, so there is no migrated state worth +# preserving across a restart. +rm -f .runner .credentials .credentials_rsaparams \ + .runner_migrated .credentials_migrated echo "Configuring GitHub Actions Runner as ${FULL_RUNNER_NAME}..." echo "URL: $URL" From 7adc233efa2f7be3b760a262b9090e16c5190aef Mon Sep 17 00:00:00 2001 From: Riccardo Persello Date: Sat, 8 Aug 2026 12:59:21 +0200 Subject: [PATCH 09/18] Build hbf as well as firmware: bun, webkit and a second label hbf CI runs on GitHub-hosted runners today and reinstalls its toolchain on every job. Moving it here needs two things the image lacked. Add bun and the Tauri desktop dependencies. `cargo build -p hbf-gui` links against webkit2gtk-4.1 and fails at pkg-config time without the -dev package; librsvg2 and appindicator3 are Tauri's SVG and tray-icon dependencies. bun builds the SvelteKit bundle that `tauri::generate_context!()` embeds at COMPILE time, which makes it a build dependency rather than a test-only tool, and it is pinned to the version hbf CI's `oven-sh/setup-bun` requests so lockfile resolution matches. BUN_INSTALL puts the binary on the shared PATH instead of under /root, which is mode 0700 and so invisible to the unprivileged runner user -- the same trap the uv block already documents. Both layers go AFTER espup deliberately. Docker invalidates every layer below an edited one, and rebuilding the Xtensa toolchain costs many minutes. Verified the `esp` toolchain survived the rebuild untouched. Image grows 8.86 -> 9.6 GB. Add `hbf-builder` to every replica rather than reserving a subset for it. A runner is offered a job only when its labels are a SUPERSET of the job's `runs-on`, so splitting them (1-6 fw-builder, 7-8 hbf-builder) would leave six containers ineligible for hbf work and idle whenever hbf work is all that is queued. Both labels everywhere means any replica serves either repo, and capacity is added by adding replicas. One trap worth recording: `docker compose build` must be run with NO service argument. Each service declares its own `build: .`, so compose tags a separate image per service, and `docker compose build runner-1` silently leaves the other seven on the old image -- they still start and register, so nothing looks wrong until a job needs a tool only the rebuilt image has. hbf CI cannot move here yet: `.github/actions/setup-canvas` downloads pkl-linux-amd64 while these runners are arm64, and it writes to /usr/local/bin, which the runner user cannot do. pkl is also skewed three ways (0.30.1 here, 0.31.1 in hbf CI, 0.32.1 on the dev machine). Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_018cbZiFrvjiXMRHLq1L9PDf --- Dockerfile | 29 +++++++++++++++++++++++++++++ docker-compose.yml | 10 +++++++--- 2 files changed, 36 insertions(+), 3 deletions(-) diff --git a/Dockerfile b/Dockerfile index 624058d..dd7fb67 100644 --- a/Dockerfile +++ b/Dockerfile @@ -84,6 +84,35 @@ RUN curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR=/usr/local/b UV_TOOL_BIN_DIR=/usr/local/bin uv tool install maturin && \ chmod -R a+rX /opt/uv +# ============================================================================ +# Web UI and Tauri desktop dependencies (hbf) +# ============================================================================ +# Deliberately placed AFTER the espup layer. Docker invalidates every layer +# below an edited one, and rebuilding the Xtensa toolchain costs many minutes, +# so anything added later must stay later. +# +# `cargo build -p hbf-gui` links against webkit2gtk-4.1 and fails at +# pkg-config time without the -dev package; librsvg2 and appindicator3 are +# Tauri's SVG and tray-icon dependencies. This mirrors the apt list hbf CI +# installs per job, minus what the firmware layers above already provide +# (libudev-dev, pkg-config, libssl-dev). +RUN apt-get update && \ + apt-get install -y --no-install-recommends \ + libwebkit2gtk-4.1-dev libayatana-appindicator3-dev \ + librsvg2-dev && \ + apt-get clean && rm -rf /var/lib/apt/lists/* + +# bun builds the SvelteKit bundle that `tauri::generate_context!()` embeds at +# COMPILE time, so it is a build dependency of hbf-gui rather than a test-only +# tool. Pinned to the version hbf CI's `oven-sh/setup-bun` requests so lockfile +# resolution is identical on both. BUN_INSTALL places the binary on the shared +# PATH instead of under /root, which is mode 0700 and therefore invisible to the +# unprivileged runner user -- the same trap the uv block above documents. +ENV BUN_INSTALL=/usr/local +RUN curl -fsSL https://bun.sh/install | bash -s "bun-v1.3.14" && \ + chmod a+rx /usr/local/bin/bun && \ + bun --version + # ============================================================================ # Create runner directory and download GitHub Actions Runner # ============================================================================ diff --git a/docker-compose.yml b/docker-compose.yml index 2152f78..a12ebbb 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -39,9 +39,13 @@ x-runner: &runner # real errors underneath. RUNNER_TOKEN: ${RUNNER_TOKEN:-} RUNNER_NAME: ${RUNNER_NAME:-} - # firmware_ci.yml targets `runs-on: labels: [fw-builder]`, so a runner - # without this label is registered but never assigned any job. - RUNNER_LABELS: ${RUNNER_LABELS:-fw-builder} + # A runner is offered a job only when its label set is a SUPERSET of the + # job's `runs-on`. Both labels therefore go on every replica: splitting them + # across replicas (say 1-6 fw-builder, 7-8 hbf-builder) would leave six + # containers ineligible for hbf jobs and idle whenever hbf work is all that + # is queued. firmware_ci.yml asks for [fw-builder]; hbf asks for + # [hbf-builder]; every replica can serve either. + RUNNER_LABELS: ${RUNNER_LABELS:-fw-builder,hbf-builder} # Match cargo's internal parallelism to this replica's CPU allotment. # cargo defaults to one codegen unit per *host* core, so without this each # replica would spawn ~16 threads and 8 replicas would oversubscribe the From 3942a4aa31fac8d280f28f672a3b2112d33b22ff Mon Sep 17 00:00:00 2001 From: Riccardo Persello Date: Sat, 8 Aug 2026 16:11:14 +0200 Subject: [PATCH 10/18] Install Node alongside bun hbf's `ts_export` test execs `ui/node_modules/.bin/prettier` directly from Rust. That file is a .cjs script whose shebang is `#!/usr/bin/env node`, so on this image the exec failed with status 127 and the drift check reported "bindings would drift from CI's regen" -- a misleading message for a missing interpreter. bun does not substitute for node here. `bun run lint` and `bun run check` work because `bun run` interprets the JS itself and never consults the shebang, which is why the gap stays invisible until something shells out to a .bin entry. npm comes along for `npx`, which the same test falls back to when the project-local binary is absent. GitHub-hosted runners preinstall both, so this could only surface here. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_018cbZiFrvjiXMRHLq1L9PDf --- Dockerfile | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/Dockerfile b/Dockerfile index dd7fb67..877b77f 100644 --- a/Dockerfile +++ b/Dockerfile @@ -102,6 +102,24 @@ RUN apt-get update && \ librsvg2-dev && \ apt-get clean && rm -rf /var/lib/apt/lists/* +# Node is needed even though bun is the package manager, because bun does not +# replace it as a script *interpreter*. hbf's `ts_export` test execs +# `ui/node_modules/.bin/prettier` directly from Rust; that file is a .cjs script +# whose shebang is `#!/usr/bin/env node`, so without node the exec fails with +# status 127 and the drift check reports "bindings would drift". `bun run lint` +# and `bun run check` are unaffected because `bun run` interprets the JS itself +# and never consults the shebang -- which is exactly why this gap is invisible +# until something shells out to a .bin entry. +# +# npm comes along for `npx`, which the same test falls back to when the +# project-local binary is absent. GitHub-hosted runners preinstall both, which is +# why this only surfaced on the fleet. +RUN apt-get update && \ + apt-get install -y --no-install-recommends \ + nodejs npm && \ + apt-get clean && rm -rf /var/lib/apt/lists/* && \ + node --version && npx --version + # bun builds the SvelteKit bundle that `tauri::generate_context!()` embeds at # COMPILE time, so it is a build dependency of hbf-gui rather than a test-only # tool. Pinned to the version hbf CI's `oven-sh/setup-bun` requests so lockfile From 92a81192b2390401b49ed25fb2d5fc7a37f02582 Mon Sep 17 00:00:00 2001 From: Riccardo Persello Date: Sat, 8 Aug 2026 16:43:25 +0200 Subject: [PATCH 11/18] Scale to 12 replicas at 2 CPU / 4 GB This block claimed "Memory is the binding constraint, not CPU" at 8 x 6 GB. Measured under a full load of firmware and hbf jobs, that was wrong on both counts: peak memory across all replicas was 756 MiB against the 6 GiB limit -- an 8x overshoot -- and only 4 of 8 containers were computing at all (~195% CPU each), the rest sitting near idle on network and setup. Roughly half the host's cores went unused while jobs queued. The real constraint was SLOTS. An hbf run measured 16.8 minutes of job time inside an 11.8 minute span -- an average concurrency of 1.4 -- because firmware held 7 of the 8 slots, so a pipeline that takes 2m41s on a hosted runner took 11m48s here. Hence more, smaller replicas. Total memory is unchanged at 48 GB of the ~58 GB the VM exposes. CPU is now oversubscribed 1.5:1 (12 x 2 = 24 on 16 cores), which the measured idle time justifies. Disk is the limiting factor now, not memory: each replica keeps its own target/ for both repositories on one 200 GB volume. Reclaiming 27 GB of stale build cache brought it to 43% before this change; check `docker system df` before going wider. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_018cbZiFrvjiXMRHLq1L9PDf --- docker-compose.yml | 75 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 58 insertions(+), 17 deletions(-) diff --git a/docker-compose.yml b/docker-compose.yml index a12ebbb..0bcc36a 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -17,14 +17,20 @@ # Each runner therefore gets its own registry AND sccache volume, which is only # expressible as its own service. The cost is N copies of the crate downloads. # -# `docker compose up -d --build` starts all 8. To run fewer, name them: +# `docker compose up -d --build` starts all 12. To run fewer, name them: # docker compose up -d --build runner-1 runner-2 runner-3 +# +# `docker compose build` MUST be run with no service argument. Each service +# declares its own `build: .`, so compose tags a separate image per service, and +# `docker compose build runner-1` silently leaves the others on the old image -- +# they still start and register, so nothing looks wrong until a job needs a tool +# only the rebuilt image has. x-runner: &runner build: . # `on-failure:5` used to be the policy here and it cost the fleet ten days of # downtime: a bug in entrypoint.sh's config cleanup made every replica exit 1 # on restart, the five retries were spent in seconds, and Docker then left all - # eight containers dead with no surviving process to notice. A runner fleet + # containers dead with no surviving process to notice. A runner fleet # should heal rather than latch off, and the entrypoint mints a fresh # registration token per start, so a genuinely broken image loops visibly in # the logs instead of failing silently. @@ -35,33 +41,48 @@ x-runner: &runner GITHUB_PAT: ${GITHUB_PAT} # Defaulted to empty: entrypoint.sh prefers GITHUB_PAT and only falls back # to a static token, but an unset variable makes `docker compose` print a - # warning per service per invocation -- 16 lines of noise that bury the - # real errors underneath. + # warning per service per invocation -- noise that buries the real errors + # underneath. RUNNER_TOKEN: ${RUNNER_TOKEN:-} RUNNER_NAME: ${RUNNER_NAME:-} # A runner is offered a job only when its label set is a SUPERSET of the # job's `runs-on`. Both labels therefore go on every replica: splitting them - # across replicas (say 1-6 fw-builder, 7-8 hbf-builder) would leave six - # containers ineligible for hbf jobs and idle whenever hbf work is all that - # is queued. firmware_ci.yml asks for [fw-builder]; hbf asks for - # [hbf-builder]; every replica can serve either. + # across replicas would leave some containers ineligible for hbf jobs and + # idle whenever hbf work is all that is queued. firmware_ci.yml asks for + # [fw-builder]; hbf asks for [hbf-builder]; every replica can serve either. RUNNER_LABELS: ${RUNNER_LABELS:-fw-builder,hbf-builder} # Match cargo's internal parallelism to this replica's CPU allotment. # cargo defaults to one codegen unit per *host* core, so without this each - # replica would spawn ~16 threads and 8 replicas would oversubscribe the - # machine 8-fold. The cpus limit below only throttles the result; capping - # the thread count is what actually avoids the thrashing. + # replica would spawn ~16 threads and every replica would oversubscribe the + # machine. The cpus limit below only throttles the result; capping the thread + # count is what actually avoids the thrashing. CARGO_BUILD_JOBS: ${RUNNER_CPUS:-2} deploy: resources: limits: - # Sized for a 16-core / 64 GB host. Memory is the binding constraint, - # not CPU: 8 x 6g = 48 GB of the ~58 GB the VM exposes, leaving - # headroom for the host. CPUs are deliberately oversubscribed 1:1 - # (8 x 2 = 16) because jobs spend much of their wall time on network - # and link steps rather than pegged compute. + # 12 replicas at 2 CPU / 4 GB on a 16-core / 64 GB host. + # + # This block previously read "Memory is the binding constraint, not CPU" + # at 8 x 6 GB. Measured under a full load of firmware and hbf jobs, that + # was wrong on both counts: peak usage across all replicas was 756 MiB + # against the 6 GiB limit -- an 8x overshoot -- and only 4 of 8 + # containers were computing at all (~195% CPU each), the rest sitting + # near idle on network and setup. Roughly half the host's cores were + # unused while jobs queued. + # + # The real constraint was SLOTS. An hbf run measured 16.8 minutes of job + # time inside an 11.8 minute span -- an average concurrency of 1.4 -- + # because firmware held 7 of the 8 slots. So: more, smaller replicas. + # + # Total memory is unchanged at 48 GB of the ~58 GB the VM exposes. CPU is + # deliberately oversubscribed 1.5:1 (12 x 2 = 24 on 16 cores), which the + # observed idle time justifies. + # + # DISK is now the limiting factor, not memory: each replica keeps its own + # target/ for both repositories on one 200 GB volume, which was 47% full + # at 8 replicas. Watch `docker system df` before going wider. cpus: ${RUNNER_CPUS:-2} - memory: ${RUNNER_MEMORY:-6g} + memory: ${RUNNER_MEMORY:-4g} services: runner-1: @@ -88,6 +109,18 @@ services: runner-8: <<: *runner volumes: [cargo-registry-8:/home/runner/.cargo/registry, sccache-8:/home/runner/.cache/sccache] + runner-9: + <<: *runner + volumes: [cargo-registry-9:/home/runner/.cargo/registry, sccache-9:/home/runner/.cache/sccache] + runner-10: + <<: *runner + volumes: [cargo-registry-10:/home/runner/.cargo/registry, sccache-10:/home/runner/.cache/sccache] + runner-11: + <<: *runner + volumes: [cargo-registry-11:/home/runner/.cargo/registry, sccache-11:/home/runner/.cache/sccache] + runner-12: + <<: *runner + volumes: [cargo-registry-12:/home/runner/.cargo/registry, sccache-12:/home/runner/.cache/sccache] volumes: cargo-registry-1: @@ -98,6 +131,10 @@ volumes: cargo-registry-6: cargo-registry-7: cargo-registry-8: + cargo-registry-9: + cargo-registry-10: + cargo-registry-11: + cargo-registry-12: sccache-1: sccache-2: sccache-3: @@ -106,3 +143,7 @@ volumes: sccache-6: sccache-7: sccache-8: + sccache-9: + sccache-10: + sccache-11: + sccache-12: From 22e87e6009a1e094b5f37e03c139696c4ba433e3 Mon Sep 17 00:00:00 2001 From: Riccardo Persello Date: Sun, 9 Aug 2026 18:50:01 +0200 Subject: [PATCH 12/18] perf(fleet): drop CI incremental state, document the real disk profile The host volume hit 294 MB free of 926 GB and took firmware CI down with it. Disk exhaustion does not present as "out of disk" -- it surfaces as `collect2: ld terminated with signal 7 [Bus error]`, the linker dying mid-write, which reads like an LLVM bug and is not one. hbf's own ci.yml documents the same symptom on hosted runners. Measured rather than assumed: of ~200 GB of Docker, 170 GB is the twelve container writable layers, and 85% of that is one directory per replica -- hbf's target/ at 11-13 GB, against firmware's already-lean 0.9 GB. The per-replica cargo registry and sccache volumes that this file goes to such lengths to keep separate are only ~29 GB combined, so the duplication this file warns about is not what filled the disk, and sharing those would not have fixed it. target/ cannot be shared at all: cargo takes an exclusive lock per target directory, so one shared dir would serialise all twelve replicas and destroy the parallelism that is the point of the fleet. The lever is a smaller target/, not fewer of them. CARGO_INCREMENTAL=0 is the actual reclaim, at 2.8 GB per replica. That state is pure waste in CI, where every job is a different commit and nothing reuses the dep-graph fingerprints; the same directory reaches 14 GB in a long-lived developer checkout. CARGO_PROFILE_DEV_DEBUG is a floor rather than a saving. hbf already set it in 3a6e615 and the measured dirs are already reduced -- objdump shows .debug_loc at 0 bytes with .debug_line dominant. It is set here so the property holds for every job regardless of per-workflow config, and because firmware_ci.yml covers only its release profile. Both live here rather than in either workflow so they apply fleet-wide without changing hosted runners or a developer's laptop, and neither repo sets them in workflow env, so the container's value is what a job sees. These are prospective: existing target dirs keep their bloat until rebuilt. Recreating a container is what clears _work, since it lives on the writable layer while the registry and sccache volumes survive. Also records that sccache is dead -- absent from the image, never set as RUSTC_WRAPPER, last written 2026-07-28 -- because that is precisely why wiping a target/ dir today is a cold rebuild instead of a cheap one. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01TUxoBmSQ9ZNqe1A2FusEsE --- docker-compose.yml | 67 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 65 insertions(+), 2 deletions(-) diff --git a/docker-compose.yml b/docker-compose.yml index 0bcc36a..6c67902 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -57,6 +57,59 @@ x-runner: &runner # machine. The cpus limit below only throttles the result; capping the thread # count is what actually avoids the thrashing. CARGO_BUILD_JOBS: ${RUNNER_CPUS:-2} + # Disk, not memory or CPU, is what this fleet runs out of. Measured + # 2026-08-09 with the host volume down to 294 MB free: the 12 container + # writable layers held 170 GB, and 85% of that was ONE directory per + # replica -- hbf's target/ at 11-13 GB (firmware's is 0.9 GB). The + # per-replica cargo registry and sccache volumes that this file goes to + # such lengths to keep separate are only ~29 GB combined, so they are not + # the problem and sharing them would not fix it. target/ cannot be shared + # at all: cargo takes an exclusive lock per target directory, so one + # shared dir would serialise all 12 replicas and destroy the parallelism + # that is the entire point of the fleet. The lever is to make each target/ + # smaller, not fewer of them. + # + # Both variables are set here rather than in either repo's workflow so + # they apply to every job on the fleet without changing what hosted + # runners or a developer's laptop do. Neither repo sets these in its + # workflow env, so the container's value is what the job sees. + # + # Incremental state is pure waste in CI -- each job is a different commit + # and nothing downstream reuses the dep-graph fingerprints. It was + # measured at 2.8 GB per replica in hbf's target/ (14 GB in a long-lived + # developer checkout, which is what this avoids becoming). + CARGO_INCREMENTAL: "0" + # A floor, NOT a saving -- do not expect this one to reclaim anything. + # hbf's ci.yml already set it on 2026-08-08 (3a6e615), and the 11-13 GB + # target dirs measured here are ALREADY the reduced size: objdump on the + # largest test executable shows .debug_loc at 0 bytes with .debug_line the + # dominant section, which is the line-tables-only signature. It is set here + # so the property holds for every job on the fleet regardless of what any + # one workflow remembers to configure, and because firmware_ci.yml sets + # only CARGO_PROFILE_RELEASE_DEBUG, leaving its dev profile uncovered. + # + # Keeps file and line numbers -- what RUST_BACKTRACE=1 in both CI configs + # actually consumes -- and drops the variable and type records nothing here + # reads. Verified safe for hbf: no crate links gimli, object or addr2line, + # and its current_exe() calls locate sibling binaries rather than parse + # them. probe-rs reads DWARF from *target firmware* ELFs, unaffected by how + # hbf itself is compiled. + CARGO_PROFILE_DEV_DEBUG: line-tables-only + # NOTE: sccache is currently DEAD here, which is why wiping a target/ dir + # is a genuinely cold rebuild rather than a cheap one. The binary is not in + # the image at all (`sccache --version` -> not found), no RUSTC_WRAPPER is + # ever set, and the newest entry in the 1.1 GB-per-replica cache volumes + # dates to 2026-07-28 -- an abandoned experiment holding ~13 GB fleet-wide. + # + # Wiring it up properly is the real fix for the 12x target/ duplication: + # sccache keys per-crate rustc output on content in a store OUTSIDE + # target/, which is what makes target/ disposable. Note CARGO_INCREMENTAL=0 + # above is a PREREQUISITE, not a conflict -- sccache cannot cache + # incrementally compiled units and silently bypasses them. Unlike the cargo + # registry (see the header comment for why sharing that corrupts builds), a + # single sccache CAN be shared safely across all 12 replicas, but only via a + # server backend (redis/S3/webdav); a shared local directory is unsafe + # because each sccache server process keeps its own in-memory LRU index. deploy: resources: limits: @@ -79,8 +132,18 @@ x-runner: &runner # observed idle time justifies. # # DISK is now the limiting factor, not memory: each replica keeps its own - # target/ for both repositories on one 200 GB volume, which was 47% full - # at 8 replicas. Watch `docker system df` before going wider. + # target/ for both repositories. At 12 replicas this reached 170 GB of + # container layers and took the host volume down to 294 MB free, which + # does not fail as "out of disk" -- it surfaces as + # `collect2: ld terminated with signal 7 [Bus error]`, the linker dying + # mid-write. hbf's own ci.yml documents the same symptom on hosted + # runners. Treat any inexplicable linker or codegen failure across + # several replicas as a disk check first. + # + # See CARGO_INCREMENTAL / CARGO_PROFILE_DEV_DEBUG above for the fix, and + # note it is PROSPECTIVE: existing target/ dirs keep their incremental + # state and fat debuginfo until rebuilt, so landing those variables + # reclaims nothing on its own. Run `docker system df` before going wider. cpus: ${RUNNER_CPUS:-2} memory: ${RUNNER_MEMORY:-4g} From 2adb60652a681c850c8a6089385bb72c05c63128 Mon Sep 17 00:00:00 2001 From: Riccardo Persello Date: Sun, 9 Aug 2026 18:54:55 +0200 Subject: [PATCH 13/18] fix(fleet): apply the cargo knobs on restart, not only on recreate The previous commit set CARGO_INCREMENTAL in docker-compose.yml, which cannot take effect without destroying the thing it was meant to save. `docker compose` bakes a container's environment at CREATION, so an edited compose file reaches a job only after `up -d` recreates the container -- and `_work`, holding the 11-13 GB warm `target/` dir per replica, lives on the writable layer and goes with it. Since sccache is dead here (absent from the image, never set as RUSTC_WRAPPER, last written 2026-07-28), that recreate is a genuinely cold rebuild across all twelve replicas rather than a cheap one. Exporting from entrypoint.sh instead means a plain `docker restart` applies a change while preserving every warm `deps/` dir. The runner inherits the entrypoint's environment and hands it to each job step, so the export reaches the compiler. /actions-runner/.env would have been the obvious mechanism and does not work: it is read only by the systemd unit `svc.sh` generates, whereas this entrypoint execs ./run.sh directly and run.sh contains no reference to it -- checked rather than assumed. The stock file is empty and `env.sh` only writes it for that service path. The defaults therefore live in one place, entrypoint.sh, with docker-compose.yml documenting the override point rather than repeating the literals. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01TUxoBmSQ9ZNqe1A2FusEsE --- docker-compose.yml | 35 +++++++++++------------------------ entrypoint.sh | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+), 24 deletions(-) diff --git a/docker-compose.yml b/docker-compose.yml index 6c67902..2f8bbec 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -69,32 +69,19 @@ x-runner: &runner # that is the entire point of the fleet. The lever is to make each target/ # smaller, not fewer of them. # - # Both variables are set here rather than in either repo's workflow so - # they apply to every job on the fleet without changing what hosted - # runners or a developer's laptop do. Neither repo sets these in its - # workflow env, so the container's value is what the job sees. + # The cargo knobs that act on this -- CARGO_INCREMENTAL and + # CARGO_PROFILE_DEV_DEBUG -- are deliberately NOT set here, and + # entrypoint.sh owns their defaults instead. A value set in this file only + # reaches a job once `up -d` RECREATES the container, which destroys _work + # and with it the warm target/ dir on the writable layer -- the very thing + # being economised. Exported from the entrypoint, a plain `docker restart` + # applies them and keeps the caches. Add either name here to override, and + # see entrypoint.sh for why /actions-runner/.env cannot serve this purpose. # - # Incremental state is pure waste in CI -- each job is a different commit - # and nothing downstream reuses the dep-graph fingerprints. It was - # measured at 2.8 GB per replica in hbf's target/ (14 GB in a long-lived - # developer checkout, which is what this avoids becoming). - CARGO_INCREMENTAL: "0" - # A floor, NOT a saving -- do not expect this one to reclaim anything. - # hbf's ci.yml already set it on 2026-08-08 (3a6e615), and the 11-13 GB - # target dirs measured here are ALREADY the reduced size: objdump on the - # largest test executable shows .debug_loc at 0 bytes with .debug_line the - # dominant section, which is the line-tables-only signature. It is set here - # so the property holds for every job on the fleet regardless of what any - # one workflow remembers to configure, and because firmware_ci.yml sets - # only CARGO_PROFILE_RELEASE_DEBUG, leaving its dev profile uncovered. + # Setting them on the fleet rather than in either repo's workflow keeps + # hosted runners and developer laptops unaffected. Neither repo sets + # CARGO_INCREMENTAL, so the fleet's value is what a job sees. # - # Keeps file and line numbers -- what RUST_BACKTRACE=1 in both CI configs - # actually consumes -- and drops the variable and type records nothing here - # reads. Verified safe for hbf: no crate links gimli, object or addr2line, - # and its current_exe() calls locate sibling binaries rather than parse - # them. probe-rs reads DWARF from *target firmware* ELFs, unaffected by how - # hbf itself is compiled. - CARGO_PROFILE_DEV_DEBUG: line-tables-only # NOTE: sccache is currently DEAD here, which is why wiping a target/ dir # is a genuinely cold rebuild rather than a cheap one. The binary is not in # the image at all (`sccache --version` -> not found), no RUSTC_WRAPPER is diff --git a/entrypoint.sh b/entrypoint.sh index 88c35b2..d9a861e 100644 --- a/entrypoint.sh +++ b/entrypoint.sh @@ -151,6 +151,43 @@ handle_shutdown() { } trap handle_shutdown SIGTERM SIGINT +# Cargo knobs applied to every job this replica runs. +# +# Exporting here is what makes these changeable by a plain `docker restart`. +# `docker compose` bakes a container's environment at CREATION time, so setting +# them only in docker-compose.yml means they reach a job solely after +# `up -d` -- which RECREATES the container, destroying `_work` along with the +# 11-13 GB warm `target/` dir that lives on the writable layer. Restart keeps +# it. The runner inherits this process's environment and hands it to each job +# step, so an export reaches the compiler. +# +# Do NOT move these into /actions-runner/.env. That file is read only by the +# systemd unit `svc.sh` generates; this entrypoint execs ./run.sh directly and +# run.sh contains no reference to it -- verified, not assumed. The stock .env is +# empty here and `env.sh` merely writes it for that service path. +# +# The defaults live here rather than in docker-compose.yml so that one file owns +# them; compose or `docker run -e` can still override either value. + +# Incremental state is pure waste in CI: every job builds a different commit and +# nothing downstream reuses the dep-graph fingerprints. Measured at 2.8 GB per +# replica in hbf's target/ (and 14 GB in a long-lived developer checkout, which +# is what this keeps the fleet from becoming). It is also a prerequisite for +# sccache rather than a rival to it -- sccache cannot cache incrementally +# compiled units and silently bypasses them. +export CARGO_INCREMENTAL="${CARGO_INCREMENTAL:-0}" + +# A floor, NOT a saving -- do not expect this to reclaim anything. hbf's ci.yml +# already sets it workflow-wide as of 3a6e615, and the target dirs measured on +# this fleet are already the reduced size: objdump on the largest test +# executable shows .debug_loc at 0 bytes with .debug_line the dominant section, +# the line-tables-only signature. Set here so the property holds for every job +# whatever an individual workflow remembers to configure; firmware_ci.yml sets +# only CARGO_PROFILE_RELEASE_DEBUG and leaves its dev profile uncovered. +export CARGO_PROFILE_DEV_DEBUG="${CARGO_PROFILE_DEV_DEBUG:-line-tables-only}" + +echo "Cargo: CARGO_INCREMENTAL=${CARGO_INCREMENTAL} CARGO_PROFILE_DEV_DEBUG=${CARGO_PROFILE_DEV_DEBUG}" + echo "Starting runner..." gosu runner ./run.sh & RUNNER_PID=$! From 9854dbe8fd33d43ba268ef14f7e0d9a1afbd0ec7 Mon Sep 17 00:00:00 2001 From: Riccardo Persello Date: Sun, 9 Aug 2026 19:53:58 +0200 Subject: [PATCH 14/18] fix(fleet): correct the rationale for CARGO_INCREMENTAL=0 Both previous commits justified it as "pure waste in CI, since every job builds a different commit and nothing reuses the dep-graph fingerprints". That is wrong for this fleet: hbf's ci.yml uses `clean: false` precisely to keep `target/` warm across jobs, so successive jobs on one replica can hit an incremental cache. It is a trade, not a free win, and the comment now says so. The trade still favours dropping it, for reasons that are bounded rather than absolute. Incremental covers only hbf's dozen workspace crates -- registry dependencies, the entire 8.4 GB bulk of debug/deps, compile non-incrementally either way. A hit further requires the same replica to rebuild a nearly identical commit, and jobs land on whichever of the 12 is free with no branch affinity. Where nothing changed, ordinary fingerprinting already skips the crate. And a hit is not free: incremental raises the codegen-unit count and gives some of the saving back at link time. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01TUxoBmSQ9ZNqe1A2FusEsE --- entrypoint.sh | 32 ++++++++++++++++++++++++++------ 1 file changed, 26 insertions(+), 6 deletions(-) diff --git a/entrypoint.sh b/entrypoint.sh index d9a861e..de0d00d 100644 --- a/entrypoint.sh +++ b/entrypoint.sh @@ -169,12 +169,32 @@ trap handle_shutdown SIGTERM SIGINT # The defaults live here rather than in docker-compose.yml so that one file owns # them; compose or `docker run -e` can still override either value. -# Incremental state is pure waste in CI: every job builds a different commit and -# nothing downstream reuses the dep-graph fingerprints. Measured at 2.8 GB per -# replica in hbf's target/ (and 14 GB in a long-lived developer checkout, which -# is what this keeps the fleet from becoming). It is also a prerequisite for -# sccache rather than a rival to it -- sccache cannot cache incrementally -# compiled units and silently bypasses them. +# Worth 2.8 GB per replica (14 GB in a long-lived developer checkout, which is +# what this keeps the fleet from becoming). +# +# This is a trade, NOT free: an earlier version of this comment called +# incremental state "pure waste in CI, since every job is a different commit", +# which is wrong here. hbf's ci.yml deliberately uses `clean: false` to keep +# `target/` warm across jobs, so successive jobs on one replica genuinely can +# hit an incremental cache. +# +# The exposure is bounded and judged worth the disk: +# - It only ever covers hbf's own dozen workspace crates. Registry +# dependencies -- which are the whole 8.4 GB bulk of debug/deps -- are +# compiled non-incrementally regardless of this setting. +# - A hit needs the SAME replica to rebuild a NEARLY IDENTICAL commit. Jobs +# go to whichever of the 12 replicas is free, with no branch affinity, so +# that is luck rather than design. +# - Where nothing changed at all, cargo's ordinary fingerprinting skips the +# crate outright and incremental adds nothing. +# - It is not free even when it hits: incremental raises the codegen-unit +# count, which costs some link time back. +# +# It is also a prerequisite for sccache rather than a rival to it -- sccache +# cannot cache incrementally compiled units and silently bypasses them. sccache +# would hit across every crate, replica and commit rather than only the +# same-replica-similar-commit case, so trading incremental for it is a clear win +# whenever someone wires it up. export CARGO_INCREMENTAL="${CARGO_INCREMENTAL:-0}" # A floor, NOT a saving -- do not expect this to reclaim anything. hbf's ci.yml From 8aebf38218786e02ce467f81149737621312024e Mon Sep 17 00:00:00 2001 From: Riccardo Persello Date: Sun, 9 Aug 2026 22:28:57 +0200 Subject: [PATCH 15/18] feat(fleet): wire a shared sccache so target/ becomes disposable Twelve replicas each keep their own `target/`, hbf's reaches 11-13 GB, and that filled a 926 GB volume to 294 MB free. sccache does not shrink those dirs -- rlibs and test executables still land at full size -- but it makes DELETING them cheap, which is what allows them to be treated as disposable instead of hoarded. Disk stays bounded by sccache plus a recurring sweep; neither does it alone. Shared via MinIO rather than per-replica. Twelve private caches would each warm from scratch, leaving the first build on every replica cold, which is most of what a cache exists to prevent. Pointed at one bucket, whichever replica compiles a crate first serves the other eleven. This does not contradict the header comment forbidding a shared cache DIRECTORY: that hazard is sccache's per-process in-memory LRU index, which a server backend does not have. MinIO over Redis because a cache this size belongs on disk; Redis would hold it in RAM, competing with the 48 of ~58 GB already committed to the replicas. Verified end to end rather than assumed. A cold build recorded 11 misses against `Cache location s3`; after `cargo clean`, the rebuild took hits 1 -> 13 with misses unchanged at 11, so the entire rebuild came from the bucket. That is the property the wipe depends on. Two failure modes found while verifying, both silent, both now handled: - The server takes its cache config from whichever process first starts it. Spawned implicitly by cargo's first wrapper call it came up on LOCAL DISK, with every request invisible to the bucket. It is now started explicitly from here, and SCCACHE_IDLE_TIMEOUT=0 stops it exiting after the default 600s -- a runner idles far longer than that between jobs, which would have made the bad path the normal one. - `sccache --show-stats` alone is not a valid probe: with no server running it reports `Local disk` from client-side defaults without starting one. The startup check therefore routes one throwaway compile through the wrapper first, so the backend it reports is real. The cache is fail-safe: entrypoint.sh enables the wrapper only if the bucket answers, and compose orders the runners after MinIO without gating on its health, so a cache outage means slower CI rather than no CI. The bucket is bounded, which sccache will not do itself -- it has an LRU for a local directory but never deletes from S3. sccache-s3-init sets a 14-day expiry rule plus a 30 GiB quota backstop; expiry is primary because a hard quota makes writes fail once reached, while expiry just forgets what has not been useful lately. The twelve per-replica sccache volumes are removed, since an S3-backed sccache keeps no local cache directory. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01TUxoBmSQ9ZNqe1A2FusEsE --- Dockerfile | 38 ++++++++++++ docker-compose.yml | 141 ++++++++++++++++++++++++++++++++------------- entrypoint.sh | 116 ++++++++++++++++++++++++++++++++++++- 3 files changed, 254 insertions(+), 41 deletions(-) diff --git a/Dockerfile b/Dockerfile index 877b77f..ff83c37 100644 --- a/Dockerfile +++ b/Dockerfile @@ -159,6 +159,44 @@ RUN mkdir -p /root/.ssh && \ ssh-keyscan github.com >> /root/.ssh/known_hosts && \ chmod 644 /root/.ssh/known_hosts +# ============================================================================ +# sccache -- shared compilation cache +# ============================================================================ +# Placement is deliberate on both sides. It sits AFTER the espup and bun layers, +# so adding it never invalidates the multi-GB Xtensa toolchain, and BEFORE +# `COPY entrypoint.sh`, because that COPY invalidates every layer after it +# whenever the entrypoint changes -- re-downloading sccache on each entrypoint +# tweak would be pure waste. +# +# Why it exists: every replica keeps its own `target/`, and hbf's reaches +# 11-13 GB, so twelve of them took a 926 GB volume down to 294 MB free on +# 2026-08-09, at which point CI began failing with +# `collect2: ld terminated with signal 7 [Bus error]` -- disk exhaustion wearing +# a linker bug's clothing. +# +# sccache does NOT shrink `target/`. It caches rustc invocations in a store +# outside it, so the rlibs and test executables still land there at full size. +# What it buys is that DELETING a target dir becomes cheap, which is what makes +# those dirs disposable rather than something to hoard. Bounding disk therefore +# needs sccache AND a recurring sweep; sccache on its own does not do it. +# +# The musl build is static, so it is indifferent to the glibc version of whatever +# base image this is rebuilt on. +ARG SCCACHE_VERSION=v0.17.0 +RUN ARCH="${TARGETARCH:-$(dpkg --print-architecture)}" && \ + case "$ARCH" in \ + amd64) SCCACHE_ARCH=x86_64 ;; \ + arm64) SCCACHE_ARCH=aarch64 ;; \ + *) echo "ERROR: unsupported architecture for sccache: $ARCH" >&2; exit 1 ;; \ + esac && \ + SCCACHE_PKG="sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl" && \ + curl -fsSL -o /tmp/sccache.tar.gz \ + "https://github.com/mozilla/sccache/releases/download/${SCCACHE_VERSION}/${SCCACHE_PKG}.tar.gz" && \ + tar -xzf /tmp/sccache.tar.gz -C /tmp && \ + install -m 0755 "/tmp/${SCCACHE_PKG}/sccache" /usr/local/bin/sccache && \ + rm -rf /tmp/sccache.tar.gz "/tmp/${SCCACHE_PKG}" && \ + sccache --version + # Copy entrypoint script COPY entrypoint.sh /entrypoint.sh RUN chmod +x /entrypoint.sh diff --git a/docker-compose.yml b/docker-compose.yml index 2f8bbec..93b5304 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -3,7 +3,9 @@ # is safe to share between concurrently building runners: # # - sccache keeps its LRU index in memory per server process, so several -# containers on one cache directory evict against each other. +# containers on one cache DIRECTORY evict against each other. This is why +# the shared sccache below is reached over S3 instead: a server backend has +# no such per-process index, so sharing it is safe where a directory is not. # - The cargo registry was shared here initially and caused real CI failures: # error: could not compile `crc32fast` (lib) # Caused by: could not execute process `.../bin/rustc --crate-name @@ -14,9 +16,15 @@ # a working directory that no longer exists. Cargo's package-cache lock does # not protect a build for its whole duration across separate containers. # -# Each runner therefore gets its own registry AND sccache volume, which is only +# Each runner therefore gets its own cargo registry volume, which is only # expressible as its own service. The cost is N copies of the crate downloads. # +# sccache is the exception and is now SHARED, via the `sccache-s3` service below. +# Twelve private caches would each have to warm from scratch, so the first build +# on every replica stayed cold -- most of what a cache exists to prevent. The +# per-replica sccache volumes that used to be mounted here are gone: with an S3 +# backend sccache keeps no local cache directory. +# # `docker compose up -d --build` starts all 12. To run fewer, name them: # docker compose up -d --build runner-1 runner-2 runner-3 # @@ -36,6 +44,12 @@ x-runner: &runner # the logs instead of failing silently. restart: unless-stopped stop_grace_period: 5m + # Ordering only, deliberately NOT `condition: service_healthy`. Gating twelve + # runners on the cache's health would turn an optimisation into a single point + # of failure for the whole fleet -- a sick MinIO would mean no CI at all rather + # than slow CI. entrypoint.sh probes the bucket itself and degrades to + # wrapper-less builds if it cannot be reached. + depends_on: [sccache-s3] environment: URL: ${URL} GITHUB_PAT: ${GITHUB_PAT} @@ -82,21 +96,21 @@ x-runner: &runner # hosted runners and developer laptops unaffected. Neither repo sets # CARGO_INCREMENTAL, so the fleet's value is what a job sees. # - # NOTE: sccache is currently DEAD here, which is why wiping a target/ dir - # is a genuinely cold rebuild rather than a cheap one. The binary is not in - # the image at all (`sccache --version` -> not found), no RUSTC_WRAPPER is - # ever set, and the newest entry in the 1.1 GB-per-replica cache volumes - # dates to 2026-07-28 -- an abandoned experiment holding ~13 GB fleet-wide. + # sccache was DEAD here until 2026-08-09 -- absent from the image, no + # RUSTC_WRAPPER ever set, and 1.1 GB per replica of cache last written + # 2026-07-28. That is why wiping a target/ dir used to mean a genuinely cold + # rebuild. It is now installed in the image and wired to the shared bucket by + # entrypoint.sh, which enables the wrapper only if that bucket answers, so a + # cache outage degrades build speed instead of failing CI. + # + # CARGO_INCREMENTAL=0 is a PREREQUISITE for it, not a rival: sccache cannot + # cache incrementally compiled units and silently bypasses them. # - # Wiring it up properly is the real fix for the 12x target/ duplication: - # sccache keys per-crate rustc output on content in a store OUTSIDE - # target/, which is what makes target/ disposable. Note CARGO_INCREMENTAL=0 - # above is a PREREQUISITE, not a conflict -- sccache cannot cache - # incrementally compiled units and silently bypasses them. Unlike the cargo - # registry (see the header comment for why sharing that corrupts builds), a - # single sccache CAN be shared safely across all 12 replicas, but only via a - # server backend (redis/S3/webdav); a shared local directory is unsafe - # because each sccache server process keeps its own in-memory LRU index. + # What sccache does NOT do is shrink target/ -- the rlibs and test + # executables still land there at full size, so a warm fleet returns to + # roughly 140 GB. It makes deleting those dirs cheap, which is what makes a + # recurring sweep affordable. Disk stays bounded by sccache AND the sweep + # together; neither suffices alone. deploy: resources: limits: @@ -135,44 +149,103 @@ x-runner: &runner memory: ${RUNNER_MEMORY:-4g} services: + # The one cache all 12 replicas share. See the header comment for why this is + # S3 rather than a shared directory, and why the cargo registry deliberately is + # NOT shared the same way. + # + # MinIO is used in preference to Redis because a compilation cache of this size + # belongs on disk: Redis would hold the whole thing in RAM, and 48 of the VM's + # ~58 GB is already committed to the replicas, so it would compete with the + # builds it is meant to accelerate. + # + # Not published to the host. Only the runners need it, and the fleet's compose + # network is enough -- exposing an unauthenticated-by-default object store on a + # laptop's interfaces would be a poor trade for a build cache. + sccache-s3: + image: minio/minio:latest + restart: unless-stopped + command: server /data + environment: + MINIO_ROOT_USER: ${SCCACHE_ACCESS_KEY:-sccache} + MINIO_ROOT_PASSWORD: ${SCCACHE_SECRET_KEY:-sccache-secret} + volumes: [sccache-s3-data:/data] + healthcheck: + test: ["CMD", "mc", "ready", "local"] + interval: 10s + timeout: 5s + retries: 5 + start_period: 20s + + # Creates the bucket and, more importantly, BOUNDS it. sccache has a built-in + # LRU for a local cache directory (SCCACHE_CACHE_SIZE) but none whatsoever for + # an S3 backend -- it never deletes anything it uploads. Left alone this bucket + # would grow without limit and become the disk problem it was added to solve. + # + # Expiry by age rather than a hard quota is the primary bound: a hard quota + # makes writes start failing once reached, which sccache reports as errors on + # every miss, whereas expiry keeps the working set and simply forgets what has + # not been useful lately. 14 days comfortably covers a fortnight of branches + # while capping the bucket at roughly one fortnight of unique compilations. + # The quota is a generous backstop against a pathological week. + # + # Runs to completion and exits; `restart: "no"` keeps it from looping. + sccache-s3-init: + image: minio/mc:latest + depends_on: + sccache-s3: + condition: service_healthy + restart: "no" + entrypoint: > + /bin/sh -c " + mc alias set fleet http://sccache-s3:9000 + '${SCCACHE_ACCESS_KEY:-sccache}' '${SCCACHE_SECRET_KEY:-sccache-secret}' && + mc mb --ignore-existing fleet/sccache && + mc ilm rule add --expire-days 14 fleet/sccache 2>/dev/null || true && + mc quota set fleet/sccache --size ${SCCACHE_MAX_SIZE:-30GiB} 2>/dev/null || true && + echo 'sccache bucket ready:' && mc du fleet/sccache + " + runner-1: <<: *runner - volumes: [cargo-registry-1:/home/runner/.cargo/registry, sccache-1:/home/runner/.cache/sccache] + volumes: [cargo-registry-1:/home/runner/.cargo/registry] runner-2: <<: *runner - volumes: [cargo-registry-2:/home/runner/.cargo/registry, sccache-2:/home/runner/.cache/sccache] + volumes: [cargo-registry-2:/home/runner/.cargo/registry] runner-3: <<: *runner - volumes: [cargo-registry-3:/home/runner/.cargo/registry, sccache-3:/home/runner/.cache/sccache] + volumes: [cargo-registry-3:/home/runner/.cargo/registry] runner-4: <<: *runner - volumes: [cargo-registry-4:/home/runner/.cargo/registry, sccache-4:/home/runner/.cache/sccache] + volumes: [cargo-registry-4:/home/runner/.cargo/registry] runner-5: <<: *runner - volumes: [cargo-registry-5:/home/runner/.cargo/registry, sccache-5:/home/runner/.cache/sccache] + volumes: [cargo-registry-5:/home/runner/.cargo/registry] runner-6: <<: *runner - volumes: [cargo-registry-6:/home/runner/.cargo/registry, sccache-6:/home/runner/.cache/sccache] + volumes: [cargo-registry-6:/home/runner/.cargo/registry] runner-7: <<: *runner - volumes: [cargo-registry-7:/home/runner/.cargo/registry, sccache-7:/home/runner/.cache/sccache] + volumes: [cargo-registry-7:/home/runner/.cargo/registry] runner-8: <<: *runner - volumes: [cargo-registry-8:/home/runner/.cargo/registry, sccache-8:/home/runner/.cache/sccache] + volumes: [cargo-registry-8:/home/runner/.cargo/registry] runner-9: <<: *runner - volumes: [cargo-registry-9:/home/runner/.cargo/registry, sccache-9:/home/runner/.cache/sccache] + volumes: [cargo-registry-9:/home/runner/.cargo/registry] runner-10: <<: *runner - volumes: [cargo-registry-10:/home/runner/.cargo/registry, sccache-10:/home/runner/.cache/sccache] + volumes: [cargo-registry-10:/home/runner/.cargo/registry] runner-11: <<: *runner - volumes: [cargo-registry-11:/home/runner/.cargo/registry, sccache-11:/home/runner/.cache/sccache] + volumes: [cargo-registry-11:/home/runner/.cargo/registry] runner-12: <<: *runner - volumes: [cargo-registry-12:/home/runner/.cargo/registry, sccache-12:/home/runner/.cache/sccache] + volumes: [cargo-registry-12:/home/runner/.cargo/registry] volumes: + # The single shared compilation cache, bounded by the ilm rule and quota that + # sccache-s3-init applies. Replaces the twelve per-replica sccache volumes. + sccache-s3-data: cargo-registry-1: cargo-registry-2: cargo-registry-3: @@ -185,15 +258,3 @@ volumes: cargo-registry-10: cargo-registry-11: cargo-registry-12: - sccache-1: - sccache-2: - sccache-3: - sccache-4: - sccache-5: - sccache-6: - sccache-7: - sccache-8: - sccache-9: - sccache-10: - sccache-11: - sccache-12: diff --git a/entrypoint.sh b/entrypoint.sh index de0d00d..66986d5 100644 --- a/entrypoint.sh +++ b/entrypoint.sh @@ -206,7 +206,121 @@ export CARGO_INCREMENTAL="${CARGO_INCREMENTAL:-0}" # only CARGO_PROFILE_RELEASE_DEBUG and leaves its dev profile uncovered. export CARGO_PROFILE_DEV_DEBUG="${CARGO_PROFILE_DEV_DEBUG:-line-tables-only}" -echo "Cargo: CARGO_INCREMENTAL=${CARGO_INCREMENTAL} CARGO_PROFILE_DEV_DEBUG=${CARGO_PROFILE_DEV_DEBUG}" +# sccache, backed by the fleet's shared S3 (MinIO) bucket. +# +# Shared rather than per-replica on purpose. Twelve private caches would each +# have to warm from scratch, so the first build on every replica stays cold -- +# which is most of what a cache is supposed to prevent. Pointed at one bucket, +# whichever replica compiles a crate first serves the other eleven. Note the +# per-replica *local* sccache volumes this fleet used to mount are unnecessary +# in this mode and have been removed from docker-compose.yml: with an S3 backend +# sccache does not use a local cache directory. +# +# A shared local DIRECTORY would not be safe here -- each sccache server process +# keeps its own in-memory LRU index, so several containers writing one directory +# corrupt each other's accounting. A server backend has no such problem, which +# is the distinction the header comment in docker-compose.yml draws for the cargo +# registry as well. +SCCACHE_BUCKET="${SCCACHE_BUCKET:-sccache}" +SCCACHE_ENDPOINT="${SCCACHE_ENDPOINT:-http://sccache-s3:9000}" +SCCACHE_REGION="${SCCACHE_REGION:-us-east-1}" +AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID:-sccache}" +AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY:-sccache-secret}" + +# Enable the wrapper ONLY if the bucket is actually reachable. A cache is an +# optimisation and must never be able to break CI: if MinIO is down, mis-DNSed or +# still starting, the correct outcome is slower builds, not failed ones. sccache +# degrades gracefully once running, but a server that cannot start at all would +# take every `cargo` invocation down with it, so this is checked up front rather +# than hoped for. Retried because the runner and MinIO come up concurrently. +# The window is generous (~60s) because compose only orders startup here and does +# not wait for health -- see docker-compose.yml for why gating the fleet on the +# cache would be worse. On a cold `up -d` MinIO may still be initialising its +# volume while the runners boot, and a replica that gives up early would run +# every job uncached until something restarted it. +sccache_reachable=0 +for attempt in $(seq 1 20); do + if curl -fsS --max-time 3 "${SCCACHE_ENDPOINT}/minio/health/live" >/dev/null 2>&1; then + sccache_reachable=1 + break + fi + echo "sccache: ${SCCACHE_ENDPOINT} not ready (attempt ${attempt}/20), retrying..." + sleep 3 +done + +if [[ "$sccache_reachable" == "1" ]]; then + export SCCACHE_BUCKET SCCACHE_ENDPOINT SCCACHE_REGION + export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY + export SCCACHE_S3_USE_SSL="${SCCACHE_S3_USE_SSL:-false}" + # Surfaces storage errors in the job log instead of silently degrading to a 0% + # hit rate, which is exactly how the previous sccache attempt here died + # unnoticed -- it left 1.1 GB per replica of cache last written 2026-07-28 and + # no wrapper ever configured. + export SCCACHE_ERROR_LOG=/tmp/sccache.log + export SCCACHE_LOG="${SCCACHE_LOG:-warn}" + export RUSTC_WRAPPER=sccache + + # Start the server HERE, explicitly, and never let it idle out. Both halves are + # load-bearing, and this was found the hard way. + # + # The sccache server takes its cache configuration from whichever process first + # starts it. Started explicitly with the environment above it comes up on s3 + # ("Cache location s3, name: sccache"); left to be spawned implicitly by + # cargo's first `sccache rustc ...` wrapper call it came up on LOCAL DISK + # instead, reporting `Cache location Local disk` with every compile request + # invisible to the shared bucket. That failure is silent -- builds succeed at + # full speed-looking cost, the bucket stays empty, and the only symptom is a + # cache that never hits. It is exactly the shape of the previous dead sccache + # attempt here, so it gets a real fix rather than a hope. + # + # SCCACHE_IDLE_TIMEOUT=0 keeps the server alive for the container's lifetime. + # The default is 600s, after which the server exits and the NEXT wrapper call + # respawns it -- landing back on local disk and silently unsharing the cache + # between jobs. A runner is idle far longer than ten minutes between jobs, so + # the default would have made this bug the normal case. + export SCCACHE_IDLE_TIMEOUT=0 + gosu runner env \ + SCCACHE_BUCKET="$SCCACHE_BUCKET" SCCACHE_ENDPOINT="$SCCACHE_ENDPOINT" \ + SCCACHE_REGION="$SCCACHE_REGION" SCCACHE_S3_USE_SSL="$SCCACHE_S3_USE_SSL" \ + AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" \ + SCCACHE_IDLE_TIMEOUT=0 SCCACHE_ERROR_LOG="$SCCACHE_ERROR_LOG" \ + SCCACHE_LOG="$SCCACHE_LOG" \ + sccache --start-server 2>&1 | tail -2 || true + + # Assert the server really is on s3, via a throwaway compile first. + # + # `sccache --show-stats` on its own is NOT a trustworthy probe: with no server + # running it reports `Cache location Local disk` from client-side defaults + # without starting one (no SCCACHE_ERROR_LOG is even created), which reads as a + # broken S3 config when nothing is wrong. Diagnosing that cost real time here. + # Routing one trivial compilation through the wrapper guarantees a server + # exists, so the backend line that follows describes reality. + # + # This matters because a cache silently on local disk is worse than no cache: + # it consumes the very volume this exists to relieve and returns a 0% hit rate, + # which is precisely how the previous sccache attempt on this fleet died + # unnoticed. + sccache_canary="$(mktemp -d)" + echo 'fn main() {}' > "${sccache_canary}/canary.rs" + chown -R runner:runner "$sccache_canary" + gosu runner env RUSTC_WRAPPER=sccache SCCACHE_IDLE_TIMEOUT=0 \ + SCCACHE_BUCKET="$SCCACHE_BUCKET" SCCACHE_ENDPOINT="$SCCACHE_ENDPOINT" \ + SCCACHE_REGION="$SCCACHE_REGION" SCCACHE_S3_USE_SSL="$SCCACHE_S3_USE_SSL" \ + AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" \ + sccache rustc --crate-name canary --crate-type lib --emit=metadata \ + -o "${sccache_canary}/canary.rmeta" "${sccache_canary}/canary.rs" >/dev/null 2>&1 || true + rm -rf "$sccache_canary" + + sccache_backend="$(gosu runner sccache --show-stats 2>/dev/null | grep -i 'Cache location' || true)" + case "$sccache_backend" in + *s3*) echo "sccache: ENABLED -> ${SCCACHE_ENDPOINT}/${SCCACHE_BUCKET}" ;; + *) echo "sccache: WARNING -- not on s3 after canary compile, got: ${sccache_backend:-}" ;; + esac +else + echo "sccache: DISABLED (${SCCACHE_ENDPOINT} unreachable) -- builds will be slower but will still succeed" +fi + +echo "Cargo: CARGO_INCREMENTAL=${CARGO_INCREMENTAL} CARGO_PROFILE_DEV_DEBUG=${CARGO_PROFILE_DEV_DEBUG} RUSTC_WRAPPER=${RUSTC_WRAPPER:-}" echo "Starting runner..." gosu runner ./run.sh & From 24c450bc7b6684b46c7a5a46856a40454944a97d Mon Sep 17 00:00:00 2001 From: Riccardo Persello Date: Sun, 9 Aug 2026 22:33:01 +0200 Subject: [PATCH 16/18] feat(fleet): bound target/ with a post-job sweep hook sccache made rebuilding a target dir cheap; it did not make target dirs small. The rlibs, shared objects and test executables still land there at full size, so hbf's reaches 11-13 GB per replica and twelve of them drift back to ~140 GB -- which is how the host volume reached 294 MB free. The cache was therefore only half the fix. ACTIONS_RUNNER_HOOK_JOB_COMPLETED is the runner's own between-jobs hook, so the sweep can never delete a target dir out from under a live compile, which a host cron racing twelve replicas could. It also needs no scheduler and no state on the host. Sweeps largest-first and stops as soon as the replica is back under budget, so it keeps as much warmth as the budget allows rather than emptying wholesale, and it only removes directories carrying CACHEDIR.TAG, debug/ or release/ so a source tree merely named "target" is safe. It always exits 0: this runs after the work is already reported and must never turn a green job red. Budget arithmetic, aimed at keeping the fleet under 100 GB total: images ~10 + cargo registries ~16 + sccache bucket <=20 = ~46 GB fixed 12 replicas x SWEEP_MAX_GB (default 4) = ~48 GB ~94 GB The bucket quota drops 30 -> 20 GiB to make that arithmetic work. Note the hook bounds steady state, not peak: a build in flight may exceed the budget and is swept only once it finishes. SWEEP_MAX_GB trades disk against sweep frequency, and so against job time. Verified with a fake _work: under budget it reports and keeps, over budget it removed a 300 MB then a 120 MB target and stopped, leaving a same-named non-cargo directory and its contents intact. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01TUxoBmSQ9ZNqe1A2FusEsE --- Dockerfile | 5 ++-- docker-compose.yml | 2 +- entrypoint.sh | 9 ++++++ job-completed-hook.sh | 67 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 80 insertions(+), 3 deletions(-) create mode 100644 job-completed-hook.sh diff --git a/Dockerfile b/Dockerfile index ff83c37..f3bc288 100644 --- a/Dockerfile +++ b/Dockerfile @@ -197,9 +197,10 @@ RUN ARCH="${TARGETARCH:-$(dpkg --print-architecture)}" && \ rm -rf /tmp/sccache.tar.gz "/tmp/${SCCACHE_PKG}" && \ sccache --version -# Copy entrypoint script +# Copy entrypoint script and the post-job sweep hook COPY entrypoint.sh /entrypoint.sh -RUN chmod +x /entrypoint.sh +COPY job-completed-hook.sh /usr/local/bin/job-completed-hook.sh +RUN chmod +x /entrypoint.sh /usr/local/bin/job-completed-hook.sh # Create a non-root user and copy tools RUN useradd -m runner && \ diff --git a/docker-compose.yml b/docker-compose.yml index 93b5304..8d26029 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -201,7 +201,7 @@ services: '${SCCACHE_ACCESS_KEY:-sccache}' '${SCCACHE_SECRET_KEY:-sccache-secret}' && mc mb --ignore-existing fleet/sccache && mc ilm rule add --expire-days 14 fleet/sccache 2>/dev/null || true && - mc quota set fleet/sccache --size ${SCCACHE_MAX_SIZE:-30GiB} 2>/dev/null || true && + mc quota set fleet/sccache --size ${SCCACHE_MAX_SIZE:-20GiB} 2>/dev/null || true && echo 'sccache bucket ready:' && mc du fleet/sccache " diff --git a/entrypoint.sh b/entrypoint.sh index 66986d5..8d0efe1 100644 --- a/entrypoint.sh +++ b/entrypoint.sh @@ -320,7 +320,16 @@ else echo "sccache: DISABLED (${SCCACHE_ENDPOINT} unreachable) -- builds will be slower but will still succeed" fi +# Bound `target/` between jobs. sccache makes rebuilding cheap but does NOT make +# target/ small -- the rlibs and test executables still land there at full size, +# so without this the twelve replicas drift back to ~140 GB and refill the disk. +# The runner runs this hook between jobs, so unlike a host cron racing twelve +# replicas it can never delete a target dir out from under a live compile. +export ACTIONS_RUNNER_HOOK_JOB_COMPLETED=/usr/local/bin/job-completed-hook.sh +export SWEEP_MAX_GB="${SWEEP_MAX_GB:-4}" + echo "Cargo: CARGO_INCREMENTAL=${CARGO_INCREMENTAL} CARGO_PROFILE_DEV_DEBUG=${CARGO_PROFILE_DEV_DEBUG} RUSTC_WRAPPER=${RUSTC_WRAPPER:-}" +echo "Sweep: target/ budget ${SWEEP_MAX_GB} GB per replica, enforced after each job" echo "Starting runner..." gosu runner ./run.sh & diff --git a/job-completed-hook.sh b/job-completed-hook.sh new file mode 100644 index 0000000..6a2042c --- /dev/null +++ b/job-completed-hook.sh @@ -0,0 +1,67 @@ +#!/bin/bash +# Runs after EVERY job on this replica, via ACTIONS_RUNNER_HOOK_JOB_COMPLETED. +# +# Why this exists +# --------------- +# sccache made rebuilding a `target/` dir cheap, but it does not make `target/` +# small: rlibs, shared objects and test executables still land there at full +# size, so hbf's reaches 11-13 GB per replica. Twelve of those is ~140 GB, which +# is how a 926 GB volume ended up at 294 MB free on 2026-08-09 -- surfacing not +# as "out of disk" but as `collect2: ld terminated with signal 7 [Bus error]`. +# +# So the disk bound is this hook, and sccache is what makes it affordable: a +# swept replica refills from the shared bucket instead of recompiling. A cold +# rebuild measured 11 misses; after `cargo clean` the same build took 13 hits and +# added no misses, i.e. it came entirely from the cache. +# +# Why a job hook rather than a cron +# --------------------------------- +# The runner invokes this between jobs, so it can never delete a `target/` out +# from under a running compile -- which a host-side cron racing 12 replicas could +# easily do. It also needs no state on the host and no scheduler to keep alive. +# +# Budget +# ------ +# Total is meant to stay under 100 GB: +# images ~10 + cargo registries ~16 + sccache bucket <=20 = ~46 GB fixed +# 12 replicas x SWEEP_MAX_GB = the rest +# At the default of 4 GB that lands near 94 GB. Raising it trades disk for fewer +# sweeps (and so faster jobs); lowering it does the reverse. Note this bounds +# STEADY STATE, not the peak: a build in flight can exceed the threshold, and is +# only swept once it finishes. +set -uo pipefail + +SWEEP_MAX_GB="${SWEEP_MAX_GB:-4}" +WORK_DIR="${RUNNER_WORKSPACE:-/actions-runner/_work}" +# _work holds more than checkouts (_tool, _temp, _actions), so measure the whole +# thing -- that is what actually occupies the writable layer. +[[ -d "$WORK_DIR" ]] || exit 0 + +used_mb=$(du -sm "$WORK_DIR" 2>/dev/null | cut -f1) +[[ -n "$used_mb" ]] || exit 0 +limit_mb=$((SWEEP_MAX_GB * 1024)) + +if (( used_mb <= limit_mb )); then + echo "sweep: _work at ${used_mb} MB, under the ${limit_mb} MB budget -- keeping it warm" + exit 0 +fi + +echo "sweep: _work at ${used_mb} MB exceeds ${limit_mb} MB -- removing target dirs" + +# Largest first, stopping as soon as the budget is met, so a replica keeps as +# much warmth as the budget allows instead of being emptied wholesale. Only +# genuine cargo target dirs are touched: the CACHEDIR.TAG / debug / release test +# avoids deleting a source directory that merely happens to be called "target". +while IFS= read -r dir; do + (( used_mb <= limit_mb )) && break + [[ -f "${dir}/CACHEDIR.TAG" || -d "${dir}/debug" || -d "${dir}/release" ]] || continue + freed=$(du -sm "$dir" 2>/dev/null | cut -f1) + rm -rf "$dir" && used_mb=$((used_mb - ${freed:-0})) + echo "sweep: removed ${dir} (${freed:-?} MB), now ~${used_mb} MB" +done < <(find "$WORK_DIR" -type d -name target -prune 2>/dev/null \ + | while IFS= read -r d; do echo "$(du -sm "$d" 2>/dev/null | cut -f1) $d"; done \ + | sort -rn | cut -d' ' -f2-) + +# Never fail the job. This hook runs after the work that matters is already +# done and reported; a sweep problem must not turn a green job red. +exit 0 From 673bf04280547aa2aff6905f80bb30a5908470b6 Mon Sep 17 00:00:00 2001 From: Riccardo Persello Date: Mon, 10 Aug 2026 08:55:37 +0200 Subject: [PATCH 17/18] fix(fleet): sweep the runner root, not one repository's workspace The hook read `${RUNNER_WORKSPACE:-/actions-runner/_work}`, and RUNNER_WORKSPACE is per-REPOSITORY (`_work/`), not the runner root. With firmware and hbf both checked out, the budget was therefore enforced once per repo, letting each replica hold 2 x SWEEP_MAX_GB. Measured the morning after rollout: five of twelve replicas sat at 6.8-7.2 GB against a nominal 4 GB budget, and the fleet reached 52 GB of _work against its 48 GB ceiling -- the hook was firing correctly (7-13 invocations per replica) and still leaving the fleet over budget, because it was measuring the wrong directory. Now always the runner root, with SWEEP_WORK_DIR left overridable for tests. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01TUxoBmSQ9ZNqe1A2FusEsE --- job-completed-hook.sh | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/job-completed-hook.sh b/job-completed-hook.sh index 6a2042c..ada2cb2 100644 --- a/job-completed-hook.sh +++ b/job-completed-hook.sh @@ -32,7 +32,16 @@ set -uo pipefail SWEEP_MAX_GB="${SWEEP_MAX_GB:-4}" -WORK_DIR="${RUNNER_WORKSPACE:-/actions-runner/_work}" +# The runner root's _work, NOT $RUNNER_WORKSPACE. +# +# This originally read `${RUNNER_WORKSPACE:-/actions-runner/_work}`, which was +# wrong: RUNNER_WORKSPACE is per-REPOSITORY (`_work/`), so the budget was +# enforced once per repo rather than once per replica. With firmware and hbf both +# checked out, each replica could hold 2 x SWEEP_MAX_GB. Measured the morning +# after rollout: five replicas sat at 6.8-7.2 GB against a nominal 4 GB budget, +# and the fleet total reached 52 GB against a 48 GB ceiling. SWEEP_WORK_DIR +# stays overridable for testing. +WORK_DIR="${SWEEP_WORK_DIR:-/actions-runner/_work}" # _work holds more than checkouts (_tool, _temp, _actions), so measure the whole # thing -- that is what actually occupies the writable layer. [[ -d "$WORK_DIR" ]] || exit 0 From f6447085003786efb13a3cf7cda99335e5a72ed8 Mon Sep 17 00:00:00 2001 From: Riccardo Persello Date: Sat, 15 Aug 2026 16:24:56 +0200 Subject: [PATCH 18/18] fix: make Pkl self-install work and stop gitconfig collisions on persistent runners Two bugs only surface on this fleet's long-lived, job-reused containers (never on ephemeral GitHub-hosted runners): 1. Consuming workflows self-install a version-pinned Pkl with `curl -o /usr/local/bin/pkl && chmod +x`. That path is root:root 0755, so the unprivileged runner user hits EACCES. The image also only baked in 0.30.1 while bender-driver/dti-fsic-driver/vehicle-message-definitions all pin 0.31.1 (verified against those repos' ci.yml). Fix: bump the baked-in version to 0.31.1 and chown /usr/local/bin to runner:runner so future version drift no longer hard-fails. A PATH-based redirect can't work here since the destination is a literal absolute path in those workflows, not PATH-resolved. 2. A shared canvas setup snippet writes a git `insteadOf` rewrite via `git config --global set` then `--add`. Values accumulate in the runner user's $HOME/.gitconfig across every job a replica has ever served, until a later plain `set` call hits an already multi-valued key and fails with "cannot overwrite multiple values with a single value". Fix: a new job-started-hook.sh, wired via ACTIONS_RUNNER_HOOK_JOB_STARTED, resets $HOME/.gitconfig before every job -- chosen over job-completed-hook.sh because a cancelled/killed job skips the completed hook and would leak pollution into the next job regardless. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01K6wzprHLZeXyWAxdSM98Hy --- Dockerfile | 37 ++++++++++++++++++++++++++++++---- README.md | 4 +++- entrypoint.sh | 11 +++++++++++ job-started-hook.sh | 48 +++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 95 insertions(+), 5 deletions(-) create mode 100644 job-started-hook.sh diff --git a/Dockerfile b/Dockerfile index f3bc288..063a81c 100644 --- a/Dockerfile +++ b/Dockerfile @@ -62,13 +62,17 @@ RUN curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash # ============================================================================ # Install Pkl (Apple's configuration language - used by canvas) # ============================================================================ +# Version matches what bender-driver/dti-fsic-driver/vehicle-message-definitions +# actually pin in their "Install PKL CLI" workflow step (verified against those +# repos' ci.yml, not assumed) -- see the useradd block below for why this alone +# does not fix those workflows' install step. RUN ARCH="${TARGETARCH:-$(dpkg --print-architecture)}" && \ case "$ARCH" in \ amd64) PKL_ARCH=amd64 ;; \ arm64) PKL_ARCH=aarch64 ;; \ *) echo "Unsupported architecture: $ARCH" >&2; exit 1 ;; \ esac && \ - curl -fL -o /usr/local/bin/pkl "https://github.com/apple/pkl/releases/download/0.30.1/pkl-linux-${PKL_ARCH}" && \ + curl -fL -o /usr/local/bin/pkl "https://github.com/apple/pkl/releases/download/0.31.1/pkl-linux-${PKL_ARCH}" && \ chmod +x /usr/local/bin/pkl # ============================================================================ @@ -197,10 +201,12 @@ RUN ARCH="${TARGETARCH:-$(dpkg --print-architecture)}" && \ rm -rf /tmp/sccache.tar.gz "/tmp/${SCCACHE_PKG}" && \ sccache --version -# Copy entrypoint script and the post-job sweep hook +# Copy entrypoint script, the post-job sweep hook, and the pre-job gitconfig +# reset hook COPY entrypoint.sh /entrypoint.sh COPY job-completed-hook.sh /usr/local/bin/job-completed-hook.sh -RUN chmod +x /entrypoint.sh /usr/local/bin/job-completed-hook.sh +COPY job-started-hook.sh /usr/local/bin/job-started-hook.sh +RUN chmod +x /entrypoint.sh /usr/local/bin/job-completed-hook.sh /usr/local/bin/job-started-hook.sh # Create a non-root user and copy tools RUN useradd -m runner && \ @@ -226,7 +232,30 @@ RUN useradd -m runner && \ # Add source export-esp.sh to runner's bashrc echo 'source $HOME/export-esp.sh 2>/dev/null || true' >> /home/runner/.bashrc && \ # Fix ownership - chown -R runner:runner /home/runner + chown -R runner:runner /home/runner && \ + # Several consuming repos' workflows self-install a version-pinned tool by + # curling a binary straight into /usr/local/bin and chmod +x-ing it -- e.g. + # bender-driver/dti-fsic-driver/vehicle-message-definitions all run: + # curl -L -o /usr/local/bin/pkl https://.../pkl- && chmod +x ... + # On a GitHub-hosted runner this succeeds because the job owns the whole VM. + # Here it hits EACCES: /usr/local/bin is root:root 0755 from the apt/curl + # installs above, and `curl -o` truncates the EXISTING pkl binary in place + # (an open() with O_TRUNC), which needs write on that file's inode, not just + # search/exec on the directory. A PATH-based redirect (e.g. exporting a + # writable $RUNNER_TEMP/bin) cannot fix this: the destination is a literal + # absolute path in those workflows, not something resolved via PATH, and + # editing every consuming repo's workflow is exactly the per-repo workaround + # this fleet's image is meant to avoid. So the directory itself has to + # become writable by the user that actually runs jobs. + # + # chown rather than chmod a+w to match this file's own idiom (chown -R + # runner:runner appears twice above) instead of leaving a world-writable + # system directory. /usr/local/bin holds nothing but the tools this image + # installs (just, pkl, uv/maturin, sccache, bun -- no apt package puts + # anything here), so handing it to runner does not touch anything owned by + # another principal, and the runner user already executes arbitrary job + # code with far broader access than this. + chown -R runner:runner /usr/local/bin # Environment variables for runner user ENV RUSTUP_HOME=/home/runner/.rustup \ diff --git a/README.md b/README.md index 318003d..d5ad604 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,9 @@ This runner includes all tools required for the firmware CI pipeline: ### Build Tools - **just** - Command runner used by the firmware project -- **Pkl** (v0.29.1) - Apple's configuration language (used by canvas) +- **Pkl** (v0.31.1) - Apple's configuration language (used by canvas); `/usr/local/bin` + is writable by the `runner` user so consuming workflows can self-install a + different pinned version without hitting `EACCES` - **maturin** - Build Python wheels from Rust code ### Python diff --git a/entrypoint.sh b/entrypoint.sh index 8d0efe1..9aef3cb 100644 --- a/entrypoint.sh +++ b/entrypoint.sh @@ -328,8 +328,19 @@ fi export ACTIONS_RUNNER_HOOK_JOB_COMPLETED=/usr/local/bin/job-completed-hook.sh export SWEEP_MAX_GB="${SWEEP_MAX_GB:-4}" +# Reset $HOME/.gitconfig before every job. Canvas-consuming repos' shared setup +# snippet writes a git `insteadOf` rewrite with `git config --global set` (then +# `--add`), which is safe on an ephemeral GitHub-hosted runner but accumulates +# in this container's persistent $HOME/.gitconfig job after job until a later +# `set` call hits an already multi-valued key and fails outright. See +# job-started-hook.sh for why this is a job-STARTED hook rather than only +# living in job-completed-hook.sh: it must run regardless of whether the +# previous job finished, was cancelled, or was killed. +export ACTIONS_RUNNER_HOOK_JOB_STARTED=/usr/local/bin/job-started-hook.sh + echo "Cargo: CARGO_INCREMENTAL=${CARGO_INCREMENTAL} CARGO_PROFILE_DEV_DEBUG=${CARGO_PROFILE_DEV_DEBUG} RUSTC_WRAPPER=${RUSTC_WRAPPER:-}" echo "Sweep: target/ budget ${SWEEP_MAX_GB} GB per replica, enforced after each job" +echo "Gitconfig: reset to a clean baseline before each job (ACTIONS_RUNNER_HOOK_JOB_STARTED)" echo "Starting runner..." gosu runner ./run.sh & diff --git a/job-started-hook.sh b/job-started-hook.sh new file mode 100644 index 0000000..41bacef --- /dev/null +++ b/job-started-hook.sh @@ -0,0 +1,48 @@ +#!/bin/bash +# Runs before EVERY job on this replica, via ACTIONS_RUNNER_HOOK_JOB_STARTED. +# +# Why this exists +# ---------------- +# A shared setup snippet used across canvas-consuming repos (bender-driver, +# dti-fsic-driver, vehicle-message-definitions, and others being migrated onto +# this fleet) configures a git `insteadOf` rewrite with +# `git config --global set` (then `--add` for a second value under the same +# key). On an ephemeral GitHub-hosted runner this is harmless: the VM, and +# $HOME/.gitconfig with it, is destroyed the moment the job ends. +# +# On this fleet the container -- and therefore the runner user's +# $HOME/.gitconfig -- outlives any one job, so values written by `--add` +# accumulate under the same key across every job a replica has ever run. A +# later job's plain `set` call then collides with an already multi-valued key +# and the whole step fails with: +# error: cannot overwrite multiple values with a single value +# This is invisible in any single job and only shows up after a replica has +# served enough canvas-consuming jobs to pile up a second value -- which is +# exactly what surfaced once bender-driver/dti-fsic-driver/ +# vehicle-message-definitions started sharing this fleet with firmware/hbf. +# +# Why a job-STARTED hook, and not (only) job-completed-hook.sh +# -------------------------------------------------------------- +# job-completed-hook.sh (see its own header) only runs after a job finishes +# normally. A cancelled, timed-out, or forcibly-killed job skips it entirely, +# and that job's accumulated $HOME/.gitconfig survives into the next one -- +# exactly the collision this hook exists to prevent. A job-STARTED hook runs +# before every job regardless of how the PREVIOUS job ended, so it is the only +# placement that actually closes the gap rather than narrowing it. +# +# What "clean baseline" means here +# --------------------------------- +# Nothing in this image's build or entrypoint.sh ever writes to +# $HOME/.gitconfig for the runner user -- verified by grep, not assumed -- so +# the baseline a freshly created container starts with is simply the file's +# absence. This hook reproduces exactly that, every time, rather than trying +# to selectively undo just the insteadOf rewrite (which would have to know +# every key any consuming repo's setup snippet might someday add). +set -uo pipefail + +rm -f "${HOME:-/home/runner}/.gitconfig" + +# Never fail the job. Like job-completed-hook.sh, this runs adjacent to work +# that must not be put at risk by a cleanup step -- a non-zero exit here would +# fail the job it is meant to protect. +exit 0