Overhaul the runner fleet: arm64, 8 replicas, per-replica caches, and hbf support - #3
Draft
persello wants to merge 19 commits into
Draft
Overhaul the runner fleet: arm64, 8 replicas, per-replica caches, and hbf support#3persello wants to merge 19 commits into
persello wants to merge 19 commits into
Conversation
Select architecture-specific downloads from BuildKit's TARGETARCH so the image builds natively on arm64 (e.g. Apple Silicon via OrbStack) as well as amd64: - GitHub Actions runner: linux-x64 / linux-arm64 - Pkl: pkl-linux-amd64 / pkl-linux-aarch64 TARGETARCH is declared without a default, since a default shadows the value the builder injects and would silently fetch amd64 binaries into an arm64 image. Steps fall back to `dpkg --print-architecture` when it is unset so non-BuildKit builds still resolve the host architecture. Downloads now use `curl -f` so a 404 fails the build instead of writing an HTML error page in place of the binary. The Rust, espup and uv toolchains already resolve their own host architecture, and entrypoint.sh has no architecture assumptions. Verified by building --platform linux/arm64 and running the image: pkl 0.30.1 (native), runner 2.336.0, rustc/cargo-nextest on aarch64-unknown-linux-gnu, and the esp toolchain with xtensa-esp-elf-gcc 15.2.0. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CM6943zQZosnY2QugpiQxf
uv installed itself and maturin under /root/.local, and the image copied
that tree into /home/runner/.local. The copy brought along maturin's
launcher symlink, which points at an absolute path inside
/root/.local/share/uv/tools. /root is mode 0700, so the unprivileged
runner user that actually executes jobs could not traverse it:
$ gosu runner maturin --version
error: exec: "maturin": executable file not found in $PATH
It worked as root, which is why this went unnoticed. Both architectures
were affected.
Install uv into /usr/local/bin and its tools into /opt/uv (via
UV_INSTALL_DIR / UV_TOOL_BIN_DIR / UV_TOOL_DIR) so they sit on the shared
PATH and are readable by every user. This also removes the need to copy
the tree into the runner's home at all.
Verified in the arm64 image: `gosu runner maturin --version` reports
1.14.1 and `maturin list-python` resolves CPython 3.12.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CM6943zQZosnY2QugpiQxf
A runner executes one job at a time, so parallelism is purely the replica count. Default it to 4 and bound what each replica may consume. Sized for a 16-core / 64 GB host: 4 replicas x 4 CPUs x 10 GB, leaving headroom for the host OS. CARGO_BUILD_JOBS is pinned to RUNNER_CPUS because cargo otherwise sizes its thread pool from the host core count, so each replica would spawn ~16 threads and N replicas would oversubscribe the machine N-fold; the cpus limit alone only throttles the result rather than preventing the thrashing. All three knobs are overridable via RUNNER_COUNT / RUNNER_CPUS / RUNNER_MEMORY. Replicas now share a cargo-registry volume instead of each re-downloading the full dependency set. Only the registry is shared, not the whole CARGO_HOME: cargo locks that directory so concurrent access is safe, whereas a shared target/ dir would race. entrypoint.sh repairs ownership of the registry volume when it comes back root-owned, which happens for a volume not seeded from the image and would otherwise silently break every build. Also drops the runner-data volume, which was declared but never mounted. Verified on the arm64 image: compose applies the limits outside swarm (NanoCpus=4000000000, Memory=10737418240) across 4 replicas; the runner user can write to the registry volume both when seeded from the image and after the root-owned repair path; and a crate fetched in one container is served to a second via `cargo fetch --offline`, confirming real sharing. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CM6943zQZosnY2QugpiQxf
Every job in the firmware repo's firmware_ci.yml targets `runs-on: labels: [fw-builder]`. With no labels set, a runner registers successfully and then sits idle forever, since no job ever matches it. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CM6943zQZosnY2QugpiQxf
Observed CI state shows runs queuing while host CPU sits idle, so job throughput is runner-starved rather than compute-bound. A single run only reaches 5 concurrent jobs, but firmware_ci.yml keys its concurrency group per branch, so several runs execute simultaneously and jobs queue globally — more replicas do get used. Memory is the binding constraint: 8 x 6 GB = 48 GB of the ~58 GB the OrbStack VM exposes, leaving host headroom. CPUs are oversubscribed 1:1 (8 x 2 = 16) since jobs spend much of their wall time on network and link steps rather than pegged compute. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CM6943zQZosnY2QugpiQxf
Containers are not long-lived, so $HOME/.cache/sccache in the writable layer is discarded on every recreate and each new container recompiles from cold. setup-rust-dual in the firmware repo configures a 25 GB sccache there and describes it as living on persistent runner storage, so that cache is worth keeping across recreates. It cannot be one shared volume. sccache maintains its LRU index in memory per server process, so several containers pointed at one cache directory evict against each other and corrupt it. Since every replica of a scaled service shares one set of volumes, `deploy.replicas` cannot express per-replica storage — the replicas are now eight explicit services built from a YAML anchor, each with its own sccache volume. The crate registry stays shared, which is safe because cargo locks it. _work is still not persisted: the stale submodule target/ it would preserve is exactly what was breaking builds. The Dockerfile pre-creates the sccache directory so its volume is seeded with runner ownership, and the entrypoint's ownership repair now covers both volume paths — a volume that is non-empty and root-owned is not re-seeded by Docker and would otherwise be unwritable by the runner user. Verified: compose resolves 8 services each with a distinct sccache volume and a shared cargo-registry; the runner user can write the sccache volume when seeded from the image; and with a deliberately root-owned non-empty volume, an unrepaired write fails with EACCES while the entrypoint loop restores ownership and the write succeeds. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CM6943zQZosnY2QugpiQxf
Sharing one cargo-registry volume across the 8 runners introduced a new
CI failure absent from every run before it:
error: could not compile `crc32fast` (lib)
Caused by: could not execute process `.../bin/rustc --crate-name crc32fast
.../registry/src/index.crates.io-*/crc32fast-1.5.0/src/lib.rs`
Caused by: No such file or directory (os error 2)
`could not execute process` appears 0 times across runs predating the
shared volume and immediately after it, with the vanished path inside the
shared registry. Unpacked sources under registry/src are removed
mid-compile when another container's cargo garbage-collects the global
cache, so the rustc spawn fails on a working directory that no longer
exists. Cargo's package-cache lock does not cover a build for its whole
duration, and it cannot arbitrate between separate containers.
Give each runner its own registry volume, matching sccache. This costs N
copies of the crate downloads and removes the only remaining shared
mutable state between concurrently building runners.
Note this is distinct from the pre-existing `could not parse/generate dep
info` failures, which point at a submodule's target/ rather than the
registry and are addressed separately in the firmware repo.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CM6943zQZosnY2QugpiQxf
All eight replicas had been offline for about ten days, which also meant firmware CI could not run at all. The failure was silent: nothing was left running to report it. The runner self-updates in place, and a post-update runner drops a `.runner_migrated` marker beside its config. `config.sh` treats that marker ALONE as proof the runner is already configured -- confirmed by creating only `.runner_migrated` and passing a deliberately bogus token, which fails with "Cannot configure the runner because it is already configured" without even attempting to authenticate. The cleanup here stopped at `.credentials_rsaparams`, so every replica that had auto-updated exited 1 on its next restart. Deleting the marker is correct rather than expedient: this entrypoint always reconfigures from a freshly minted registration token, so no migrated state is worth preserving across a restart. `restart: on-failure:5` turned that per-restart failure into a permanent one -- five retries were spent in seconds, after which Docker left the containers dead. A runner fleet should heal rather than latch off, so it becomes `unless-stopped`. The entrypoint mints one token per start, so even a genuinely broken image loops visibly in the logs instead of failing silently. Default RUNNER_TOKEN and RUNNER_NAME to empty as well. Both are optional when GITHUB_PAT is set, but leaving them unset made `docker compose` print two warnings per service -- sixteen lines that buried the real error underneath. Regression test: plant `.runner_migrated` in a live container, restart it, and confirm the count of "Listening for Jobs" lines increases. Do not test this by grepping `docker logs | tail -N` for that string without counting: a container that booted fine and then broke still has the line in its history, which reports a crash-looping runner as healthy. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018cbZiFrvjiXMRHLq1L9PDf
hbf CI runs on GitHub-hosted runners today and reinstalls its toolchain on every job. Moving it here needs two things the image lacked. Add bun and the Tauri desktop dependencies. `cargo build -p hbf-gui` links against webkit2gtk-4.1 and fails at pkg-config time without the -dev package; librsvg2 and appindicator3 are Tauri's SVG and tray-icon dependencies. bun builds the SvelteKit bundle that `tauri::generate_context!()` embeds at COMPILE time, which makes it a build dependency rather than a test-only tool, and it is pinned to the version hbf CI's `oven-sh/setup-bun` requests so lockfile resolution matches. BUN_INSTALL puts the binary on the shared PATH instead of under /root, which is mode 0700 and so invisible to the unprivileged runner user -- the same trap the uv block already documents. Both layers go AFTER espup deliberately. Docker invalidates every layer below an edited one, and rebuilding the Xtensa toolchain costs many minutes. Verified the `esp` toolchain survived the rebuild untouched. Image grows 8.86 -> 9.6 GB. Add `hbf-builder` to every replica rather than reserving a subset for it. A runner is offered a job only when its labels are a SUPERSET of the job's `runs-on`, so splitting them (1-6 fw-builder, 7-8 hbf-builder) would leave six containers ineligible for hbf work and idle whenever hbf work is all that is queued. Both labels everywhere means any replica serves either repo, and capacity is added by adding replicas. One trap worth recording: `docker compose build` must be run with NO service argument. Each service declares its own `build: .`, so compose tags a separate image per service, and `docker compose build runner-1` silently leaves the other seven on the old image -- they still start and register, so nothing looks wrong until a job needs a tool only the rebuilt image has. hbf CI cannot move here yet: `.github/actions/setup-canvas` downloads pkl-linux-amd64 while these runners are arm64, and it writes to /usr/local/bin, which the runner user cannot do. pkl is also skewed three ways (0.30.1 here, 0.31.1 in hbf CI, 0.32.1 on the dev machine). Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018cbZiFrvjiXMRHLq1L9PDf
hbf's `ts_export` test execs `ui/node_modules/.bin/prettier` directly from Rust. That file is a .cjs script whose shebang is `#!/usr/bin/env node`, so on this image the exec failed with status 127 and the drift check reported "bindings would drift from CI's regen" -- a misleading message for a missing interpreter. bun does not substitute for node here. `bun run lint` and `bun run check` work because `bun run` interprets the JS itself and never consults the shebang, which is why the gap stays invisible until something shells out to a .bin entry. npm comes along for `npx`, which the same test falls back to when the project-local binary is absent. GitHub-hosted runners preinstall both, so this could only surface here. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018cbZiFrvjiXMRHLq1L9PDf
This block claimed "Memory is the binding constraint, not CPU" at 8 x 6 GB. Measured under a full load of firmware and hbf jobs, that was wrong on both counts: peak memory across all replicas was 756 MiB against the 6 GiB limit -- an 8x overshoot -- and only 4 of 8 containers were computing at all (~195% CPU each), the rest sitting near idle on network and setup. Roughly half the host's cores went unused while jobs queued. The real constraint was SLOTS. An hbf run measured 16.8 minutes of job time inside an 11.8 minute span -- an average concurrency of 1.4 -- because firmware held 7 of the 8 slots, so a pipeline that takes 2m41s on a hosted runner took 11m48s here. Hence more, smaller replicas. Total memory is unchanged at 48 GB of the ~58 GB the VM exposes. CPU is now oversubscribed 1.5:1 (12 x 2 = 24 on 16 cores), which the measured idle time justifies. Disk is the limiting factor now, not memory: each replica keeps its own target/ for both repositories on one 200 GB volume. Reclaiming 27 GB of stale build cache brought it to 43% before this change; check `docker system df` before going wider. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018cbZiFrvjiXMRHLq1L9PDf
The host volume hit 294 MB free of 926 GB and took firmware CI down with it. Disk exhaustion does not present as "out of disk" -- it surfaces as `collect2: ld terminated with signal 7 [Bus error]`, the linker dying mid-write, which reads like an LLVM bug and is not one. hbf's own ci.yml documents the same symptom on hosted runners. Measured rather than assumed: of ~200 GB of Docker, 170 GB is the twelve container writable layers, and 85% of that is one directory per replica -- hbf's target/ at 11-13 GB, against firmware's already-lean 0.9 GB. The per-replica cargo registry and sccache volumes that this file goes to such lengths to keep separate are only ~29 GB combined, so the duplication this file warns about is not what filled the disk, and sharing those would not have fixed it. target/ cannot be shared at all: cargo takes an exclusive lock per target directory, so one shared dir would serialise all twelve replicas and destroy the parallelism that is the point of the fleet. The lever is a smaller target/, not fewer of them. CARGO_INCREMENTAL=0 is the actual reclaim, at 2.8 GB per replica. That state is pure waste in CI, where every job is a different commit and nothing reuses the dep-graph fingerprints; the same directory reaches 14 GB in a long-lived developer checkout. CARGO_PROFILE_DEV_DEBUG is a floor rather than a saving. hbf already set it in 3a6e615 and the measured dirs are already reduced -- objdump shows .debug_loc at 0 bytes with .debug_line dominant. It is set here so the property holds for every job regardless of per-workflow config, and because firmware_ci.yml covers only its release profile. Both live here rather than in either workflow so they apply fleet-wide without changing hosted runners or a developer's laptop, and neither repo sets them in workflow env, so the container's value is what a job sees. These are prospective: existing target dirs keep their bloat until rebuilt. Recreating a container is what clears _work, since it lives on the writable layer while the registry and sccache volumes survive. Also records that sccache is dead -- absent from the image, never set as RUSTC_WRAPPER, last written 2026-07-28 -- because that is precisely why wiping a target/ dir today is a cold rebuild instead of a cheap one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TUxoBmSQ9ZNqe1A2FusEsE
The previous commit set CARGO_INCREMENTAL in docker-compose.yml, which cannot take effect without destroying the thing it was meant to save. `docker compose` bakes a container's environment at CREATION, so an edited compose file reaches a job only after `up -d` recreates the container -- and `_work`, holding the 11-13 GB warm `target/` dir per replica, lives on the writable layer and goes with it. Since sccache is dead here (absent from the image, never set as RUSTC_WRAPPER, last written 2026-07-28), that recreate is a genuinely cold rebuild across all twelve replicas rather than a cheap one. Exporting from entrypoint.sh instead means a plain `docker restart` applies a change while preserving every warm `deps/` dir. The runner inherits the entrypoint's environment and hands it to each job step, so the export reaches the compiler. /actions-runner/.env would have been the obvious mechanism and does not work: it is read only by the systemd unit `svc.sh` generates, whereas this entrypoint execs ./run.sh directly and run.sh contains no reference to it -- checked rather than assumed. The stock file is empty and `env.sh` only writes it for that service path. The defaults therefore live in one place, entrypoint.sh, with docker-compose.yml documenting the override point rather than repeating the literals. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TUxoBmSQ9ZNqe1A2FusEsE
Both previous commits justified it as "pure waste in CI, since every job builds a different commit and nothing reuses the dep-graph fingerprints". That is wrong for this fleet: hbf's ci.yml uses `clean: false` precisely to keep `target/` warm across jobs, so successive jobs on one replica can hit an incremental cache. It is a trade, not a free win, and the comment now says so. The trade still favours dropping it, for reasons that are bounded rather than absolute. Incremental covers only hbf's dozen workspace crates -- registry dependencies, the entire 8.4 GB bulk of debug/deps, compile non-incrementally either way. A hit further requires the same replica to rebuild a nearly identical commit, and jobs land on whichever of the 12 is free with no branch affinity. Where nothing changed, ordinary fingerprinting already skips the crate. And a hit is not free: incremental raises the codegen-unit count and gives some of the saving back at link time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TUxoBmSQ9ZNqe1A2FusEsE
Twelve replicas each keep their own `target/`, hbf's reaches 11-13 GB, and that filled a 926 GB volume to 294 MB free. sccache does not shrink those dirs -- rlibs and test executables still land at full size -- but it makes DELETING them cheap, which is what allows them to be treated as disposable instead of hoarded. Disk stays bounded by sccache plus a recurring sweep; neither does it alone. Shared via MinIO rather than per-replica. Twelve private caches would each warm from scratch, leaving the first build on every replica cold, which is most of what a cache exists to prevent. Pointed at one bucket, whichever replica compiles a crate first serves the other eleven. This does not contradict the header comment forbidding a shared cache DIRECTORY: that hazard is sccache's per-process in-memory LRU index, which a server backend does not have. MinIO over Redis because a cache this size belongs on disk; Redis would hold it in RAM, competing with the 48 of ~58 GB already committed to the replicas. Verified end to end rather than assumed. A cold build recorded 11 misses against `Cache location s3`; after `cargo clean`, the rebuild took hits 1 -> 13 with misses unchanged at 11, so the entire rebuild came from the bucket. That is the property the wipe depends on. Two failure modes found while verifying, both silent, both now handled: - The server takes its cache config from whichever process first starts it. Spawned implicitly by cargo's first wrapper call it came up on LOCAL DISK, with every request invisible to the bucket. It is now started explicitly from here, and SCCACHE_IDLE_TIMEOUT=0 stops it exiting after the default 600s -- a runner idles far longer than that between jobs, which would have made the bad path the normal one. - `sccache --show-stats` alone is not a valid probe: with no server running it reports `Local disk` from client-side defaults without starting one. The startup check therefore routes one throwaway compile through the wrapper first, so the backend it reports is real. The cache is fail-safe: entrypoint.sh enables the wrapper only if the bucket answers, and compose orders the runners after MinIO without gating on its health, so a cache outage means slower CI rather than no CI. The bucket is bounded, which sccache will not do itself -- it has an LRU for a local directory but never deletes from S3. sccache-s3-init sets a 14-day expiry rule plus a 30 GiB quota backstop; expiry is primary because a hard quota makes writes fail once reached, while expiry just forgets what has not been useful lately. The twelve per-replica sccache volumes are removed, since an S3-backed sccache keeps no local cache directory. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TUxoBmSQ9ZNqe1A2FusEsE
sccache made rebuilding a target dir cheap; it did not make target dirs
small. The rlibs, shared objects and test executables still land there at
full size, so hbf's reaches 11-13 GB per replica and twelve of them drift
back to ~140 GB -- which is how the host volume reached 294 MB free. The
cache was therefore only half the fix.
ACTIONS_RUNNER_HOOK_JOB_COMPLETED is the runner's own between-jobs hook, so
the sweep can never delete a target dir out from under a live compile, which
a host cron racing twelve replicas could. It also needs no scheduler and no
state on the host.
Sweeps largest-first and stops as soon as the replica is back under budget,
so it keeps as much warmth as the budget allows rather than emptying
wholesale, and it only removes directories carrying CACHEDIR.TAG, debug/ or
release/ so a source tree merely named "target" is safe. It always exits 0:
this runs after the work is already reported and must never turn a green job
red.
Budget arithmetic, aimed at keeping the fleet under 100 GB total:
images ~10 + cargo registries ~16 + sccache bucket <=20 = ~46 GB fixed
12 replicas x SWEEP_MAX_GB (default 4) = ~48 GB
~94 GB
The bucket quota drops 30 -> 20 GiB to make that arithmetic work. Note the
hook bounds steady state, not peak: a build in flight may exceed the budget
and is swept only once it finishes. SWEEP_MAX_GB trades disk against sweep
frequency, and so against job time.
Verified with a fake _work: under budget it reports and keeps, over budget it
removed a 300 MB then a 120 MB target and stopped, leaving a same-named
non-cargo directory and its contents intact.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TUxoBmSQ9ZNqe1A2FusEsE
The hook read `${RUNNER_WORKSPACE:-/actions-runner/_work}`, and
RUNNER_WORKSPACE is per-REPOSITORY (`_work/<repo>`), not the runner root. With
firmware and hbf both checked out, the budget was therefore enforced once per
repo, letting each replica hold 2 x SWEEP_MAX_GB.
Measured the morning after rollout: five of twelve replicas sat at 6.8-7.2 GB
against a nominal 4 GB budget, and the fleet reached 52 GB of _work against
its 48 GB ceiling -- the hook was firing correctly (7-13 invocations per
replica) and still leaving the fleet over budget, because it was measuring the
wrong directory.
Now always the runner root, with SWEEP_WORK_DIR left overridable for tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TUxoBmSQ9ZNqe1A2FusEsE
…istent runners Two bugs only surface on this fleet's long-lived, job-reused containers (never on ephemeral GitHub-hosted runners): 1. Consuming workflows self-install a version-pinned Pkl with `curl -o /usr/local/bin/pkl && chmod +x`. That path is root:root 0755, so the unprivileged runner user hits EACCES. The image also only baked in 0.30.1 while bender-driver/dti-fsic-driver/vehicle-message-definitions all pin 0.31.1 (verified against those repos' ci.yml). Fix: bump the baked-in version to 0.31.1 and chown /usr/local/bin to runner:runner so future version drift no longer hard-fails. A PATH-based redirect can't work here since the destination is a literal absolute path in those workflows, not PATH-resolved. 2. A shared canvas setup snippet writes a git `insteadOf` rewrite via `git config --global set` then `--add`. Values accumulate in the runner user's $HOME/.gitconfig across every job a replica has ever served, until a later plain `set` call hits an already multi-valued key and fails with "cannot overwrite multiple values with a single value". Fix: a new job-started-hook.sh, wired via ACTIONS_RUNNER_HOOK_JOB_STARTED, resets $HOME/.gitconfig before every job -- chosen over job-completed-hook.sh because a cancelled/killed job skips the completed hook and would leak pollution into the next job regardless. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K6wzprHLZeXyWAxdSM98Hy
5 tasks
fix: writable /usr/local/bin for Pkl self-install, per-job gitconfig reset
persello
marked this pull request as draft
August 15, 2026 17:03
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nine commits that had accumulated locally and were never pushed — the fleet's real configuration existed only on one machine. The last two were written today in response to the fleet having been completely offline.
The outage (3f0f1e6)
All eight replicas had been dead for about ten days, which also meant firmware CI could not run at all. Nothing was left running to report it.
The runner self-updates in place, and a post-update runner drops a
.runner_migratedmarker beside its config.config.shtreats that marker alone as proof the runner is already configured — confirmed by creating only.runner_migratedand passing a deliberately bogus token, which fails withCannot configure the runner because it is already configuredwithout even attempting to authenticate. The cleanup inentrypoint.shstopped at.credentials_rsaparams, so every replica that had auto-updated exited 1 on its next restart.restart: on-failure:5then turned a per-restart failure into a permanent one: five retries were spent in seconds, after which Docker left the containers dead. Nowunless-stopped, so the fleet heals rather than latching off.Also worth knowing: OrbStack itself was
Stopped, and Docker autostarts it on socket access then can drop out mid-command. Checkorb statusbefore diagnosing anything else.hbf support (7adc233)
Adds
bun1.3.14 and the Tauri deps (libwebkit2gtk-4.1-dev,libayatana-appindicator3-dev,librsvg2-dev), and putshbf-builderalongsidefw-builderon every replica.Both labels everywhere is deliberate. A runner is offered a job only when its labels are a superset of the job's
runs-on, so splitting them (1–6fw-builder, 7–8hbf-builder) would leave six containers ineligible for hbf work and idle whenever hbf work is all that's queued. Capacity gets added by adding replicas, not by reserving them.New layers go after espup so a rebuild never invalidates the multi-GB Xtensa toolchain. Verified the
esptoolchain survived untouched. Image grows 8.86 → 9.6 GB.Earlier commits (75ae855…862ab62)
arm64 builds alongside amd64; a fix for maturin being unusable by the runner user; per-replica resource limits and
CARGO_BUILD_JOBS; 8 replicas at 2 CPU / 6 GB; and per-replica sccache and cargo-registry volumes — the shared registry caused real CI failures where unpacked sources underregistry/srcvanished mid-compile when another container's cargo garbage-collected them.Verification
fw-builder,hbf-builder,RestartCount=0..runner_migratedin a live container, restarted, and confirmed the count ofListening for Jobslines increased and the marker was deleted.Two traps recorded in the commit messages
docker compose buildmust be run with NO service argument. Each service declares its ownbuild: ., so compose tags a separate image per service;docker compose build runner-1silently leaves the other seven on the old image. They still start and register, so nothing looks wrong until a job needs a tool only the rebuilt image has. I hit this myself.docker logs | tail -NforListening for Jobs. A container that booted fine and then broke still has that line in its history, so a restart test done this way reports a crash-looping runner as healthy. Count the occurrences and require an increase.Not yet possible
hbf CI cannot move here until:
setup-canvasdownloadspkl-linux-amd64while these runners are arm64; it writes to/usr/local/bin, which the non-rootrunneruser cannot; and pkl is skewed three ways (0.30.1 here, 0.31.1 in hbf CI, 0.32.1 on the dev machine). Unifying pkl touches firmware CI, so it wants its own verification.Issues
No open issues in this repo, so this closes nothing.
🤖 Generated with Claude Code
https://claude.ai/code/session_018cbZiFrvjiXMRHLq1L9PDf