fix: writable /usr/local/bin for Pkl self-install, per-job gitconfig reset

[persello]

Summary

Two confirmed bugs in the runner fleet's image, both root-caused by the fact that these containers are long-lived and reused across many job runs — a difference from GitHub-hosted runners that neither bug's underlying pattern accounts for.

Base: this stacks on top of the still-open #3 (feat/runner-fleet-overhaul → main), since that branch is what the live fleet actually runs today (confirmed via the local deployment checkout — main itself is several months stale relative to it). This PR targets feat/runner-fleet-overhaul, not main.

Open issues: gh issue list --repo jkuracing/github-runner --state open returns none. No existing issue covers this work — recording that omission explicitly rather than leaving it silent.

Bug 1 — Pkl self-install fails with EACCES

Consuming workflows (verified against bender-driver, dti-fsic-driver, and vehicle-message-definitions ci.yml, all three identical) do:

curl -L -o /usr/local/bin/pkl https://github.com/apple/pkl/releases/download/0.31.1/pkl-linux-amd64
chmod +x /usr/local/bin/pkl

On a GitHub-hosted runner this works because the job owns the whole VM. On this fleet it fails: /usr/local/bin is root:root 0755 from the image's own apt/curl installs, and curl -o truncates the existing file in place (open(..., O_TRUNC)), which needs write on that file's inode — not just search/exec on the directory. Separately, the image only baked in Pkl 0.30.1, while all three migrated repos pin 0.31.1, so even a working install step would still be re-fetching a version the image doesn't already ship.

Why not the $RUNNER_TEMP/bin / $GITHUB_PATH option: that only helps when a job resolves the tool through PATH. These workflows write to the literal absolute path /usr/local/bin/pkl, not something PATH-resolved, so exporting a different writable directory can't intercept it — the write still targets /usr/local/bin regardless. Editing the destination path in every consuming workflow is exactly the per-repo workaround this task explicitly ruled out. So the only central fix is making that directory (and the files in it) writable by the user that actually runs jobs.

Fix (Dockerfile):

• Bump the image's baked-in Pkl from 0.30.1 → 0.31.1, matching what's actually pinned downstream.
• chown -R runner:runner /usr/local/bin, folded into the existing useradd -m runner block (after it, so the user exists; after every tool install including sccache's install -m 0755, so nothing installed later reverts to root-only). Chose chown over chmod a+w to match this file's own idiom (chown -R runner:runner already appears twice in that block) rather than leaving a world-writable system directory. /usr/local/bin holds only tools this image itself installs (just, pkl, uv/maturin, sccache, bun — no apt package puts anything there), so handing ownership to runner doesn't touch anything owned by another principal, and the runner user already executes arbitrary job code with far broader access than this.
• This also means future Pkl (or similar tool) version drift in a consuming workflow no longer hard-fails — the install step just overwrites the file, which is now permitted.
• Also updated the one Pkl version reference in README.md.

Bug 2 — git config accumulates across job runs

A shared setup snippet used by canvas-consuming repos (the same Install PKL CLI / canvas-deploy-key step family) configures a git insteadOf rewrite via git config --global set … then --add. On an ephemeral GitHub-hosted runner this is harmless — the VM, and $HOME/.gitconfig with it, is destroyed after one job. On this fleet the container (and $HOME/.gitconfig for the runner user) outlives any single job, so values accumulate under the same key run after run, until a later job's plain set call collides with an already multi-valued key and fails with:

error: cannot overwrite multiple values with a single value
       Use a regexp, --add or --replace-all to change url.git@github.com:.insteadOf.

Fix: new job-started-hook.sh, wired via ACTIONS_RUNNER_HOOK_JOB_STARTED in entrypoint.sh, resets $HOME/.gitconfig to a clean baseline (its absence — nothing in this image's build or entrypoint ever writes to it, verified by grep) before every job.

Why a job-STARTED hook, not (only) job-completed-hook.sh: job-completed-hook.sh — the repo's existing per-job hook point, used today for the disk sweep — only runs after a job finishes normally. A cancelled, timed-out, or forcibly-killed job skips it entirely, and that job's accumulated .gitconfig would survive into the next job regardless — exactly the collision this is meant to prevent. ACTIONS_RUNNER_HOOK_JOB_STARTED runs before every job no matter how the previous one ended, so it's the only placement that actually closes the gap rather than narrowing it. It mirrors the existing hook's own idiom: set -uo pipefail (not -e), unconditional exit 0, a header explaining why a cleanup step must never be able to fail the job it's protecting.

What I verified vs. couldn't

Verified (no Docker needed) — reproduced the exact reported failure and confirmed the fix, in plain bash against a real, empty HOME:

$ git config --global url."git@github.com:".insteadOf "https://github.com/"
$ git config --global --add url."git@github.com:".insteadOf "https://github.com/"
$ git config --global url."git@github.com:".insteadOf "https://github.com/"
warning: url.git@github.com:.insteadof has multiple values
error: cannot overwrite multiple values with a single value
       Use a regexp, --add or --replace-all to change url.git@github.com:.insteadOf.
$ bash job-started-hook.sh   # the fix
$ ls ~/.gitconfig
ls: No such file or directory                     # clean baseline restored
$ git config --global url."git@github.com:".insteadOf "https://github.com/"
$ echo $?
0                                                  # now succeeds

NOT verified — disclosing plainly: I did not get a full docker build . of this Dockerfile to complete. A build attempt OOM'd partway through an unrelated, pre-existing layer (libwebkit2gtk-4.1-dev, before my changes are even reached), and shortly afterward OrbStack's engine went fully "Stopped" on the shared host, taking the live 12-container fleet down with it. The coordinator restarted OrbStack and confirmed the fleet is back up and serving real CI, including a safety-relevant firmware PR in flight. Given the timing correlation, retrying the build right now was judged not worth the risk of taking the VM down a second time while real CI depends on it, so a full build-verification pass is deliberately deferred to a quieter moment rather than blocking this PR.

Concretely, this means: the chown -R runner:runner /usr/local/bin step, the Pkl 0.31.1 fetch/URL, and job-started-hook.sh's wiring into the Dockerfile COPY/chmod have not been exercised in an actual built image or container — only read closely and reasoned about. The job-started-hook.sh script logic itself (the part that matters for correctness) is verified per above; what's unverified is purely the image-build mechanics (the COPY, the chown, the new Pkl URL actually resolving inside the build's network context, etc.).

Rollout note (intentionally NOT done here)

This PR is source-only. Deploying it requires rebuilding the live fleet's images with docker compose build (no service argument — that file's own comment warns that scoping to one service silently leaves the other eleven on the old image) and then recreating/restarting the 12 running containers. Until that rebuild happens, the currently-running containers still carry:

• The old /usr/local/bin permissions and Pkl 0.30.1 (Bug 1 still live), and
• Whatever $HOME/.gitconfig state they've already accumulated (Bug 2's existing damage is not cleaned up by this PR — landing the fix only stops further accumulation from the next rebuild onward; any replica already sitting on a multi-valued key needs that file cleared directly, which is a live-infrastructure action outside this PR's scope, to be handled separately with explicit confirmation).

Test plan

• Reproduced Bug 2's exact error message and confirmed job-started-hook.sh fixes it, standalone in bash
• docker build . end-to-end (deferred — see verification section)
• Exec into a throwaway (non-fleet) container built from this branch and confirm pkl --version resolves 0.31.1 as the runner user with which -a pkl showing no earlier shadow copy
• Roll out via docker compose build (no service arg) + recreate the 12 containers, once reviewed
• Separately, clear accumulated .gitconfig state on any replica that already has it (not part of this PR)

🤖 Generated with Claude Code