What happened, live
A bench deploy this session ran sudo os/rauc/mkbundle.sh --dev without a fresh os/build-image.sh first. mkbundle found the previous session's os/build/pithead-root.tar, bundled it without complaint, RAUC installed it, the appliance booted it, every service came up green — and /opt/pithead/BUILD_COMMIT read the old commit. The fix under test was never on the box. Only the deploy checklist's explicit BUILD_COMMIT comparison caught it; nothing in the tooling did.
mkbundle's only guard today is existence: missing $TARBALL — run os/build-image.sh first. A present-but-stale tarball is the dangerous case, because everything downstream looks like a successful deploy of the new code.
Why it deserves a guard, not a runbook line
This is the green-lie class the repo already treats as upgrade-blocking elsewhere (a failed build inside an && chain leaving an old artifact for a later verify to bless). The information to catch it is already inside the artifact: the tarball contains /opt/pithead/BUILD_COMMIT, written by build-image.sh from the tree it actually built.
Fix direction
In mkbundle.sh (and mkimage.sh, which consumes the same tarball): extract opt/pithead/BUILD_COMMIT from the tarball (one tar -xO of one file — cheap) and compare against git rev-parse HEAD of the working tree:
- match → proceed;
- mismatch or dirty suffix → refuse by default, naming both commits, with an explicit override (
PITHEAD_STALE_TARBALL_OK=1 or similar) for the rare deliberate case — the same fail-closed-with-explicit-escape shape the version and variant guards already use.
A stack-suite case pins it with a fixture tarball (the tar layout is trivial to fabricate; no image build needed).
What happened, live
A bench deploy this session ran
sudo os/rauc/mkbundle.sh --devwithout a freshos/build-image.shfirst. mkbundle found the previous session'sos/build/pithead-root.tar, bundled it without complaint, RAUC installed it, the appliance booted it, every service came up green — and/opt/pithead/BUILD_COMMITread the old commit. The fix under test was never on the box. Only the deploy checklist's explicit BUILD_COMMIT comparison caught it; nothing in the tooling did.mkbundle's only guard today is existence:
missing $TARBALL — run os/build-image.sh first. A present-but-stale tarball is the dangerous case, because everything downstream looks like a successful deploy of the new code.Why it deserves a guard, not a runbook line
This is the green-lie class the repo already treats as upgrade-blocking elsewhere (a failed build inside an
&&chain leaving an old artifact for a later verify to bless). The information to catch it is already inside the artifact: the tarball contains/opt/pithead/BUILD_COMMIT, written by build-image.sh from the tree it actually built.Fix direction
In
mkbundle.sh(andmkimage.sh, which consumes the same tarball): extractopt/pithead/BUILD_COMMITfrom the tarball (onetar -xOof one file — cheap) and compare againstgit rev-parse HEADof the working tree:PITHEAD_STALE_TARBALL_OK=1or similar) for the rare deliberate case — the same fail-closed-with-explicit-escape shape the version and variant guards already use.A stack-suite case pins it with a fixture tarball (the tar layout is trivial to fabricate; no image build needed).