fix(ci): Upload a shard's results three times, not once

[krlmlr]

What happened to shard 16

Not a driver failure. The UI reports:

The hosted runner lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

and the job log ends:

14:59:17  Install phase complete; the check phase runs as its own step
15:00:30  Checking 87 package(s), old and new side by side
15:03:45  ##[error]The runner has received a shutdown signal.
15:03:45  ##[error]Process completed with exit code 143.

Three minutes into a 196-minute check budget, while the other 39 shards of that wave carried on. #2852 addresses the cause side — a resource sampler in the shard so the next one says which of CPU, memory or network it was, and nice/ionice on the checks so the runner agent is never the process that loses. This PR addresses what it cost.

What made it expensive, which is ours

The driver has written its results as it goes since #2836 — one manifest line per package, appended, precisely so that a shard killed part way through still accounts for what it finished. But the upload was a single step after the checks, and if: always() cannot save a step on a runner that is no longer answering. It was skipped:

 16 Check the shard                     completed  failure  14:59:17 -> 15:03:45
 17 Upload the shard results            completed  skipped  15:03:45 -> 15:03:45

All 87 packages came back missing, after an hour of runner time (8 min setup, 5 min install, 4 min checking).

The change

The checks run in three slices, each followed by an upload. The slices share the results directory and overwrite one artifact name, so the newest upload to survive is a superset of the ones before it, and the loss is bounded to a third of the shard.

Packages are dealt round robin, not in contiguous blocks: runnable is ordered heaviest-first, so a block cut would put every long check in slice 1 and leave slice 3 nothing but cheap ones — and the deadline, which stops the shard when the next check will not fit, would then bite unevenly.

Three things had to become slice-aware:

• The end-of-run "whatever the loop never reached" pass reads the manifest back and skips packages already accounted for. Without that, slice 2 would bury slice 1's results under a deferred line — and a later line wins in the collector. It still writes deferred for later slices' packages, which is the truth at that moment and better than the missing the collector would otherwise reconcile them into.
• timing.json accumulates checks and check_seconds across slices. The collector fits the cost model from these, and a third of a shard's checks next to a whole job's script_seconds would price every shard at a third of what it costs — exactly the kind of underestimate that tips a plan into extra waves.
• The per-slice summary covers that slice's packages, not the shard's, whose other two thirds are still at their initial deferred in this process.

CHECK_SLICE is unset outside the workflow, and then nothing slices — a local PHASE=check invocation behaves exactly as before.

Rebased onto #2849, #2852 and #2853

The step list is the merge of this PR's slices with #2852's sampler and consumption report, in the order that keeps both working:

Start the resource sampler
Install packages
Check the shard (1 of 3)  → Upload the shard results (1 of 3)
Check the shard (2 of 3)  → Upload the shard results (2 of 3)
Check the shard (3 of 3)
Report what the shard consumed        (always)
Upload the shard results              (always, overwrite)

The consumption report stays immediately before the final upload, so its last sample and OOM verdict still ride in the artifact; every upload gained overwrite: true, which the repeated name now requires. In the README, the three rows that had grown to describe the same event — #2852's starvation row, and two of mine — are one starvation row plus one merged "a shard job dies hard" row.

shard.R merged cleanly and carries all three changes: CHECK_SLICE here, ensure_check_sysreqs() from #2853, depmissing from #2849.

Verified

• Slice arithmetic on 10 packages: 1/3 → 4, 2/3 → 3, 3/3 → 3; union equals the input, no overlaps.
• The manifest guard: with pkgA=ok and pkgB=deferred on disk and slice 2 having checked pkgB, the unreached pass writes nothing, pkgA is protected, and last-line-wins leaves pkgA=ok pkgB=ok.
• needs_recheck() is TRUE for both deferred and the newly merged depmissing, FALSE for ok — so a slice never reached is retried either way.
• revdep2.yaml parses and the step order above is what it yields; all five R scripts parse and are air-formatted; all three shell scripts pass bash -n.

Not fixed here

Nothing prevents the runner going away. What this bounds is the cost — and retry-run still picks up whatever a slice never reached.

🤖 Generated with Claude Code

https://claude.ai/code/session_01D1xpHRV7yVfgtJg4vp9P7z