perf(ci): Bigger install chunks, a shard resource sampler, and lower-priority checks

[krlmlr]

Three changes to what the preflight and the shards spend, and to what they say while spending it.

1. A shard samples its resources, and checks at a lower priority

Shard 16 of run 31951756102 died with

The hosted runner lost communication with the server. Anything in your
workflow that terminates the runner process, starves it for CPU/Memory,
or blocks its network access can cause this error.

and there was not one number from that job to say which of those it was.

The preflight has sampled memory, disk, load and the three largest processes every 30 seconds since the incident that motivated watch-resources.sh. The shards, which run for hours rather than minutes, had nothing. A runner that stops answering takes its if: always() steps with it, so the only record that survives is what was already streamed to the log — which is the entire reason the sampler exists, and it was in the wrong job.

It is backgrounded once, before the install: a process started in one step outlives it, and Actions reaps it with the job. RESOURCE_LOG also puts the series in the shard artifact for the jobs that do reach their upload. A final always() step adds the last sample and asks dmesg whether the kernel killed anything — the difference between running out of memory and the host going away.

And the checks now run under nice -n 10, plus ionice -c3 where it exists. Two R CMD check processes run at once, each with children of its own — a test suite that opens a PSOCK cluster, a vignette that knits — and together they can take every core the runner has. The runner agent is a process on that machine too, and it has to reach the service regularly or the job dies. nice costs nothing when there is headroom: the scheduler only consults priority when there is more work than cores, which is exactly the case worth protecting. ionice does the same for the disk, where a check writing its .Rcheck directory competes with the agent writing logs.

This does not prove starvation was the cause. It means the next one says so.

2. 400 packages per pak call, not 100

A chunk pays one resolution whether or not it installs anything. Run 31930350338's preflight, verbatim:

✔ 80 pkgs + 214 deps: kept 294 [44s]
Preflight: chunk 4/4 (80 packages) installed after 0.7 min

44 seconds of pak for a chunk that built nothing. At 100 per chunk the 4406-package universe is 45 chunks — on the order of half an hour of resolution before a single build starts, all of it on the critical path, because every shard waits for the preflight. At 400 it is 12 chunks.

Why it was 100. One pak call for the whole universe killed run 31270092803 — ten minutes inside pak, not one install started, then the runner was shut down. That resolution was a few thousand refs; 400 is an order of magnitude below it, and still small enough that a chunk which dies costs a chunk rather than the job. The cost is that a failed chunk is four times as much to redo. REVDEP2_INSTALL_CHUNK remains the knob.

Chunking is otherwise identical: on 1200 CRAN packages, 3 chunks of 400 instead of 12 of 100, and every strong dependency still precedes the package that needs it — 0 out of order.

3. The load test says what it is doing

It captured everything and printed it once at the end, sorted slowest first. That group answers "what was slow", which arrival order cannot, so it stays — but a sweep of 1173 packages taking half an hour said nothing while it ran.

[load    1/1173] OK      red                              19s
[load    2/1173] OK      PopGenReport                     13s
[load    3/1173] TIMEOUT somethingslow                   600s

Each verdict goes to stdout (the caller's data, unchanged) and to stderr (the live log); the caller captures the first and inherits the second. The running count needs no lock — each finished package appends one byte and reads the size back, and single-byte O_APPEND writes do not interleave.

Verified

• nice -n 10 + ionice -c3 prefix applies: the child reports niceness 10 and I/O class idle. The ionice half is probed before use, so an image without it degrades to nice alone.
• The sampler wiring, with SHARD=16: a labelled "before the install" line, then shard 16 samples every interval, both to the log and to RESOURCE_LOG in the results directory.
• Chunking on 1200 packages: 3 × 400, no out-of-order dependencies.
• Load test on 8 packages and on a list containing one that does not exist: indices 1..n with no gaps or repeats, ERROR on the live line, FAIL nosuchpackage error 0 on stdout, temp file removed on exit.
• revdep2.yaml parses; the two shell scripts pass bash -n.

Complements #2854, which bounds what a lost runner costs; this one is about seeing it coming and making it less likely.

🤖 Generated with Claude Code

https://claude.ai/code/session_01D1xpHRV7yVfgtJg4vp9P7z