feat(ci): Add revdep4, sequential halves driven by a two-ended bash work queue

[krlmlr]

Proposes revdep4, the queue engine: old and new halves of each reverse dependency checked sequentially, with the lost concurrency won back across packages by a custom bash work queue — one of two sibling proposals (the other is revdep3, #2856). Both reuse the good bits of revdep2 through a shared, engine-agnostic core, and are fully interoperable with each other.

The problem

revdep2 checks each reverse dependency against the released and the dev igraph simultaneously on one host. The PSOCK port collision (patched with R_PARALLEL_PORT) was one member of an open-ended interference class — shared /tmp, caches, locks, any singleton a check assumes it owns. Simultaneously checking the same package with two libraries is not a supported mode of operation.

What revdep4 does

Removes the simultaneity instead of isolating it: per package, the old half runs, then the new half — no overlap, ever. Parallelism moves to the package level: a shard runs workers packages at once (default 4, the runner's cores), each check in its own container, so different packages cannot collide either (stronger than revdep2, which only separated the two halves' ports).

The queue (revdep4/queue.sh, bash, flock-guarded cursors) consumes the shard's heaviest-first package list from both ends: one heavy lane takes the heaviest remaining package — the longest checks start first and are never the straggler discovered last — while the remaining light lanes drain the long tail of cheap packages from the other end; the two meet in the middle. Deadline-aware claiming defers what no longer fits; a crashed claim still writes an error manifest line through a four-rung fallback ladder (compare-one → --error rerun → printf-JSON template → supervisor sweep), so no claimed package can ever vanish silently; three check slices with an upload after each bound what a reclaimed runner can lose.

Heaviness — answering the design question directly: the queue key is expected check seconds — measured on this infrastructure by earlier revdepx runs (either workflow, youngest wins), else CRAN's T_total × a self-calibrating scale, else the cohort median. Package size and dependency count/size are deliberately not in the key: they price installation, which the shared universe image amortizes to zero (dependency count still gates the depfail screen). This is revdep2's proven cost model, re-priced for sequential halves (×2 per package, ÷workers per shard, with a max(heaviest, sum/workers) bound so a giant-dominated shard is not flattered).

Sequential halves make per-half timings real: t_old/t_new become true measurements (revdep2 could only record the pair's wall clock for both), which sharpens the cost model over time.

The old half always runs; a stored old result is a second opinion only. Where the plan certifies an earlier run's old-version result as comparable — same revdep version, our CRAN version, container R, base-image tag, dependency fingerprint, within age — the fresh old check is compared against it and baseline_agrees records whether they match, with disagreements reported as drift. It never substitutes for the check, however tempting the saved wall clock: a fresh old is the only result whose provenance the run fully controls. (revdep2-era baselines lack the base-image key and are never offered — they were measured on a different platform entirely.)

Pipeline (shared with revdep3): plan + base image (parallel) → build (dev binary inside the container) → universe image (whole dependency universe + sysreqs on GHCR, delta-updated, age-bounded) → test shards (queue) → collect. Containers also bring per-check memory caps (an OOM kills one container, not the runner) and a pinned check platform (r-version, default oldrel — a fixed target).

Kept from revdep2 / dropped

Kept (via .github/workflows/revdepx/): the planner with all its inputs (packages/broken/retry-run/part/dry-run), the manifest schema and result vocabulary, baseline lineage, timings + calibration, three-slice uploads, resource sampler, chunked deadline-bounded installs, load test, comparison/diff/salvage, report/collect and the committed revdep/ record. Dropped: the port hack, the preflight job and all prebuilt-library artifacts/donor walks/tar machinery, host toolchain setup on shards, per-shard host installs (kept only as disaster-recovery fallback when the universe image is unavailable). revdep2 itself is untouched by this PR.

Compatibility with revdep3

Both PRs ship .github/workflows/revdepx/ byte-for-byte identical (this PR adds revdep4.yaml + revdep4/; #2856 adds revdep3.yaml + revdep3/; merging both leaves one copy, no conflicts). One artifact family (revdepx-*), one manifest/baseline/timings schema, one comparison code path, one universe-image lineage, one concurrency group per ref. Either workflow reads the other's baselines as second opinions, consumes the other's timings (canonical per-half seconds; each plan prices its own engine from them), retries the other's runs, and starts from whichever universe image was refreshed last.

Notes for review

• First push creates the GHCR packages; org policy forbidding GITHUB_TOKEN package writes degrades gracefully to artifact transport.
• revdep2-era baselines/timings are never consumed (different platform, and the base-image key walls them off); the first run checks everything fresh.
• Validated locally: plan.R dry runs against live CRAN metadata for both engines (775 revdeps → 20 shards; queue weights verified at 2× pair weights, worker-aware shard walls, the giant multinma isolated by the LPT bound); the queue exercised standalone with stub checks (single-claim guarantees, two-ended consumption, deadline defer, all fallback rungs, the final sweep) and end-to-end against a local Docker daemon with rocker/r-ver:4.5.3 — two real CRAN packages checked sequentially in containers with real per-half timings, and the second-opinion path verified (fresh old check ran, stored result consulted, baseline_agrees recorded on the manifest line); bash -n/parse() clean; actionlint findings identical to what revdep2.yaml already carries.
• The "Smoke test: stock R" failures on the first push were the GitHub outage (HTTP 503 from gh one second into the job); the follow-up push re-triggers CI cleanly.
• The first live run (most, depth 2) paid for itself in infrastructure findings, all fixed here: docker commit of the universe image filled the runner's single disk (there is no /mnt on ubuntu-26.04 runners) and killed the runner agent mid-copy — the commit step now measures the delta first and skips with a loud warning when the copy cannot fit; the dev binary silently vanished from its artifact because file.rename() across two bind mounts fails with EXDEV and its FALSE went unread — now a checked file.copy(); a resource sampler backgrounded in its own step stops reaching the job log once that step ends, so the long steps now run in-step samplers (a dying runner leaves streamed evidence); and the pak cache never saved because root-in-container wrote it — now chowned back. The shard-local fallback got a production validation for free: with no universe image, every shard built its own from the base image in ~12 minutes and carried on.
• The branch temporarily carries a push trigger with hardcoded inputs (most, depth 2) that drives the live test runs, because workflow_dispatch registration requires the file on the default branch; it will be reverted once the test run is green.

---

• By submitting this pull request, I assign the copyright of my contribution to The igraph development team.