fix(ci): make releases/latest wait for a complete asset set

[MichaelTaylor3d]

All required checks green, asserted by name via check-merge-preconditions.sh (exit 0); zero unresolved threads.

Risk note for the merging orchestrator: this changes the release path itself, and dig-node's midnight cron cuts a real stable release unattended, so the first cut after merge exercises the new promote job for real. It fails safe — if promote does not run, the release is published but not latest, and the previous complete release keeps serving installs. runbooks/release.md records the one-line manual re-promote.

Closes #335.

What actually happened

releases/latest was not missing its assets permanently. It was missing them for 5m03s, and the ticket measured inside that window.

moment	event
01:26:07Z	tag v0.145.0 pushed; both release.yml and package.yml runs created
01:47:47Z	package.yml attaches the 4 native packages
01:47:49Z	package.yml promotes the release to latest — with no binaries on it
01:52:52Z	release.yml attaches the 10 raw dig-node-* / dign-* binaries
01:53:00Z	release.yml completes

Between 01:47:49Z and 01:52:52Z, releases/latest was a release carrying four .deb/.pkg/.msi files and zero binaries. dig-installer resolves both the dig-node and dign stems through releases/latest (dig-installer/src/release.rs:187, stems at :59/:78), so every fresh install 404'd for that window. Nothing was red, because both workflows genuinely succeeded at the job each was given.

Cause: a stable release is assembled by two workflows that do not wait for each other, and both used softprops/action-gh-release, whose make_latest defaults to true. Attaching assets was promoting the release, and the promotion was won by whichever job finished first.

The ticket's defect 1 is falsified

The skipped "Nightly — native packages / Attach packages to the release" job in run 32679750245 is correct and by design, not the cause. That is the nightly channel's workflow_call into package.yml, whose publish job is if: github.ref_type == 'tag' — and a workflow_call inherits the caller's ref type, which is branch. The nightly channel publishes those artifacts through its own job. The stable path's attach ran normally in run 32679766273 (event=push, success).

Defect 2 is real and is what this PR is mostly about

"Stable — verify release assets" passed because its expected set was only the four native package names. It was written for dig-updater's feedsign; dig-installer is a second consumer nobody had told it about. It was satisfied at 01:47:47Z — two seconds before the incomplete release became latest.

The fix

• The expected set moves to .github/actions/check-release-assets and covers both consumers: 14 names — 4 native packages, plus dig-node-<v>-<platform> and dign-<v>-<platform> for all five platforms.
• Neither publisher promotes. release.yml and package.yml both set make_latest: false. release.yml gains a verify job (which polls until both publishers have landed, or fails) and a promote job that moves latest only after it passes. An incomplete release is therefore never latest; the previous complete release keeps serving installs, which is the correct failure mode.
• The guard is now falsifiable. A selftest job drives the same action over the exact four-package asset list releases/latest carried while installs were 404ing, and fails the build unless the guard rejects it.

Proof the fixed guard goes red

Both scripts extracted from the YAML and executed, so this is the real code path, not a paraphrase:

OLD guard, 4-package input:  "v0.145.0 carries all 4 native install packages"   exit=0
NEW guard, same input:       missing 10 of 14, all dig-node-*/dign-* binaries   exit=1
NEW guard, complete 14 set:  "carries all 14 consumer-resolvable assets"        exit=0

The failing case names exactly the ten assets #335 measured as 404.

And live, against real releases

The literal-list self-test could in principle drift from the live path, so the fixed guard was also dispatched against two real tags on this branch, reading their actual asset lists through gh release view:

run	tag	job	result
32683223388	v0.144.1 (complete, 14 assets)	Verify	success
32683228884	v0.30.0 (no dign-* at all, no arm64)	Verify	failure

The v0.30.0 failure names the seven real gaps: dig-node_0.30.0_arm64.deb, dig-node-0.30.0-linux-arm64, and all five dign-0.30.0-*. Control and abuse, on the same code path the release run uses.

The self-test job also passes on the runner (7s) on this PR, and package.yml's Attach packages to the release shows skipping here — independently confirming the ref-type gate is what skips it, not an event-name guard.

v0.145.0 today — fetched, not inferred

All 14 assets present. Each fetched through GitHub's 302 redirect with Content-Length matched against the API-declared size:

asset	HTTP	Content-Length
dig-node-0.145.0-linux-x64	200	74322512
dign-0.145.0-linux-x64	200	74322488
dig-node-0.145.0-windows-x64.exe	200	54050816
dign-0.145.0-macos-arm64	200	54048544
dig-node_0.145.0_arm64.deb	200	14619156

No repair action was needed — the assets self-completed at 01:52:52Z.

Blast radius

gitnexus impact is not applicable: this diff contains no source symbols. Radius established by grep instead — verify-release-assets.yml is referenced by nightly-release.yml:322 (unchanged, and it now inherits the widened check) and by runbooks/release.md; make_latest appears only in the two publishers changed here and in SPEC.md. All four call sites updated. No src/ file is touched.

Scope

.github/workflows/** + .github/actions/**, plus SPEC.md / runbooks/release.md / DEVELOPMENT_LOG.md coherence. No src/ change — PR #334 owns that. Version: smallest patch bump, 0.145.0 → 0.145.1, required by the version-increment gate; expect to rebase behind #334.

---

Round 2 — all three gate findings addressed (d717c25, 6cd9090)

Finding 1 — the race was narrowed, not closed. The poll read asset names; GitHub creates the asset row when an upload starts (state: "starting"), and verify is needs: publish, which orders it against release.yml's own upload only — package.yml has no ordering relationship to it. The query now selects .state == "uploaded". No sleep: it would shrink the window without closing it and could not be shown to fail. Verified against the live API (gh release view does expose state; the filtered query returns 14 on v0.144.1), then re-dispatched live — run 32686543775 @ v0.144.1 success, run 32686550399 @ v0.30.0 failure naming its seven real gaps.

Finding 2 — additive drift. New guard crates/dig-node-service/tests/release_asset_matrix_guard.rs, in the existing include_str! idiom. Asserts the PLATFORMS set equals the build matrix's out_name: set (equality both ways — a superset wedges every release, a subset is the silent hole), that every action-gh-release step declines promotion (count-matched), that exactly one --latest site exists and is in release.yml (--latest=false counted separately as a demotion), and that the state filter is present. Plus a non-triviality control, since two empty scrapes compare equal.

Each assertion was shown to fail on the defect it claims to catch, by mutating one workflow at a time — a new build platform, a deleted make_latest, a second --latest site, and a dropped state filter each redden exactly one test.

That last mutation caught a vacuity in the guard itself: the first version searched the whole YAML and passed with the filter deleted, because the comment explaining the filter contains the same text. It now asserts against the gh release view command line specifically.

Finding 3 — one line. release.yml's header no longer claims the release is marked latest on publish.

All five required checks green, asserted by name (Lint commit messages had a CANCELLED run in the rollup; the gate script reads branch protection and the later SUCCESS). Three review threads resolved. Held DRAFT pending re-gate.

[MichaelTaylor3d]

CHANGES-REQUIRED

Head reviewed: d8f2962 (resolved from the remote, not from the dispatch prompt). Correctness gate, fresh context, read-only — no shared checkout was mutated; every read went through git show <ref>:<path> and gh.

GitHub 422s a verdict-bearing review event from the PR author identity, so this is recorded as a comment review. The three open inline threads are what bars the merge, via required_conversation_resolution.

Verdict in one line

The diagnosis is right, the shape of the fix is right, and the regression proof is real — but the race is narrowed, not closed (a still-uploading asset satisfies the poll), and the PR adds a hand-maintained cross-repo asset table with no mechanical drift guard, in a repo that already has the idiom for one.

Findings, ranked

1. check-release-assets/action.yml:120 — the poll reads .assets[].name without filtering state == "uploaded", so promote can move latest onto a release whose package.yml assets are mid-upload. One jq filter. GATING.
2. check-release-assets/action.yml:90 — the 14-name expectation is a third copy of a producer contract with no mechanical guard. Additive drift fails silently, and the selftest cannot see it because it matches its own needle. Executable fix in the existing include_str! workflow-shape-test idiom. GATING.
3. release.yml:5 — the file header still states the release is marked latest at publish. A superseded rule surviving in the same file. GATING, one line.

The five verification points

1. Is the race closed? At the promotion level, yes, and this was enumerated rather than sampled. Exactly two softprops/action-gh-release call sites exist repo-wide — release.yml:81 and package.yml:333 — and both now carry make_latest: "false". The only other release-writing path is nightly-release.yml:499-500, which passes --latest=false on both the create and the edit, and which is itself already guarded by nightly_release_workflow_shape.rs:211-216. Exactly one promotion site exists, release.yml:138. The workflow_call path into package.yml inherits the caller ref type and never reaches its publish job. No implicit promoter survives. At the completeness level, finding 1 leaves a residual window.

2. Can verify pass early? Not on a partial NAME set. The check is grep -qxF per expected name across all 14, so it is exact-match over the complete set, never a count and never an any-present test. Timeout is 75 minutes on the workflow_call path (5 on dispatch); on timeout it writes the missing list to the step summary, emits an error annotation and exits 1, so verify fails and promote is skipped. It can pass early only in the asset-state sense of finding 1.

3. Is the failure mode as claimed? Yes. With both publishers at make_latest: false, a release that never verifies is created but never promoted, so releases/latest keeps naming the previous complete release, and the unpromoted release stays visible for diagnosis. The runbook one-liner gh release edit vX.Y.Z --repo DIG-Network/dig-node --latest is the identical command promote runs, so it works as written. The runbook also documents the new sharp edge this creates — an out-of-band package.yml re-run now DEMOTES an already-latest release — which is the non-obvious consequence most likely to have been missed.

4. Is the expected set right for BOTH consumers? Yes, verified against producers and consumers rather than against the PR body. Producers: the two build-binaries.yml matrices emit exactly linux-x64, linux-arm64, macos-x64, macos-arm64, windows-x64.exe for both the dig-node and dign stems — an exact match for PLATFORMS, both arm64 variants included; the stable package.yml matrix builds both the amd64 and arm64 debs plus the universal .pkg and the .msi. Consumer: dig-installer/src/release.rs resolves both the dig-node and the dign stems, taking dign deliberately from the dig-node release rather than dig-app, through latest_release_api(), which is releases/latest. The strongest single piece of evidence is the live control run against a real complete release, below. Requiring arm64.deb is stricter than feedsign fails-closed, and the file justifies that explicitly against the arm64 floor tickets — correct, and worth not relaxing. The gap is the drift guard: finding 2.

5. Regression proof — re-verified independently, not taken from the PR body.

run	tag	trigger	Verify job	evidence read from the log
32683223388	v0.144.1	workflow_dispatch on this branch	success in 427ms	v0.144.1 carries all 14 consumer-resolvable assets
32683228884	v0.30.0	workflow_dispatch on this branch	failure after polling to timeout	release v0.30.0 is missing 7 of 14 ..., naming dig-node_0.30.0_arm64.deb, dig-node-0.30.0-linux-arm64 and all five dign-0.30.0-*

Both ran at head d8f2962, and both read a real release asset list over the network — ASSETS_OVERRIDE is empty in the passing run — so this is control-and-abuse on the live code path, not synthetic input. The failing run polled twice before failing, which also exercises the poll-then-fail path rather than a single sample. The selftest job passed on this PR as well, so the literal-list battery and the live battery agree.

The falsified defect-1 diagnosis

Correctly falsified, and it does not hide a second defect. package.yml gates its publish job on if: github.ref_type == 'tag', and the skipped occurrence was the nightly channel calling package.yml via workflow_call, where the ref type is the one belonging to the CALLER, namely branch. The nightly channel publishes those same artifacts through its own job, which uses gh release create/edit ... --latest=false and uploads the flattened package set at nightly-release.yml:439-458 — amd64 deb, macos pkg, windows msi, deliberately no arm64 deb on nightly since feedsign has one Linux entry. Nothing is dropped on either channel. Corroborated on this very PR: Attach packages to the release shows SKIPPED here for the same ref-type reason, on a run whose ref type is branch. It is a ref-type gate, not an event-name guard, and #335 should close on this PR.

Also confirmed

• Version 0.145.0 to 0.145.1, with Cargo.lock updated in the same commit (one line each), so --locked cannot redden.
• Scope claim holds. Nine files: .github/actions/check-release-assets/action.yml, the three workflows, Cargo.toml, Cargo.lock, SPEC.md, runbooks/release.md, DEVELOPMENT_LOG.md. No src/ and no crates/ file is touched, so there is no collision with PR #334.
• SPEC coherence is good. 11.1 now states that promotion is separate and gated, and enumerates the fourteen names normatively; the new 11.1b states the make_latest: false requirement on BOTH publishers plus the falsifiability requirement on the guard. runbooks/release.md agrees. The only place the old rule survives is finding 3.
• dig-constants check, both directions. Nothing in this diff belongs in dig-constants, and nothing here should be consuming it: the shared values are release-asset FILE NAMES, produced by GitHub Actions YAML in this repo and consumed by Rust in two others, and a Rust constants crate cannot be read by a workflow. Finding 2 is the right centralization for this particular contract.
• No UI surface, and no custody, key, spend or crypto path. promote takes contents: write scoped to that job alone, and the guard workflow declares contents: read; no least-privilege regression.

Merge preconditions at this head

Do not merge on this state. Beyond the three open threads, Lint commit messages on run 32684533232 is CANCELLED in the rollup, with a later run 32684557257 SUCCESS for the same context. Assert the required set by name with .claude/scripts/check-merge-preconditions.sh --repo DIG-Network/dig-node --pr 336 and gate on its exit code, rather than reading the rollup.

Risk note for the merging orchestrator

nightly-release.yml:80 includes github.event_name == 'schedule' in the stable job condition, so the midnight-UTC cron cuts a real stable release unattended, and the FIRST exercise of promote will have no human present. Finding 1 decides whether that first unattended cut can publish an incomplete latest, so it should land before the next midnight.

[MichaelTaylor3d]

Independent re-gate of 6cd9090 — interim result 1/4 (finding 1, the state race).

Reproduced the mutation that matters most, in my own worktree, not the lane's:

• Deleted select(.state == "uploaded") from the gh release view command line only, leaving the explanatory comment block above it fully intact (grep confirms the string still occurs once in the file, in prose).
• cargo test -p dig-node-service --test release_asset_matrix_guard → FAILED, the_guard_counts_only_fully_uploaded_assets, and the panic message quotes the mutated query line back.

That is the distinguishing test: the earlier version of this assertion passed with the filter deleted because it scanned the whole file and matched its own documentation. The current one anchors on the line containing gh release view, so the prose cannot satisfy it.

Also confirmed the filter is against a real field rather than a plausible-looking one — gh release view v0.144.1 --json assets exposes state on every asset (all uploaded on a settled release), so the filter selects on something GitHub actually reports rather than silently matching nothing. A select on a non-existent key would have yielded an empty set and wedged every release loudly, but it is worth having measured rather than assumed.

Baseline before mutation: all 5 guard tests green at 6cd9090.

Remaining mutations (matrix drift, make_latest deletion, second promotion site) in progress; results follow.

[MichaelTaylor3d]

Interim result 2/4 — finding 2 (additive matrix drift), independently reproduced.

Mutation: added a single out_name: freebsd-x64 line to the build-binaries.yml matrix and changed nothing else — i.e. the build gains a platform and the hand-maintained PLATFORMS=(…) expectation does not. That is the silent direction, the one the workflow self-test structurally cannot see because it drives the action over literal lists written from the same expectation.

Result: FAILED, the_expected_platforms_are_exactly_the_platforms_the_build_produces, printing both sets:

  expected by the guard: ["linux-arm64", "linux-x64", "macos-arm64", "macos-x64", "windows-x64.exe"]
  produced by the build: ["freebsd-x64", "linux-arm64", ...]

Two properties I checked beyond "an assertion exists":

• It is stated as set equality, so it catches drift in both directions rather than only the loud one.
• The out_name: scrape is deduped, which matters because build-binaries.yml repeats those tokens in the glibc-verification matrix (lines 253-268). Repetition is therefore invisible while a genuinely new platform is not — the scrape distinguishes the two cases rather than getting lucky. The the_platform_sets_are_non_trivial control guards the vacuity case where a broken scrape returns two empty sets that compare equal.

Interim results 3/4 and 4/4 — the promotion guards.

• Deleted make_latest: "false" from package.yml → FAILED, no_asset_upload_may_promote_the_release_to_latest. This is the exact P0: releases/latest (v0.145.0) is missing every raw binary and every dign-* CLI asset — fresh installs 404 #335 regression: action-gh-release defaults make_latest to true, so an omitted setting is not a neutral omission.
• Appended a second gh release edit "$TAG" --latest site → FAILED, exactly_one_site_promotes_a_release_to_latest, reporting Found 2.

Control against a false positive, which I added and the lane did not report: appended gh release edit "$TAG" --latest=false — a legitimate demotion, the nightly channel's correct behaviour — and the suite stayed green, 5 passed. So the single-site assertion discriminates promotion from demotion rather than substring-matching --latest and rejecting correct code. Both arms of that distinction are now measured.

Every mutation was grep-confirmed applied before the run (a cargo fmt or a silently-failed edit would otherwise produce a green that means nothing), and the worktree was restored to a clean tree after each.

[MichaelTaylor3d]

PASS — independent scoped re-gate

Head reviewed: 6cd909068fe4a3571fa3d3796ba362e4e42908c2 (resolved from the remote myself; the head did not move during this pass). Delta gated: d8f2962..6cd9090 — 3 files, +228/-6.

All three findings from the prior round are genuinely fixed, verified by mutation in my own worktree rather than by reading the assertions. The lane resolved its own gate's threads, so every claim below was re-measured independently; the four mutations the lane reported are reproduced, plus one control it did not run.

Finding 1 — the state race is CLOSED, not narrowed

.github/actions/check-release-assets/action.yml:135 now reads
--jq '.assets[] | select(.state == "uploaded") | .name'.

• Not a sleep: grep confirms no sleep anywhere in the action, and the guard test asserts its absence explicitly.
• Filters on a real field — gh release view v0.144.1 --json assets exposes state on every asset. A select on a non-existent key would yield the empty set, so this needed measuring rather than assuming.
• Applied at the right stage: it is inside the poll's own read, so missing=() is computed from fully-uploaded names only. There is no count-based or any-present shortcut downstream.
• Revert-proof: deleting the filter from the command line while leaving the explanatory comment intact turns the suite RED. That is the mutation that matters, because the first version of this assertion scanned the whole file and was satisfied by its own documentation.

Finding 2 — additive drift now fails loudly

crates/dig-node-service/tests/release_asset_matrix_guard.rs, in the repo's existing include_str! idiom beside release_workflow_dign_guard.rs and nightly_release_workflow_shape.rs.

Divergence test I ran: added out_name: freebsd-x64 to the build matrix and nothing else. RED, printing both sets. Stated as set equality so it catches both directions; the out_name: scrape is deduped, which it must be because those tokens repeat in the glibc-verification matrix — so repetition is invisible while a new platform is not. the_platform_sets_are_non_trivial covers the vacuity case where a broken scrape compares two empty sets equal.

Promotion guards, also reproduced: deleting make_latest: "false" from package.yml → RED; adding a second gh release edit --latest → RED (Found 2).

Control the lane did not report: appending a legitimate gh release edit --latest=false demotion leaves the suite green. So the single-site guard discriminates promotion from demotion instead of substring-matching --latest and rejecting correct code. Both arms measured.

Every mutation was grep-confirmed applied before its run, and the tree restored clean after each.

Finding 3 — header matches behaviour

release.yml:1-13 no longer claims the upload marks the release latest; it now states promotion is the promote job at the bottom of the file, after the asset guard. Matches the code.

Delta scope, version, checks

• Version 0.145.0 → 0.145.1; Cargo.lock carries dig-node-service 0.145.1. Consistent — --locked will not redden.
• Scope holds: .github/**, Cargo.toml/Cargo.lock/SPEC.md/DEVELOPMENT_LOG.md/runbooks/release.md, plus the one new test file. No src/. The only crates/ path is a NEW test in dig-node-service; PR#334's live lane is entirely in dig-node-core/src/**, so there is no file overlap (both touch Cargo.toml/Cargo.lock/SPEC.md, an ordinary rebase, not a collision).
• The delta touched no publisher and added no --latest site, so the prior round's enumeration of promotion sites stands and did not need redoing.
• Required checks asserted BY NAME via check-merge-preconditions.sh: Lint commit messages, Check version increment, Rustfmt, Clippy, Test + coverage — all present and SUCCESS. unresolvedReviewThreads=0. The script's BLOCKED is solely draft=true, which is the correct §2.4a posture until this verdict returned.

One NON-GATING observation (I am resolving this myself; it does not bar merge)

SPEC §11.1b defines the promotion contract and its falsifiability clause, but "verified complete" is not defined to include upload state — it requires only that the self-test fail an asset list carrying just the native packages. The state == "uploaded" filter is the load-bearing half of finding 1, and a reimplementer building from §11.1b alone would count names and ship the narrowed race.

Suggested one-sentence addition while the PR is still draft (cheap, and SPEC.md is already in this PR): state that an asset counts toward completeness only when GitHub reports it uploaded, because the asset row exists from the moment the upload begins.

Deliberately non-gating: it is an omission rather than a contradiction, the operational risk of merging without it is nil, and the tonight midnight-UTC cron cuts a real stable release unattended — holding a correct fix for a doc sentence would leave that cut unguarded, which is strictly worse.