P0: releases/latest (v0.145.0) is missing every raw binary and every dign-* CLI asset — fresh installs 404

[MichaelTaylor3d]

Task

releases/latest for dig-node is v0.145.0, and that release is missing 10 of the 14 assets
v0.144.1 shipped — including every raw binary and every dign-* CLI asset. dig-installer resolves
both dig-node and dign through releases/latest, so a fresh install is broken right now.

Measured 2026-08-24

asset on v0.145.0	result
dig-node-0.145.0-linux-x64	404
dign-0.145.0-linux-x64	404
dig-node-0.145.0-windows-x64.exe	404
control: dign-0.144.1-linux-x64 on v0.144.1	200

v0.145.0 carries only the three native installers plus a second .deb:
dig-node-0.145.0-macos.pkg, dig-node-0.145.0-windows-x64.msi, dig-node_0.145.0_amd64.deb,
dig-node_0.145.0_arm64.deb. v0.144.1 has 14 assets including dig-node-0.144.1-{linux-arm64, linux-x64, macos-arm64, macos-x64, windows-x64.exe} and dign-0.144.1-* for all five platforms.

dig-installer/src/release.rs:187 resolves https://api.github.com/repos/{}/{}/releases/latest, and
:59 / :78 declare the dig-node and dign stems against that repo — so the installer asks for
exactly the assets that now 404.

Two defects, and the second is worse than the first

1. The assets were never attached. Job "Nightly — native packages / Attach packages to the
release" reports skipped in run 32679750245. Establish why it was skipped on a schedule event
when the stable job it feeds did run — a guard that assumes stable is dispatch-only would do it, and
this repo's stable job does run on schedule (nightly-release.yml:80 includes
github.event_name == 'schedule').

2. The gate that exists to catch this PASSED. Job "Stable — verify release assets / Verify
v0.145.0 carries the …" reported success on a release missing every raw binary and every CLI. That
is the more dangerous defect: the verification is either checking only the native packages, or checking
build artifacts rather than release assets. A watcher reading the run also called it GREEN, citing
build logs showing dign-0.145.0-* built for five platforms — built is not attached.

Fix the verifier so it enumerates the actual release assets and fails when the set is smaller than
the previous release's. A verify job that cannot fail on a missing asset is not a gate.

Do

1. Restore the assets on v0.145.0 — re-run the attach step, or attach them, so releases/latest
   serves a complete set. This is the user-facing half and it comes first.
2. Fix the skipped attach step so a scheduled stable cut attaches everything a dispatched one does.
3. Fix the verify job so it compares the release's real asset list against the expected platform
   matrix, and prove it fails on an incomplete release before calling it fixed.
4. Confirm by fetching each asset through GitHub's redirect and matching Content-Length — never by
   a run conclusion, and never from build logs.

Scope + coordination

A live dig-node lane owns src/ on PR#334. You own .github/workflows/** only — do not touch
src/, and rebase before merging. If a version bump is required by the version-increment gate, take the
smallest patch bump and say so, expecting to rebase.

Rules

• Branch, stub commit + push + draft PR first. Gate with
  .claude/scripts/check-merge-preconditions.sh --repo DIG-Network/dig-node --pr N --undraft-on-success
  on the EXIT CODE.
• dig-node's midnight cron cuts a real stable release unattended, so whatever lands on main reaches
  users at 00:00Z. The merge gate is the last line of defence here, not the first of two.
• Never set user.name/user.email at any scope. Never hand-push a tag.
• A red publish job can be an intentional guard — read what it asserts before changing it.

Parent: https://github.com/DIG-Network/dig_ecosystem/issues/1701 (needs dign export-seed reachable
from a published binary) and the installer path via
DIG-Network/dig-installer#73.

[MichaelTaylor3d]

Root cause: releases/latest moved 5m03s before the release was complete

The assets were not "never attached". They were attached at 01:52:52Z, and releases/latest had already flipped to v0.145.0 at 01:47:49Z. The measurement in this ticket was taken inside that window.

time (2026-08-24)	event
01:26:07Z	tag v0.145.0 pushed; both release.yml (32679766399) and package.yml (32679766273) runs created by the push
01:47:47Z	package.yml attaches the 4 native packages
01:47:49Z	package.yml promotes the release to latest — published_at, with no binaries on it
01:52:52Z	release.yml attaches the 10 raw dig-node-* / dign-* binaries
01:53:00Z	release.yml completes; both runs success

Cause: a stable release is assembled by two workflows that neither know about nor wait for each other, and both used softprops/action-gh-release, whose make_latest defaults to true. So attaching assets was also promoting the release, and the promotion was won by whichever job finished first. On this release that was the packages job, five minutes ahead of the binaries.

Nothing was red anywhere, because both workflows genuinely succeeded at the job each was given.

Defect 1 as stated is falsified — the skipped attach is correct

The skipped "Nightly — native packages / Attach packages to the release" job in run 32679750245 is by design and is not the cause. That run is the nightly channel's workflow_call into package.yml; its publish job is if: github.ref_type == 'tag', and a workflow_call inherits the caller's ref type, which is branch. The nightly channel publishes those artifacts through its own publish job instead. package.yml:317-319 already says so.

The stable path's attach is a separate run — 32679766273, event=push, on the tag — and it ran and succeeded. The hypothesis that a guard assumed stable is dispatch-only does not hold: the guard is on ref type, not on event name.

Defect 2 is real, and it is the one worth fixing

"Stable — verify release assets" passed because its EXPECTED list was exactly four names — amd64.deb, arm64.deb, macos.pkg, windows-x64.msi. It was written for dig-updater's feedsign (dig_ecosystem#2290), which resolves dig-node by native-package name. dig-installer is a second consumer nobody had told the guard about: it resolves the raw dig-node and dign stems through releases/latest (dig-installer/src/release.rs:187, stems at :59/:78).

So the guard was satisfied at 01:47:47Z — two seconds before the incomplete release became latest — and was structurally incapable of noticing the ten missing binaries. It was not checking build artifacts; it was checking real release assets, but only a quarter of them.

Current state of v0.145.0 — fetched, not inferred

All 14 assets are present now (they self-completed at 01:52:52Z; no repair was needed). Each fetched through GitHub's 302 redirect, Content-Length matched against the API-declared size:

asset	HTTP	Content-Length
dig-node-0.145.0-linux-x64	200	74322512
dign-0.145.0-linux-x64	200	74322488
dig-node-0.145.0-windows-x64.exe	200	54050816
dign-0.145.0-macos-arm64	200	54048544
dig-node_0.145.0_arm64.deb	200	14619156

A sweep of every stable release shows the same shape: every release from v0.67.0 on ends complete at 14 assets. The incompleteness has always been transient — which is exactly why it was never caught, and why the window is the defect.

Fix — PR #336

• The expected set moves to .github/actions/check-release-assets and covers both consumers: 14 names.
• Neither publisher promotes. Both set make_latest: false; release.yml gains a verify job (polls until both publishers land, or fails) and a promote job that moves latest only after it passes. An incomplete release is never latest, so the previous complete release keeps serving installs.
• The guard is made falsifiable: a self-test drives the same action over the exact four-package list releases/latest carried while installs were 404ing, and reddens the build unless the guard rejects it.

Proof, from the scripts extracted out of the YAML and executed (real code path, not a paraphrase):

OLD guard, 4-package input:  "carries all 4 native install packages"          exit=0
NEW guard, same input:       missing 10 of 14, every dig-node-*/dign- binary  exit=1
NEW guard, complete 14 set:  "carries all 14 consumer-resolvable assets"      exit=0

The self-test also passes on the real runner (7s) on PR #336.

[MichaelTaylor3d]

Correcting this ticket: my stated defect 1 is falsified, and the outage self-healed before the fix was written

I attributed the missing assets to the skipped "Attach packages to the release" job in run
32679750245, hypothesising a guard that assumes stable is dispatch-only. That is wrong. That job is
the nightly channel's workflow_call into package.yml, whose publish job is
if: github.ref_type == 'tag' — and a workflow_call inherits the caller's ref type (branch). It
is a ref-type gate, not an event-name guard, and it skips on a pull_request too. Independently
confirmed on the fix PR itself.

The real cause: a promotion race between two workflows that do not wait for each other

time (2026-08-24)	event
01:26:07Z	tag v0.145.0 pushed; both release.yml (32679766399) and package.yml (32679766273) start
01:47:47Z	package.yml attaches the 4 native packages
01:47:49Z	package.yml promotes to latest — with no binaries on the release
01:52:52Z	release.yml attaches the 10 raw dig-node-* / dign-* binaries

Both publishers used softprops/action-gh-release, whose make_latest defaults to true. So
attaching was promoting, and promotion went to whichever workflow finished first.

The user-facing outage was real but lasted 5m03s and self-healed at 01:52:52Z — before the fix was
even written. All 14 assets are now present on v0.145.0 and were fetched through the 302 with
Content-Length matched (dig-node-0.145.0-linux-x64 74,322,512 · dign-0.145.0-linux-x64 74,322,488 ·
dig-node-0.145.0-windows-x64.exe 54,050,816 · dign-0.145.0-macos-arm64 54,048,544 ·
dig-node_0.145.0_arm64.deb 14,619,156). No repair was needed. This ticket is about preventing the
recurrence, not restoring an outage.

Defect 2 stands, and its cause is more interesting than "checking the wrong thing"

The verifier was checking real release assets — but only four names (amd64.deb, arm64.deb,
macos.pkg, windows-x64.msi). It was written for dig-updater's feedsign, and dig-installer was a
second consumer nobody had told it about. It passed at 01:47:47Z, two seconds before the incomplete
release became latest, and it was right about everything it knew to check.

That is the same shape as the ever_funded and dig-keystore findings from tonight: a check that is
correct within a scope narrower than the thing it is trusted to guarantee.

The fix — #336

Neither publisher promotes (make_latest: false on both); release.yml gains a verify job that
polls until both publishers have landed, and a promote job that moves latest only after verify
passes. It fails safe: an incomplete release never becomes latest, and the previous complete
release keeps serving installs.

The expected set moves to .github/actions/check-release-assets/action.yml and covers both consumers
— 14 names — with a selftest job so the guard is falsifiable.

Proven red for the right reason, and against real releases rather than only synthetic input:

OLD guard, 4-package input:  "carries all 4 native install packages"   exit=0
NEW guard, same input:       missing 10 of 14                          exit=1
NEW guard, complete 14 set:                                            exit=0

live: run 32683223388 at v0.144.1 -> success
      run 32683228884 at v0.30.0  -> failure, naming its seven real gaps

Under an independent correctness gate now, because it re-shapes who owns latest in a repo whose
midnight cron cuts a stable release unattended — so the next cut exercises promote for real, with
nobody watching.