diff --git a/.github/actions/check-release-assets/action.yml b/.github/actions/check-release-assets/action.yml new file mode 100644 index 00000000..a736928e --- /dev/null +++ b/.github/actions/check-release-assets/action.yml @@ -0,0 +1,170 @@ +# THE EXPECTED RELEASE-ASSET MATRIX, and the only place that knows it (dig-node#335). +# +# A `vX.Y.Z` release of dig-node has TWO independent consumers, and until #335 the release gate +# only knew about one of them: +# +# * dig-updater's feedsign resolves dig-node by the NATIVE INSTALL PACKAGE names — it hands a +# package to msiexec/installer/dpkg and never places a bare binary — and it fails CLOSED on the +# whole signed manifest when one component cannot be resolved (dig_ecosystem#2290). +# * dig-installer resolves BOTH `dig-node` and `dign` as RAW BINARIES through `releases/latest` +# (`dig-installer/src/release.rs:187`, stems at `:59` / `:78`). A release that is `latest` +# without those binaries makes every fresh install 404. +# +# The two sets are produced by two DIFFERENT workflows that finish at different times — +# `package.yml` attaches the packages, `release.yml` attaches the binaries — so "the release +# exists" and "the release is usable" are not the same statement. This action expresses the second +# one, over both sets at once. +# +# It is a composite action rather than an inline `run:` block so that the SAME code path can be +# driven from a literal asset list. That is what makes the guard falsifiable: the self-test in +# `verify-release-assets.yml` feeds it a deliberately incomplete list on every PR and requires it +# to fail. A guard that has never been observed going red is a hope, not a gate. +name: Check release assets +description: >- + Assert a dig-node release carries every asset its two consumers resolve — the native install + packages (dig-updater feedsign) and the raw dig-node/dign binaries (dig-installer). Polls, + because the packages and the binaries are attached by separate workflows. + +inputs: + tag: + description: "The release tag to check (e.g. v0.145.0)." + required: true + repo: + description: "OWNER/NAME of the repository holding the release." + required: false + default: ${{ github.repository }} + timeout_minutes: + description: "How long to wait for the assets to appear before failing." + required: false + default: "75" + assets: + description: >- + A newline-separated literal asset list to check INSTEAD of querying the release. For the + self-test only — when set, the action makes exactly one pass and never polls, so a + deliberately incomplete list fails immediately rather than burning the timeout. + required: false + default: "" + github_token: + description: "Token used to read the release. Unused when `assets` is supplied." + required: false + default: "" + +runs: + using: composite + steps: + - name: Assert every consumer-resolvable asset is present + shell: bash + env: + GH_TOKEN: ${{ inputs.github_token }} + REPO: ${{ inputs.repo }} + TAG: ${{ inputs.tag }} + TIMEOUT_MINUTES: ${{ inputs.timeout_minutes }} + ASSETS_OVERRIDE: ${{ inputs.assets }} + run: | + set -euo pipefail + + # The version as it appears in asset names: the tag without its leading `v`. + VERSION="${TAG#v}" + + # Kept as a literal list rather than derived from a glob: a glob would happily accept a + # `.deb` for the wrong arch or a stale version and call the release complete, which is the + # failure this guard exists to catch. + # + # THIS IS ONE COPY OF A CROSS-REPO CONTRACT. Producers: `package.yml` (packages) and + # `build-binaries.yml` (binaries), both in this repo. Consumers: + # `dig-updater/crates/dig-updater-feedsign/src/resolve.rs` (`asset_name_parts`) and + # `dig-installer/src/release.rs`. Verifier: here. Nothing enforces that they agree, so they + # are held together by `SYSTEM.md` (dig-updater section, "dig-node release-asset file + # names") and the `canonical` skill. Change one, change all of them. + # + # macOS contributes ONE PACKAGE name, not two: the `.pkg` is universal and carries no arch + # token, so `macos/arm64` and `macos/x64` both resolve to it. The raw binaries below are + # per-arch and do carry the token. + # + # `arm64.deb` is required DELIBERATELY, and this is stricter than feedsign's own failure + # condition. feedsign fails closed only when a component resolves ZERO assets, so a release + # missing just `arm64.deb` would still publish — silently dropping linux/arm64 hosts from + # auto-update rather than reddening anything. That silent drop is exactly the arm64 + # platform floor (dig_ecosystem#1741/#1736/#2126), so the stable channel treats a missing + # arm64 package as a failed release. Do not relax this to match feedsign. + PLATFORMS=(linux-arm64 linux-x64 macos-arm64 macos-x64 windows-x64.exe) + + EXPECTED=( + # Native install packages — dig-updater feedsign. + "dig-node_${VERSION}_amd64.deb" + "dig-node_${VERSION}_arm64.deb" + "dig-node-${VERSION}-macos.pkg" + "dig-node-${VERSION}-windows-x64.msi" + ) + # Raw binaries — dig-installer, which resolves the `dig-node` and `dign` stems separately + # and 404s on either one being absent. `dign` is not an optional extra: it is the CLI, and + # an install that lands the daemon without it leaves a node with no command + # (dig_ecosystem#857). + for p in "${PLATFORMS[@]}"; do + EXPECTED+=("dig-node-${VERSION}-${p}") + EXPECTED+=("dign-${VERSION}-${p}") + done + + deadline=$(( $(date +%s) + TIMEOUT_MINUTES * 60 )) + attempt=0 + + while :; do + attempt=$(( attempt + 1 )) + + if [ -n "$ASSETS_OVERRIDE" ]; then + # Self-test mode: one pass over a literal list, no polling and no network. + assets="$ASSETS_OVERRIDE" + else + # A missing release is a legitimate "not yet" while the release workflow is still + # running, so it is treated the same as a missing asset rather than aborting early. + # + # `select(.state == "uploaded")` IS LOAD-BEARING. GitHub creates the asset row when an + # upload STARTS, in state `starting`, so a name becomes visible before its bytes are. + # Only `uploaded` means the file can actually be downloaded. + # + # Without it this guard would NARROW the dig-node#335 race rather than close it. The + # calling job is `needs: publish`, which orders it after release.yml's OWN upload — + # but package.yml is a separate workflow with no ordering relationship to it at all. + # So the poll could observe all fourteen names while a `.msi` or `.pkg` was still + # uploading, and `promote` would move `latest` onto a release whose download is + # incomplete: the same observable failure as #335, through a shorter window. + # + # A `sleep` is NOT an acceptable substitute. It would shrink the window without + # removing it and could not be shown to fail, so the release would LOOK guarded while + # still racing. Ask for the state; do not wait and hope. + assets="$(gh release view "$TAG" --repo "$REPO" --json assets --jq '.assets[] | select(.state == "uploaded") | .name' 2>/dev/null || true)" + fi + + missing=() + for name in "${EXPECTED[@]}"; do + printf '%s\n' "$assets" | grep -qxF "$name" || missing+=("$name") + done + + if [ ${#missing[@]} -eq 0 ]; then + echo "$TAG carries all ${#EXPECTED[@]} consumer-resolvable assets — feedsign can resolve dig-node, and dig-installer can fetch dig-node + dign." + { + echo "### Release assets verified — \`$TAG\`" + echo + for name in "${EXPECTED[@]}"; do echo "- \`$name\`"; done + } >> "$GITHUB_STEP_SUMMARY" + exit 0 + fi + + if [ -n "$ASSETS_OVERRIDE" ] || [ "$(date +%s)" -ge "$deadline" ]; then + { + echo "### Release assets MISSING — \`$TAG\`" + echo + echo "This release is NOT shippable. dig-updater feedsign fails closed on the whole" + echo "STABLE signed feed when a native package is absent, and dig-installer 404s on a" + echo "fresh install when a raw \`dig-node\`/\`dign\` binary is absent." + echo + echo "Missing:" + for name in "${missing[@]}"; do echo "- \`$name\`"; done + } >> "$GITHUB_STEP_SUMMARY" + echo "::error::release $TAG is missing ${#missing[@]} of ${#EXPECTED[@]} consumer-resolvable asset(s): ${missing[*]}. Attach them (dispatch package.yml and/or release.yml against the $TAG ref) before this release is allowed to stand as latest." + exit 1 + fi + + echo "attempt $attempt: still missing ${#missing[@]} of ${#EXPECTED[@]} (${missing[*]}); retrying…" + sleep 30 + done diff --git a/.github/workflows/package.yml b/.github/workflows/package.yml index 649c0b8f..75e36eaa 100644 --- a/.github/workflows/package.yml +++ b/.github/workflows/package.yml @@ -336,4 +336,10 @@ jobs: # Do NOT regenerate notes — release.yml (the binary release) owns the notes; this job # only appends the native-package assets to the same tag's release. generate_release_notes: false + # Nor does it own `latest` (dig-node#335). This job attaches FOUR of a stable release's + # fourteen assets; softprops promotes to `latest` by default, so on v0.145.0 this job + # made a release with no binaries at all the one dig-installer fetched from, five + # minutes before release.yml attached them. Promotion belongs to release.yml's + # `promote` job, which runs after the asset guard has confirmed BOTH publishers landed. + make_latest: "false" fail_on_unmatched_files: true diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 09ed8874..a750fa1c 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -1,11 +1,13 @@ # STABLE binary release. On a `vX.Y.Z` tag (cut by the nightly-release orchestrator's stable job # — nightly-release.yml — either from the midnight cron detecting a version bump or from a manual # `workflow_dispatch`), this workflow builds the `dig-node` service binary + its `dign` alias for -# every OS/arch (via the reusable build workflow) and publishes them to a STABLE GitHub Release: -# `prerelease: false`, marked `latest`. Every per-OS/arch binary is published under the canonical -# `dig-node-*` name (+ the `dign-*` alias) — SPEC §11.2; the duplicate legacy `dig-companion-*` -# copy was dropped in #585. The changelog is already -# inside the tag (the orchestrator committed it before tagging), so the notes carry the changelog. +# every OS/arch (via the reusable build workflow) and publishes them to a STABLE GitHub Release +# with `prerelease: false`. It is then marked `latest` by the `promote` job at the bottom of this +# file — never by the upload itself — and only after the asset guard has confirmed the release +# carries everything both of its consumers resolve (dig-node#335). Every per-OS/arch binary is +# published under the canonical `dig-node-*` name (+ the `dign-*` alias) — SPEC §11.2; the +# duplicate legacy `dig-companion-*` copy was dropped in #585. The changelog is already inside the +# tag (the orchestrator committed it before tagging), so the notes carry the changelog. # # This is intentionally tag-ONLY: merges to main no longer build or release here (dig_ecosystem # #590 batches releases to the nightly cron + manual dispatch). Pre-merge coverage comes from @@ -80,11 +82,60 @@ jobs: - name: Create / update the STABLE release and attach binaries uses: softprops/action-gh-release@v2 with: - # `prerelease: false` + `make_latest: true`: a stable release is the one that moves - # `latest`. Nightlies (nightly-release.yml) are always prerelease + never latest, so a - # nightly can never masquerade as this stable download. + # `prerelease: false` marks this a stable release. `make_latest: false` is DELIBERATE + # and is the dig-node#335 fix: attaching assets must not be what moves `latest`. + # + # A stable release is assembled by TWO workflows. This one attaches the binaries; + # package.yml attaches the native install packages. Whichever finished first used to + # promote the half-built release, and on v0.145.0 that was package.yml at 01:47:49Z — + # five minutes and three seconds before the binaries landed at 01:52:52Z. For that + # window `releases/latest` was a release with no `dig-node-*` or `dign-*` binary at all, + # and dig-installer resolves both stems through `releases/latest`, so every fresh + # install 404'd. + # + # Promotion now happens in the `promote` job below, gated on the asset guard, so + # `latest` can only ever name a release a user can actually install from. Nightlies + # (nightly-release.yml) remain prerelease + never latest, so a nightly still cannot + # masquerade as this stable download. prerelease: false - make_latest: "true" + make_latest: "false" files: release/* generate_release_notes: true fail_on_unmatched_files: true + + # The release is complete only when BOTH publishers have finished, and this is the only job that + # knows when that is: the guard polls the published asset list until it holds every asset + # dig-updater feedsign and dig-installer resolve, or fails. + verify: + name: Verify the release is complete + needs: publish + if: github.ref_type == 'tag' + uses: ./.github/workflows/verify-release-assets.yml + with: + tag: ${{ github.ref_name }} + + # PROMOTION IS THE LAST STEP, NOT A SIDE EFFECT OF UPLOADING (dig-node#335). + # + # `releases/latest` is a user-facing pointer: dig-installer fetches through it, so the moment it + # moves is the moment users are served that release. Moving it only after the guard has read the + # real asset list means an incomplete release is never `latest` — the previous complete release + # keeps serving installs, which is the correct failure mode. A release that never completes + # simply never gets promoted, and stays visible as a non-latest release for diagnosis. + promote: + name: Promote the release to latest + needs: verify + if: github.ref_type == 'tag' + runs-on: ubuntu-latest + permissions: + contents: write + steps: + - name: Mark the verified release as latest + shell: bash + env: + GH_TOKEN: ${{ github.token }} + REPO: ${{ github.repository }} + TAG: ${{ github.ref_name }} + run: | + set -euo pipefail + gh release edit "$TAG" --repo "$REPO" --latest + echo "$TAG is now releases/latest — verified to carry every asset dig-installer and dig-updater resolve." diff --git a/.github/workflows/verify-release-assets.yml b/.github/workflows/verify-release-assets.yml index 3e3b8c9a..b852e1b4 100644 --- a/.github/workflows/verify-release-assets.yml +++ b/.github/workflows/verify-release-assets.yml @@ -1,30 +1,31 @@ -# The RELEASE ASSET GUARD (dig_ecosystem#2290). +# The RELEASE ASSET GUARD (dig_ecosystem#2290, widened by dig-node#335). # -# A `vX.Y.Z` release of dig-node is not shippable just because its binaries exist. dig-updater's -# feedsign resolves dig-node by the NATIVE INSTALL PACKAGE file names — the beacon installs -# dig-node by handing a package to msiexec/installer/dpkg, it never places a bare binary — and it -# FAILS CLOSED on the whole signed manifest when even one component cannot be resolved. So a -# stable release that carries binaries but no `.msi`/`.pkg`/`.deb` does not merely ship an -# incomplete dig-node: it freezes auto-update for EVERY product on the stable channel. +# A `vX.Y.Z` release of dig-node is not shippable just because a release object exists. Two +# separate consumers resolve assets out of it, and a release that satisfies one and not the other +# is broken for real users: # -# That is not hypothetical. `v0.99.9` shipped binaries only, feedsign failed closed on four -# consecutive runs, and the stable manifest sat frozen and then EXPIRED for ~15 hours while every -# individual workflow run in this repo reported success. Nothing was red, because nothing was -# asking the one question that mattered: does the published release actually carry the assets the -# feed needs? +# * dig-updater's feedsign resolves dig-node by the NATIVE INSTALL PACKAGE file names and FAILS +# CLOSED on the whole signed manifest when one cannot be resolved. `v0.99.9` shipped binaries +# only, and the stable manifest sat frozen and then EXPIRED for ~15 hours while every +# individual workflow run in this repo reported success. +# * dig-installer resolves the RAW `dig-node` and `dign` binaries through `releases/latest`. +# `v0.145.0` became `latest` carrying only the four native packages, five minutes before its +# binaries were attached, and every fresh install 404'd for that window (dig-node#335). # -# This workflow asks it, and is the only place that does. +# The expected set for BOTH consumers lives in one place — +# `.github/actions/check-release-assets` — and this workflow is how it gets asked. # -# * workflow_call — the stable release path (nightly-release.yml) waits on this after cutting a -# tag, so a package-less stable release reddens the release run itself rather than surfacing -# hours later as a feed failure in another repo. -# * workflow_dispatch — point it at ANY tag on demand. This is what makes the guard falsifiable: -# dispatching it at a release known to lack packages MUST fail, and at a complete release MUST -# pass. A guard that cannot be shown to go red is not a guard. +# * workflow_call — the stable release paths (nightly-release.yml after cutting the tag, and +# release.yml before promoting the release to `latest`) wait on this, so an incomplete release +# reddens the release run itself rather than surfacing hours later as a feed failure in +# another repo or a 404 on a user's machine. +# * workflow_dispatch — point it at ANY tag on demand. +# * pull_request — the SELF-TEST below, which is what makes the guard falsifiable without +# needing a broken release to exist. # -# It polls rather than sampling once, because the binary build and the package build are separate -# workflows that finish at different times; a single sample would race them and produce a red that -# only means "not finished yet". +# It polls rather than sampling once, because the binaries and the packages are attached by +# separate workflows that finish at different times; a single sample would race them and produce a +# red that only means "not finished yet". name: Verify release assets on: @@ -52,95 +53,98 @@ on: type: number required: false default: 5 + pull_request: + paths: + - ".github/actions/check-release-assets/**" + - ".github/workflows/verify-release-assets.yml" permissions: contents: read jobs: verify: - name: Verify ${{ inputs.tag }} carries the native install packages + name: Verify the release carries every consumer-resolvable asset + # The self-test trigger supplies no tag; it exercises `selftest` instead. + if: inputs.tag != '' runs-on: ubuntu-latest steps: - - name: Assert the feed-resolvable assets are present - env: - GH_TOKEN: ${{ github.token }} - REPO: ${{ github.repository }} - TAG: ${{ inputs.tag }} - TIMEOUT_MINUTES: ${{ inputs.timeout_minutes }} - run: | - set -euo pipefail - - # The version as it appears in asset names: the tag without its leading `v`. - VERSION="${TAG#v}" - - # EXACTLY the names dig-updater's feedsign looks for. Kept as a literal list rather than - # derived from a glob: a glob would happily accept a `.deb` for the wrong arch or a - # stale version and call the release complete, which is the failure this guard exists to - # catch. - # - # THIS IS THE THIRD COPY OF A CROSS-REPO CONTRACT, and a shell step cannot import the - # Rust constant that owns it. Producer: `package.yml` in this repo. Consumer: - # `dig-updater/crates/dig-updater-feedsign/src/resolve.rs` (`asset_name_parts`). - # Verifier: here. Nothing enforces that the three agree, so they are held together by - # `SYSTEM.md` (dig-updater section, "dig-node release-asset file names") and the - # `canonical` skill (beacon/update trust anchors). Change one, change all three. - # - # Note macOS contributes ONE name, not two: the `.pkg` is universal and carries no arch - # token, so `macos/arm64` and `macos/x64` both resolve to it — feedsign's five platforms - # yield four distinct file names. - # - # `arm64.deb` is required here DELIBERATELY, and this is stricter than feedsign's own - # failure condition. feedsign fails closed only when a component resolves ZERO assets, so - # a release missing just `arm64.deb` would still publish — silently dropping linux/arm64 - # hosts from auto-update rather than reddening anything. That silent drop is exactly the - # arm64 platform floor (#1741/#1736/#2126), so the stable channel treats a missing arm64 - # package as a failed release. Do not relax this to match feedsign. - EXPECTED=( - "dig-node_${VERSION}_amd64.deb" - "dig-node_${VERSION}_arm64.deb" - "dig-node-${VERSION}-macos.pkg" - "dig-node-${VERSION}-windows-x64.msi" - ) - - deadline=$(( $(date +%s) + TIMEOUT_MINUTES * 60 )) - attempt=0 - - while :; do - attempt=$(( attempt + 1 )) + - uses: actions/checkout@v4 + with: + persist-credentials: false + - uses: ./.github/actions/check-release-assets + with: + tag: ${{ inputs.tag }} + timeout_minutes: ${{ inputs.timeout_minutes }} + github_token: ${{ github.token }} - # A missing release is a legitimate "not yet" while the release workflow is still - # running, so it is treated the same as a missing asset rather than aborting early. - assets="$(gh release view "$TAG" --repo "$REPO" --json assets --jq '.assets[].name' 2>/dev/null || true)" - - missing=() - for name in "${EXPECTED[@]}"; do - printf '%s\n' "$assets" | grep -qxF "$name" || missing+=("$name") - done + # THE GUARD'S OWN REGRESSION TEST (dig-node#335). + # + # The reason #335 shipped is that the previous guard checked only the four native packages, so + # it reported success on a release missing all ten binaries. Nothing could have caught that, + # because nothing ever asked the guard to fail. + # + # This job asks. It drives the SAME action the release path uses, over literal asset lists: + # + # * the complete v0.145.0 set MUST pass — a guard that rejects a good release is just as + # broken, and this is the control that keeps the expectation from drifting into something + # unsatisfiable; + # * a set of the four native packages and nothing else — EXACTLY the asset list + # `releases/latest` carried at 2026-08-24T01:47:49Z, missing all five `dig-node-*` and all + # five `dign-*` binaries — MUST fail. `continue-on-error` lets the step run to completion, + # and the assertion afterwards turns a PASS into a red job. + # + # The failing case deliberately omits a whole consumer's set rather than one stray file, because + # the defect was categorical: an entire class of asset was outside the guard's vocabulary. + selftest: + name: Self-test — the guard passes a complete release and FAILS an incomplete one + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false - if [ ${#missing[@]} -eq 0 ]; then - echo "$TAG carries all ${#EXPECTED[@]} native install packages — feedsign can resolve dig-node." - { - echo "### Release assets verified — \`$TAG\`" - echo - for name in "${EXPECTED[@]}"; do echo "- \`$name\`"; done - } >> "$GITHUB_STEP_SUMMARY" - exit 0 - fi + - name: A complete asset set PASSES + uses: ./.github/actions/check-release-assets + with: + tag: v0.145.0 + assets: | + dig-node_0.145.0_amd64.deb + dig-node_0.145.0_arm64.deb + dig-node-0.145.0-macos.pkg + dig-node-0.145.0-windows-x64.msi + dig-node-0.145.0-linux-arm64 + dig-node-0.145.0-linux-x64 + dig-node-0.145.0-macos-arm64 + dig-node-0.145.0-macos-x64 + dig-node-0.145.0-windows-x64.exe + dign-0.145.0-linux-arm64 + dign-0.145.0-linux-x64 + dign-0.145.0-macos-arm64 + dign-0.145.0-macos-x64 + dign-0.145.0-windows-x64.exe - if [ "$(date +%s)" -ge "$deadline" ]; then - { - echo "### Release assets MISSING — \`$TAG\`" - echo - echo "dig-updater's feedsign cannot resolve dig-node from this release, so the" - echo "STABLE signed feed will fail closed for every component until it is fixed." - echo - echo "Missing:" - for name in "${missing[@]}"; do echo "- \`$name\`"; done - } >> "$GITHUB_STEP_SUMMARY" - echo "::error::release $TAG is missing ${#missing[@]} native install package(s): ${missing[*]}. dig-updater feedsign resolves dig-node by these names and fails closed on the ENTIRE stable manifest when they are absent. Attach them (dispatch package.yml against the $TAG ref) before this release is allowed to stand as latest." - exit 1 - fi + - name: An incomplete asset set is checked (expected to fail) + id: incomplete + continue-on-error: true + uses: ./.github/actions/check-release-assets + with: + # The four native packages and nothing else — the exact asset list `releases/latest` + # carried while dig-installer was 404ing (dig-node#335). The PREVIOUS guard passed this. + tag: v0.145.0 + assets: | + dig-node_0.145.0_amd64.deb + dig-node_0.145.0_arm64.deb + dig-node-0.145.0-macos.pkg + dig-node-0.145.0-windows-x64.msi - echo "attempt $attempt: still missing ${#missing[@]} of ${#EXPECTED[@]} (${missing[*]}); retrying…" - sleep 30 - done + - name: Require that it FAILED + shell: bash + env: + OUTCOME: ${{ steps.incomplete.outcome }} + run: | + set -euo pipefail + if [ "$OUTCOME" != "failure" ]; then + echo "::error::the release-asset guard PASSED a release carrying only the four native install packages. That is the dig-node#335 defect verbatim: dig-installer resolves the raw dig-node and dign binaries through releases/latest, so this asset set 404s every fresh install. The guard is not guarding." + exit 1 + fi + echo "The guard failed the incomplete asset set, as required." diff --git a/Cargo.lock b/Cargo.lock index 4826a5e3..cb126131 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3073,7 +3073,7 @@ dependencies = [ [[package]] name = "dig-node-service" -version = "0.145.0" +version = "0.145.1" dependencies = [ "async-trait", "axum", diff --git a/Cargo.toml b/Cargo.toml index f6f473ff..fc6e38c9 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -32,7 +32,7 @@ edition = "2021" # the ROOT manifest (`[workspace.package].version`), so it MUST be set here for a # release to fire (§3.6). The library crates (dig-node-core/dig-runtime/dig-wallet) # keep their own independent versions — only the released binary tracks the workspace version. -version = "0.145.0" +version = "0.145.1" # Release hardening, matching digstore: keep integer-overflow checks ON in release. # The node parses untrusted serialized input and does offset/length arithmetic over diff --git a/DEVELOPMENT_LOG.md b/DEVELOPMENT_LOG.md index 26e64427..713d4a40 100644 --- a/DEVELOPMENT_LOG.md +++ b/DEVELOPMENT_LOG.md @@ -4,6 +4,37 @@ High-signal realizations from debugging/development: non-obvious cross-system co sharp edges, and gotchas. Concise durable facts with context — NOT a change diary. See `CLAUDE.md` §4.5 for the maintenance contract (a curator periodically re-verifies + prunes). +## Two workflows assemble one release, so whichever finishes first used to publish it half-built (#335) + +A stable `vX.Y.Z` release of dig-node is built by **two** workflows that neither know about nor wait +for each other: `release.yml` attaches the ten raw binaries (`dig-node-*`, `dign-*`) and +`package.yml` attaches the four native install packages (`.deb`/`.pkg`/`.msi`). Both used +`softprops/action-gh-release`, **whose `make_latest` defaults to true**, so *attaching assets* was +also *promoting the release* — and the promotion was won by whichever job finished first. + +On v0.145.0 that was `package.yml`, at `01:47:49Z`. The binaries landed at `01:52:52Z`. For those +five minutes `releases/latest` was a release carrying four packages and no binaries at all, and +dig-installer resolves both the `dig-node` and `dign` stems through `releases/latest` +(`dig-installer/src/release.rs:187`), so **every fresh install 404'd** — with no red anywhere, +because both workflows genuinely succeeded at the job each was given. + +Two durable lessons: + +- **`releases/latest` is a user-facing pointer, not a bookkeeping detail.** The instant it moves, + users are served that release. It must be moved by ONE step that runs last and knows the whole + asset set — never as a side effect of an upload. `make_latest: false` on both publishers plus a + `promote` job gated on the asset guard is the shape. +- **A guard is only as wide as the consumer list it was written against.** The asset guard existed + and reported success on this release, because it had been written for dig-updater's feedsign and + knew only the four package names. dig-installer was a second consumer nobody had told it about. + When a check enumerates what a release must carry, enumerate *per consumer*, and make the guard + falsifiable — the self-test in `verify-release-assets.yml` fails the build if the guard ever again + passes an asset list carrying only the packages. + +Corollary for diagnosis: a release asset set is a **race**, so measuring it once during a release +run tells you the state at that instant and nothing about the outcome. Read the assets' +`created_at` against the release's `published_at` before concluding assets were never attached. + ## `initial_sync_complete` can NEVER latch on a default install — so it cannot mean "synced" (dig_ecosystem#2609) `sync_state.initial_sync_complete` is written by exactly one statement, `WalletDb::complete_catch_up`, diff --git a/SPEC.md b/SPEC.md index 6a9f5a21..7975ee79 100644 --- a/SPEC.md +++ b/SPEC.md @@ -3385,7 +3385,9 @@ boolean, default `false`). It MUST NOT trigger on `push` to `main`. check IS the version-changed check). Cutting = `git-cliff` regenerates `CHANGELOG.md`, commits it to `main` as `chore(release): vX.Y.Z`, tags THAT commit, and pushes commit + tag with `RELEASE_TOKEN`. The pushed `v*` tag fires `release.yml` (§11.2/§11.3), which publishes a GitHub - Release with `prerelease: false` + `make_latest: true` — the ONLY release that moves `latest`. + Release with `prerelease: false`. A stable release is the ONLY release that may move `latest`, and + it moves it in a separate PROMOTION step gated on the asset verification below — never as a side + effect of attaching assets (§11.1b). - **Force re-cut (guarded).** `force: true` bypasses skip-if-tagged and re-cuts the current version (moving the tag onto a fresh changelog commit; `main` is never force-pushed). It MUST be refused — non-zero exit, clear error — when BOTH: (a) a PUBLISHED (non-draft) Release exists at the tag, @@ -3402,14 +3404,26 @@ boolean, default `false`). It MUST NOT trigger on `push` to `main`. absent. Both workflows gate publication on `github.ref_type == 'tag'`, which a dispatch against a tag satisfies, so the dispatched run is equivalent to the event-triggered one. The confirmation MUST be idempotent: where the event was delivered normally, it dispatches nothing. -- **A stable release MUST carry the native install packages.** dig-updater's feedsign resolves - dig-node by the `.deb`/`.pkg`/`.msi` file names and fails closed on the ENTIRE signed manifest - when they are absent, so a stable release of bare binaries does not ship a partial dig-node — it - freezes auto-update for every component on the channel. The stable path MUST therefore verify the - published release's asset list (`verify-release-assets.yml`) and MUST fail the release run when - `dig-node__amd64.deb`, `dig-node__arm64.deb`, `dig-node--macos.pkg`, or - `dig-node--windows-x64.msi` is missing. Repairing a failed release by publishing only the - binaries is NOT a repair. +- **A stable release MUST carry every asset its consumers resolve — packages AND binaries.** Two + consumers read assets out of a stable release, and satisfying one is not satisfying the release: + dig-updater's feedsign resolves dig-node by the `.deb`/`.pkg`/`.msi` file names and fails closed on + the ENTIRE signed manifest when they are absent (freezing auto-update for every component on the + channel), while dig-installer resolves the raw `dig-node` and `dign` binaries through + `releases/latest` and 404s a fresh install when either is absent. The stable path MUST verify the + published release's asset list (`verify-release-assets.yml`) and MUST fail the release run when any + of the fourteen names is missing: + `dig-node__amd64.deb`, `dig-node__arm64.deb`, `dig-node--macos.pkg`, + `dig-node--windows-x64.msi`, and — for each of `linux-arm64`, `linux-x64`, `macos-arm64`, + `macos-x64`, `windows-x64.exe` — both `dig-node--` and `dign--`. + Repairing a failed release by publishing only one of the two sets is NOT a repair. +- **§11.1b. `latest` MUST NOT move until the release is verified complete.** A stable release is + assembled by two workflows that finish at different times (`release.yml` attaches the binaries, + `package.yml` the native packages), so neither may promote it. Both MUST publish with + `make_latest: false`, and `releases/latest` MUST be moved by a single promotion step that runs only + after the asset verification above has passed. An incomplete release therefore never becomes + `latest`: the previous complete release keeps serving installs, which is the required failure mode. + The guard MUST be falsifiable — a self-test MUST assert that it FAILS an asset list carrying only + the native packages. 11.1a. **Doc-only commits never release** (the version is unchanged → the tag exists → the stable job is a no-op). The manual-dispatch `workflow_dispatch` on `release.yml` is a build-only "does main diff --git a/crates/dig-node-service/tests/release_asset_matrix_guard.rs b/crates/dig-node-service/tests/release_asset_matrix_guard.rs new file mode 100644 index 00000000..ebe20f7f --- /dev/null +++ b/crates/dig-node-service/tests/release_asset_matrix_guard.rs @@ -0,0 +1,205 @@ +//! Guard: the release-asset EXPECTATION cannot drift away from what the build actually +//! produces, and exactly one place may move `releases/latest` (dig-node#335). +//! +//! #335 shipped because a stable release became `latest` carrying four of its fourteen assets. +//! Two mechanisms now prevent that, and both of them are only as good as their resistance to +//! quiet drift: +//! +//! 1. `.github/actions/check-release-assets` holds a HAND-MAINTAINED platform list. Losing a +//! platform from the build fails loudly — the guard demands an asset nobody produces. But +//! GAINING one fails SILENTLY: a new leg enters the build matrix, the expectation stays at +//! the old count, and a release missing the new platform's binaries passes every check and +//! becomes `latest`. That is exactly the arm64 platform-floor class +//! (dig_ecosystem#1741/#1736/#2126) that the expectation's own comment cites. +//! 2. Promotion belongs to `release.yml`'s `promote` job alone, after verification. A second +//! promoter — an `action-gh-release` step drifting back to its `make_latest` DEFAULT of +//! true, or a stray `gh release edit --latest` — restores #335 verbatim while every job +//! still reports success. +//! +//! The self-test inside `verify-release-assets.yml` cannot catch either. It drives the guard over +//! LITERAL asset lists written from the same expectation, so it matches its own needle: a +//! platform absent from the expectation is equally absent from the fixture. Only a check that +//! reads the BUILD's matrix, rather than the expectation's restatement of it, can see the gap. +//! +//! Every file is embedded at compile time so these run hermetically. + +/// Owns the expected asset set the release is verified against. +const CHECK_ACTION_YML: &str = + include_str!("../../../.github/actions/check-release-assets/action.yml"); + +/// Owns the platform matrix the binaries are actually built for. +const BUILD_YML: &str = include_str!("../../../.github/workflows/build-binaries.yml"); + +/// The two publishers and the orchestrator — every workflow that can touch `latest`. +const RELEASE_YML: &str = include_str!("../../../.github/workflows/release.yml"); +const PACKAGE_YML: &str = include_str!("../../../.github/workflows/package.yml"); +const NIGHTLY_YML: &str = include_str!("../../../.github/workflows/nightly-release.yml"); + +/// The `PLATFORMS=(…)` array the expected asset names are generated from. +fn expected_platforms() -> Vec { + let (_, after) = CHECK_ACTION_YML + .split_once("PLATFORMS=(") + .expect("check-release-assets must declare a `PLATFORMS=(…)` array"); + let (inner, _) = after + .split_once(')') + .expect("the `PLATFORMS=(` array must be closed on one line"); + let mut platforms: Vec = inner.split_whitespace().map(str::to_owned).collect(); + platforms.sort(); + platforms +} + +/// Every distinct `out_name:` in the build workflow — the platform token that ends up in +/// `dig-node--` and `dign--`. +/// +/// Read as a SET rather than a list because `out_name` also appears in the glibc-verification +/// matrix, which repeats tokens the build already emits. Repetition is therefore invisible here, +/// while a genuinely NEW platform — the case this guard exists for — is not. +fn built_platforms() -> Vec { + let mut platforms: Vec = BUILD_YML + .lines() + .filter_map(|line| line.trim().strip_prefix("out_name:")) + .map(|value| value.trim().to_owned()) + .collect(); + platforms.sort(); + platforms.dedup(); + platforms +} + +/// The expectation must name EVERY platform the build produces, and no others. +/// +/// Stated as equality, not containment, in both directions on purpose: a superset would demand an +/// asset that will never exist and wedge every release, while a subset is the silent hole above. +#[test] +fn the_expected_platforms_are_exactly_the_platforms_the_build_produces() { + let expected = expected_platforms(); + let built = built_platforms(); + + assert_eq!( + expected, built, + "the release-asset expectation in .github/actions/check-release-assets \ + has drifted from the build matrix in .github/workflows/build-binaries.yml.\n \ + expected by the guard: {expected:?}\n \ + produced by the build: {built:?}\n\ + A platform the build produces but the guard does not expect is NOT verified, so a \ + release missing its `dig-node-*` / `dign-*` binaries would still be promoted to \ + `releases/latest` and 404 every fresh install for that platform (dig-node#335). \ + Add the platform to PLATFORMS in the same change that adds it to the build." + ); +} + +/// Sanity floor on the parsers themselves. +/// +/// Both helpers above are string scrapes, and a scrape that silently matches NOTHING returns an +/// empty set — which compares equal to another empty set and turns the assertion above into a +/// vacuous pass. This is the control that keeps that from happening quietly. +#[test] +fn the_platform_sets_are_non_trivial() { + assert!( + expected_platforms().len() >= 5, + "parsed too few expected platforms — the `PLATFORMS=(…)` scrape has broken, which would \ + make the drift check vacuous" + ); + assert!( + built_platforms().len() >= 5, + "parsed too few built platforms — the `out_name:` scrape has broken, which would make \ + the drift check vacuous" + ); +} + +/// Neither publisher may promote (dig-node#335). +/// +/// A stable release is assembled by TWO workflows that do not wait for each other: +/// `release.yml` attaches the binaries, `package.yml` the native install packages. Both use +/// `softprops/action-gh-release`, **whose `make_latest` defaults to true**, so an omitted setting +/// is not a neutral omission — it hands `latest` to whichever job happens to finish first. On +/// v0.145.0 that was the packages job, five minutes before the binaries existed. +#[test] +fn no_asset_upload_may_promote_the_release_to_latest() { + for (name, yml) in [("release.yml", RELEASE_YML), ("package.yml", PACKAGE_YML)] { + let uploads = yml.matches("softprops/action-gh-release").count(); + let declines = yml.matches(r#"make_latest: "false""#).count(); + assert_eq!( + uploads, declines, + "{name} has {uploads} action-gh-release step(s) but {declines} `make_latest: \"false\"` \ + setting(s). The action DEFAULTS to make_latest: true, so every upload step must \ + decline promotion explicitly — otherwise attaching assets promotes a release that \ + may still be half-built (dig-node#335)." + ); + assert!( + !yml.contains(r#"make_latest: "true""#), + "{name} must not promote from an upload step; promotion is release.yml's `promote` \ + job, which runs only after the asset guard passes" + ); + } +} + +/// Exactly ONE place moves `latest`, and it is the verified one. +/// +/// `--latest=false` is a DEMOTION (the nightly channel keeps its pre-releases off `latest`) and +/// is counted separately — treating it as a promotion would make this guard reject the correct +/// code, and treating a promotion as a demotion would let #335 back in. The distinction is the +/// whole assertion, so it is drawn explicitly rather than by a substring search for `--latest`. +#[test] +fn exactly_one_site_promotes_a_release_to_latest() { + let promotions: Vec<(&str, &str)> = [ + ("release.yml", RELEASE_YML), + ("package.yml", PACKAGE_YML), + ("nightly-release.yml", NIGHTLY_YML), + ] + .into_iter() + .flat_map(|(name, yml)| { + yml.lines() + .filter(|line| line.contains("--latest") && !line.contains("--latest=false")) + .map(move |line| (name, line.trim())) + }) + .collect(); + + assert_eq!( + promotions.len(), + 1, + "exactly one workflow site may move `releases/latest`, and it must be the `promote` job \ + in release.yml that runs after verification. Found {}: {promotions:#?}", + promotions.len() + ); + assert_eq!( + promotions[0].0, "release.yml", + "the single promotion site must live in release.yml, downstream of the asset guard" + ); +} + +/// The guard must read asset STATE, not merely asset names (dig-node#335, finding 1). +/// +/// GitHub creates an asset row when its upload STARTS, in state `starting`. A name is therefore +/// visible before its bytes are, and `verify` is ordered (`needs: publish`) only against +/// release.yml's own upload — package.yml is a separate workflow with no ordering relationship to +/// it. Without the state filter the poll can see all fourteen names while a `.msi` or `.pkg` is +/// still uploading, and promote a release whose download is incomplete: #335's observable +/// outcome through a shorter window. +/// +/// Asserted here rather than in the workflow's self-test because that self-test feeds the action +/// a literal asset list and never reaches the network read this filter lives on. +/// +/// Asserted against the `gh release view` COMMAND LINE specifically, never against the file as a +/// whole. The first version of this test searched the whole YAML and passed happily with the +/// filter DELETED from the query — because the comment above the query explains the filter and +/// contains the same text. A source-scanning assertion that matches its own explanatory prose is +/// satisfied by the documentation of the thing it is meant to require. +#[test] +fn the_guard_counts_only_fully_uploaded_assets() { + let query = CHECK_ACTION_YML + .lines() + .find(|line| line.contains("gh release view")) + .expect("check-release-assets must read the release's asset list with `gh release view`"); + + assert!( + query.contains(r#"select(.state == "uploaded")"#), + "the `gh release view` query must filter to `state == \"uploaded\"`, but reads:\n {query}\n\ + An asset row exists from the moment its upload BEGINS, so counting names alone lets a \ + release be promoted while a package is still uploading (dig-node#335)." + ); + assert!( + !CHECK_ACTION_YML.contains("sleep 60"), + "a sleep is not a substitute for the state filter — it narrows the race without closing \ + it, and cannot be shown to fail, which is worse than leaving it visible" + ); +} diff --git a/runbooks/release.md b/runbooks/release.md index ffd9005b..85fbc321 100644 --- a/runbooks/release.md +++ b/runbooks/release.md @@ -84,8 +84,19 @@ Actions → **Nightly + stable release** → **Run workflow** → `channel: nigh ## Verify a release went live -- **Stable:** `gh release view vX.Y.Z --repo DIG-Network/dig-node` — 4 OS/arch × (`dig-node-*` + - `dign-*`), `prerelease: false`, marked latest. Watch: `gh run watch `. +- **Stable:** `gh release view vX.Y.Z --repo DIG-Network/dig-node` — 5 platforms × (`dig-node-*` + + `dign-*`) plus the 4 native packages, 14 assets, `prerelease: false`, marked latest. Watch: + `gh run watch `. +- **A stable release is marked `latest` LAST, by `release.yml`'s `promote` job, and only after the + asset guard has read the real asset list.** So a stable release that is published but NOT latest + means the guard has not passed yet — either a build is still running, or a publisher never + landed. That is working as designed: the previous complete release keeps serving installs rather + than a half-built one taking over. Read the guard's step summary before doing anything by hand. +- **Gotcha — re-running `package.yml` by hand after a release is already latest will DEMOTE it.** + Both publishers deliberately set `make_latest: false` (dig-node#335), so an out-of-band re-run + un-marks `latest`. Re-promote with + `gh release edit vX.Y.Z --repo DIG-Network/dig-node --latest`, or re-dispatch `release.yml` + against the tag, which verifies and promotes in the right order. - **Nightly:** `gh release view nightly --repo DIG-Network/dig-node` (rolling) or `gh release view nightly-YYYYMMDD` — `prerelease: true`. - **The native packages, on EITHER channel** — the single check that tells you the update system can @@ -95,8 +106,18 @@ Actions → **Nightly + stable release** → **Run workflow** → `channel: nigh gh release view nightly --repo DIG-Network/dig-node --json assets --jq '[.assets[].name | select(endswith(".deb") or endswith(".pkg") or endswith(".msi"))]' ``` - Expect three names. Fewer means dig-updater's `Feed` workflow will fail that channel with - `no matching release assets` — the failure mode dig_ecosystem#618 fixed. + Expect three names on nightly (four on stable, which also carries `arm64.deb`). Fewer means + dig-updater's `Feed` workflow will fail that channel with `no matching release assets` — the + failure mode dig_ecosystem#618 fixed. +- **The raw binaries, on stable** — the check dig-installer depends on, and the one whose absence + caused dig-node#335: + + ```bash + gh release view vX.Y.Z --repo DIG-Network/dig-node --json assets --jq '[.assets[].name | select(startswith("dign-") or test("^dig-node-[0-9].*(linux|macos|windows)"))] | length' + ``` + + Expect ten. Fewer means a fresh install 404s, because `dig-installer` resolves both the + `dig-node` and `dign` stems through `releases/latest`. ## Gotcha — moving a Windows host from `nightly` back to `stable` @@ -120,10 +141,10 @@ a bug in the beacon: | File | Trigger | Role | |---|---|---| | `nightly-release.yml` | midnight-UTC cron + `workflow_dispatch` | Orchestrator: stable (changelog + tag) + nightly (build + pre-release + prune). | -| `release.yml` | `push: tags: v*` (+ dispatch canary) | Builds + publishes the stable Release for a `vX.Y.Z` tag. | +| `release.yml` | `push: tags: v*` (+ dispatch canary) | Builds + publishes the stable Release for a `vX.Y.Z` tag, then verifies the asset set and promotes it to `latest`. Attaching assets never moves `latest` by itself. | | `build-binaries.yml` | `workflow_call` | Reusable cross-OS build, dual-named + `dign` (both channels call it). | | `package.yml` | PR + `push: tags: v*` + `workflow_call` | Builds the `.deb`/`.pkg`/`.msi`. Attaches them itself on a `v*` tag; on a `workflow_call` (the nightly channel) it leaves them as run artifacts for the caller to publish. | -| `verify-release-assets.yml` | `workflow_call` (stable path) + `workflow_dispatch` | Asserts a `vX.Y.Z` release carries the four native install packages dig-updater's feedsign resolves dig-node by. Dispatch it at any tag to check a release by hand — a package-less release freezes the stable signed feed for every product. | +| `verify-release-assets.yml` | `workflow_call` (stable path) + `workflow_dispatch` + PR self-test | Asserts a `vX.Y.Z` release carries all fourteen consumer-resolvable assets: the four native install packages dig-updater's feedsign resolves dig-node by, plus the five `dig-node-*` and five `dign-*` raw binaries dig-installer fetches through `releases/latest`. `release.yml` waits on it before promoting the release to `latest`, so an incomplete release never becomes the one users install from. Dispatch it at any tag to check a release by hand. The expected set lives in `.github/actions/check-release-assets`; a PR self-test drives that same action over a deliberately incomplete list and requires it to fail. | | `ci.yml` | PR + push to main | fmt/clippy + `cargo llvm-cov nextest --workspace` (pre-merge). NOTE: `ubuntu-latest` only — Windows/macOS build breaks are first caught by the nightly channel, not PR CI (SPEC §11 / follow-up). | ## Local build (dev)