fix(cluster): GRD recovery liveness — same-episode remaster retry + proof-carrying WAIT_EPOCH + dead-node PCM cleanup (spec-4.6a, BUG-B/C2)#26
Merged
Conversation
added 2 commits
July 8, 2026 18:24
… WAIT_CLUSTER watchdog, event-scoped WAIT_EPOCH escape, dead-node PCM cleanup A fail-stop reconfig could wedge a settled cluster forever on three stacked liveness holes: a BLOCKED HW remaster worker was never relaunched within the same episode (BGW_NEVER_RESTART + launched-latch), WAIT_CLUSTER had no watchdog while its cluster/HW gates were held, and WAIT_EPOCH's strict progress gate wedged permanently when an IC-piggybacked epoch bump landed before the local reconfig event (baseline re-captured as the post-bump value, cur == old forever). - HW remaster: per-dead-node result/attempts/next_attempt shmem state; the FSM relaunches BLOCKED workers with exponential backoff (cap 60s) up to cluster.hw_remaster_retry_max_attempts (SIGHUP-raisable); abnormal worker exit marks BLOCKED via before_shmem_exit; the three silent BLOCKED returns now LOG their cause. wal_threads_dir-unset is detected up front as BLOCKED_STRUCTURAL: never retried, WARNING-once with the config hint, and (level 2) a multi-node shared_catalog=on boot without it now fails fast at startup. - WAIT_CLUSTER: observational watchdog on the rebuild-timeout cadence — WARNING with the missing-survivor list, gate states and per-dead-node retry state, plus a counter; it never unfreezes anything. - WAIT_EPOCH: three proof-carrying layers — abort-to-idle on a new event id, durable observed-epoch adoption, and an event-scoped coordinator witness (REDECLARE_DONE carries the event id; stale-event DONEs are dropped at accounting; the equal-epoch escape additionally requires the coordinator's DONE to have been accounted after the accept snapshot). No timeout-only advance exists; without proof the shards stay frozen. - Dead-node PCM cleanup: the GRD dead-sweep clears the dead node's x_holder/s_holders/pi_holders/master_holder and pending-X residue (LOG summary + pcm/dead_cleanup_entries counter), and a local-master S state with no local residency can now upgrade S->X through the standard invalidate/ACK path instead of failing closed forever — post-kill DDL recovers instead of timing out on cluster locks. - Serve-gate scope alignment: the merged-materialization check in gcs_block phase_for_tag now uses the same thread-recovery scope policy as the unfreeze gate, removing the permanent-RECOVERING mismatch outside the gate's applicable scope (in-scope behavior unchanged). - Observability: hw retry/exhausted + grd cluster_gate_timeout / wait_epoch_escape + pcm dead_cleanup_entries dump keys; S4-reject and HW fail-closed diagnostics. Tests: unit relaunch-decision table (8 rows) + event-scoped DONE rejection/ witness advance + PCM dead-cleanup forms incl. pending-X; t/293 gains the same-episode self-heal and retry-exhaustion legs (unfreeze-extend positive + watchdog WARNING greps); t/337 gains the level-2 fail-fast negative leg; new t/362 runs a 4-node shared_catalog formation, kill -9, and asserts remaster convergence, post-kill DDL, cleanup counters and the 0-wedge invariant end to end. Local gates (cassert): unit 158 binaries, t/293 + t/337 + t/362 + smoke subset, cluster_regress 13/13, PG regress 219/219. Spec: spec-4.6a-grd-recovery-liveness.md
t/358-361 are reserved by parallel lanes (spec-7.2 / S-xid), t/363 by the S-dead lane; occupancy verified against origin branches. The 4-node formation + kill -9 + convergence run gets its own shard for wall-clock isolation. Spec: spec-4.6a-grd-recovery-liveness.md
sqlrush
force-pushed
the
stage7-p0-reconfig-liveness
branch
from
e8fc3b9 to
e7ca433
Compare
added 11 commits
July 8, 2026 18:25
…adow unit Same inline suppression + reason as the other cluster_unit main() signatures (test_cluster_grd.c precedent); a newer local cppcheck flags it, the CI baseline version does not.
…hash) The P6 all-done gate and the WAIT_EPOCH coordinator witness keyed the cross-node DONE on event_id, but event_id folds the sender-local cssd_dead_generation, which drifts with each survivor's private flap observation history and never converges across nodes: any flap asymmetry made the survivors compute different ids for the same episode, drop each other's DONEs, and wedge P6 forever — nondeterministically reintroducing the permanent-freeze shape this branch exists to remove. The quorum-accepted dead SET is what actually converges, so the DONE payload now carries a hash over the dead bitmap alone (same kernel as the event_id hash, riding the same request-id field pair — no envelope change), stamped per episode at P0 accept. Accounting, the P6 gate and the witness compare against the stamp; event_id stays a purely local accept-dedup scope. A late DONE from a previous episode still cannot back a new one: a coordinator re-election changes the dead set (hash mismatch) and a same-node re-death rides a higher epoch through the interposed JOIN bump (epoch conjunct). Pre-accept frames now mismatch the previous stamp and are dropped; senders re-announce every tick, so accounting always lands after the accept snapshot — closing the first-event pre-accept window as a side effect. Spec: spec-4.6a-grd-recovery-liveness.md (Amendment v1.2 R2/R4)
Revert the scope short-circuit that returned NORMAL for a dead static master's blocks wherever online thread recovery cannot run (the default configuration and every >2-node deployment): it skipped both the is_materialized cold-block door and the redo-coverage lost-write door, so a committed write only the dead node saw could be silently read stale from shared storage. No other guard sits on that read path — the GRD freeze ends with the episode, the HW gate covers only extend high-water marks, and the page-SCN checks ride the ship path. Out of scope the posture is now an explicit bounded retryable error (53R9L, hint updated to name the way out: restart the failed node, or enable online thread recovery in a supported scope) for every tag whose static master is dead — including never-written blocks, which the cold-block door cannot distinguish from not-yet-replayed ones — and the door reopens the moment the failed node stops being DEAD. In-scope behavior is byte-identical. The now-orphaned scope helper is removed. t/362 asserts the honest two-outcome contract on fresh connections (success or explicit SQLSTATE — 53R9L or the TT-unknown visibility door — never a hang), rides pre-warmed sessions for convergence observability, and pins the D12 cleanup counter to a positive delta; t/293's post-kill extend leg follows the same posture. Spec: spec-4.6a-grd-recovery-liveness.md (Amendment v1.2 R1/R3)
…grade cleanup - Order the worker's terminal-result store after its backoff-deadline store (pg_memory_barrier) so the relaunch decider never pairs a fresh BLOCKED with a stale deadline and skips one backoff wait. - A structural cause discovered only by the WORKER (SIGHUP race) now leaves the retry deadline unset so the FSM's next tick takes the MARK_STRUCTURAL branch and emits the once-per-episode operator WARNING with the configuration hint. - Wrap the local S->X upgrade in PG_TRY and release the temporary S claim when the ACK wait throws (cancel via CHECK_FOR_INTERRUPTS), not only on the false return; document why the completed upgrade hard-resets the local S refcount (the X grant subsumes all local S declarations). Linker-only exception stubs for the units that link cluster_pcm_lock.o. Spec: spec-4.6a-grd-recovery-liveness.md (Amendment v1.2 R5/R6/R7/R9)
WAIT_BARRIER / WAIT_CLUSTER carried only epoch-coherence guards, so a
survivor whose CSSD detected a coordinator-folded multi-death late kept
announcing REDECLARE_DONE under its stale dead-bitmap hash: peers that
stamped the full set dropped those frames on the composite key and P6 —
which has no timeout by design — wedged permanently (survivor-behind
detection skew). Add a mid-episode guard that aborts to IDLE on a newer
local event whose dead-bitmap hash differs: the next tick re-consumes
the event, re-stamps recovery_event_bitmap_hash, re-runs recovery
against the fuller dead set and re-announces DONE under the new
composite key. Same-set dead_generation drift is absorbed without
episode churn (the event_id keeps its local ABA-scoping role). The
coordinator-behind direction needs no handling: the coordinator's own
later detection bumps the epoch again and the epoch guards abort.
Correct the stamp-site / accounting / hash-helper comments that
overstated the bitmap as quorum-ratified (each node stamps its OWN
accepted event's bitmap; convergence is eventual via the CSSD deadband
plus this guard), and assert a stamped episode hash is never 0 — 0 is
reserved as unstamped in the mark_peer_done drop test, and a JOIN
episode's all-zero bitmap hashes to a fixed nonzero constant.
Unit: survivor-behind growth leg (H({1}) stamped at the folded epoch,
grown {1,2} event arrives -> abort, re-stamp, full-set remaster, DONE
accounting converges) plus a same-set drift no-churn leg; both go red
with the guard disabled.
Spec: spec-4.6a-grd-recovery-liveness.md (r3-P2-2, r3-P3a)
…arrier hw_remaster_record_terminal stores the backoff deadline before the terminal result with a release fence between the stores, but the LMON relaunch decider loaded result-then-deadline with no read-side pairing: on a weakly-ordered CPU it could still pair a fresh BLOCKED with a stale zero deadline and skip one backoff wait. Insert pg_read_barrier() between the result and deadline loads (attempts needs no fence — it is written only by the single-writer LMON FSM) and cross-reference the pairing at the write site. Spec: spec-4.6a-grd-recovery-liveness.md (r3-P3b)
The serve-gate unseal (static master no longer CSSD-DEAD -> NORMAL) is heartbeat liveness, not a direct recovery-completion signal. Document the traced proof of why returning NORMAL is nevertheless safe: CSSD — the only heartbeat sender — is spawned by the phase-4 driver at the PM_RUN transition, after the startup process completed the node's crash recovery against shared storage; and even under a stale-ALIVE view a fetch cannot complete against a still-recovering node, because the master-side handler default-denies until the node is an in-quorum MEMBER and only the (equally post-PM_RUN) QVOTEC process can establish the quorum lease after a restart wiped shmem. Deny replies surface as bounded 53R9L, endpoint unavailability as bounded 53R90, and no path falls back to a silent local storage read. Replaces the previous wording that asserted recovery-completion semantics without grounding. Spec: spec-4.6a-grd-recovery-liveness.md (r3-P2-1 prove-safe)
… leg The Amendment v1.2 (R1) serve-gate fix had no test lock: the t/362 two-outcome legs stay green under a reverted gate (their success arm accepts a NORMAL serve of a dead-master page) and no unit links the real phase decision. Add L4h: after the kill, with the dead node never restarted, once the observed survivor's own recovery episode reached IDLE, a FRESH backend on a cold survivor runs a wide pg_class/pg_attribute scan that MUST fail with exactly 53R9L — no success arm, bounded return. An out-of-scope formation never publishes the dead node's materialization proof, so every dead-master page stays RECOVERING while the node is down; the scan spans dozens of catalog pages hashed ~1/4 to the dead node and cold-reads them past the 16MB test pool, and it touches no dead-node-written rows (heap INSERTs only, no DDL), so 53R9L is the only legal outcome. Verified red-first: with the gate reverted the suite fails at exactly this leg while every other leg stays green. The episode-IDLE wait polls the pre-warmed session with the exact query shape it ran pre-kill and compares in Perl: a cast or operator added to the SQL performs fresh syscache lookups that themselves cold-read a dead-master catalog page and trip 53R9L (the very mechanism under test). Also accept the 53R51 write-fence rejection in the pre-existing two-outcome write legs: a transient lease/epoch fence right after the reconfig is an explicit, retryable fail-closed rejection inside the same honest contract. Spec: spec-4.6a-grd-recovery-liveness.md (r3-P1-1 TAP hard leg)
The spec-4.6a Amendment v1.2 (R2) composite convergence key gated both
recovery_done_epoch[] and recovery_done_bitmap_hash[] behind one drop gate
in cluster_grd_recovery_mark_peer_done(). A rejoining node never P0-accepts
the JOIN episode as its own FSM episode (JOIN_COMMITTED is published
coordinator-side only), so its local episode-hash stamp stays 0 forever; the
composite gate then dropped every survivor REDECLARE_DONE, the join fence
(cluster_grd_block_view_rebuilt) never lifted, and every joiner-home block
access fail-closed 53R9L permanently. Nightly t/347 L4-iii (rejoin with
xid_striping) and t/353 L5 (fresh insert on reborn node0) pinned the wedge.
Split the two accounting axes:
* done_epoch[] accounts unconditionally (monotonic-max), feeding the
epoch-only join fence. Sound because REDECLARE_DONE(epoch=E) is
broadcast only after grd_block_redeclare_scan_complete(E), so a DONE at
E proves the sender finished its WAIT_BARRIER re-declare at E -- the
fence's safety condition, independent of the sender's episode dead set.
* done_bitmap_hash[] stays composite-gated for the P6 cluster gate and the
WAIT_EPOCH coordinator witness, which still AND both axes and stay
fail-closed on the withheld hash half (r3-P2-2 multi-death skew
protection unchanged).
New unit leg test_recovery_idle_joiner_accounts_done_epoch_for_fence asserts
the fence lifts once survivor DONEs land while the composite half stays
withheld; the stale-bitmap and dead-set-growth legs gain a negative
assertion that the epoch axis alone never fires the composite escape.
Spec: spec-4.6a-grd-recovery-liveness.md
The same S-reconfig nightly that pinned the join-fence wedge also flagged
two stale dump-count baselines that lagged already-shipped counters:
* t/108 pg_cluster_state pcm category 22 -> 23 (spec-4.6a D12
dead_cleanup_entries).
* t/249 grd_recovery dump roster 18 -> 31 keys plus the exact sorted key
list (spec-4.6 2.4 base + spec-5.16 D5 join direction + spec-4.6a
liveness/episode surface).
Both stay exact-count / exact-roster assertions; the code already emits the
31-key grd_recovery set (verified against cluster_debug.c).
Spec: spec-4.6a-grd-recovery-liveness.md
t/353 L5 crashes node0 and has the reborn node take a FRESH HW space lease, which requires the survivor to complete HW remaster of the dead node's shards. HW remaster reads the dead node's per-thread WAL, so on this shared_data 2-node pair it is only recoverable when cluster.wal_threads_dir is configured (cluster_hw_remaster_recoverable). The pair was created with shared_data but WITHOUT wal_threads_root, so recoverable() returned false, the remaster was marked BLOCKED_STRUCTURAL, and the reborn node's fresh-lease DDL/insert/count fail-closed (L5 subtests 28-31, nightly crossnode-b). Opt the pair into wal_threads_root, matching every other HW-remaster- dependent kill test (t/247/248/274/293). This completes the per-thread WAL harness config that the RACvsRAC/ClusterQuad harness already carries but this ClusterPair test was missing. Spec: spec-4.6a-grd-recovery-liveness.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A fail-stop reconfig could wedge a settled cluster permanently (observed 56min+, zero self-heal) on three stacked liveness holes:
BGW_NEVER_RESTART+ launched-latch) — silent BLOCKED causes included an unconfiguredcluster.wal_threads_dir.WAIT_CLUSTERhad no watchdog while its cluster/HW gates were held — 8.A fail-closed, but invisible.WAIT_EPOCH's strict progress gate wedged forever when an IC-piggybacked epoch bump landed before the local reconfig event (baseline re-captured post-bump →cur == oldpermanently).Fixes: bounded same-episode HW remaster retry (backoff + SIGHUP-raisable cap, BLOCKED_STRUCTURAL never retried + WARNING-once + multi-node shared_catalog startup fail-fast), observational WAIT_CLUSTER watchdog (never unfreezes), three-layer proof-carrying WAIT_EPOCH escape (event-scoped REDECLARE_DONE witness — stale-event DONEs dropped at accounting; no timeout-only advance exists), dead-node PCM holder/pending-X cleanup on the dead-sweep (post-kill DDL recovers instead of timing out), serve-gate scope alignment, and full observability (retry/exhausted/watchdog/escape/cleanup counters).
Tests
Gates (local, cassert build)
cluster_unit 158 binaries · t/293 + t/337 + t/362 + smoke subset · cluster_regress 13/13 · PG regress 219/219 · clang-format 0 violations · headers/scn-cmp/no-clog-overlay/cppcheck clean.
Spec: spec-4.6a-grd-recovery-liveness.md