feat(cluster): root-cause-#1 shared undo physical data plane under GCS (D1–D6 integration)#40
Merged
Merged
Conversation
added 30 commits
July 8, 2026 12:37
Name an undo block (including the segment TT header block) as a first-class cluster resource: (owner_node, undo_segment, block_no, generation) encoded into the 16-byte ClusterResId wire format as class 0xF9. Undo is the first owner-as-master resid class: the resource master IS the owning instance (cluster_undo_resid_master), never a GRD shard-hash node. The GRD hash-master lookup now fails closed (Assert + new SQLSTATE 53R9Q) if an undo resid reaches it, and the generation predicate (segment wrap_count) lets callers fail closed on recycled-segment stale references. Pure identity layer only: no grant/PI/serving data plane, no write path change, no on-disk format or wire ABI change (ClusterResId stays 16 bytes; new class byte value only). - new cluster_undo_resid.h: class byte 0xF9 + collision StaticAssert net (9 header-visible classes + LOCKTAG_LAST_TYPE) + field mapping - new cluster_undo_resid.c: encode/decode/is_undo/master/ generation_matches pure layer (standalone-linkable) - cluster_grd.c: fail-closed undo-class guard at cluster_grd_lookup_master entry - errcodes.txt: 53R9Q ERRCODE_CLUSTER_UNDO_RESID_HASH_ROUTED - tests: test_cluster_undo_resid (8 cases: round-trip, class byte, discriminator, 16B wire lock, owner bounds, owner-as-master, generation); guard-fire leg in test_cluster_grd via an ereport trampoline; t/006 SQLSTATE spot-check line Spec: spec-5.22a-undo-block-resource-identity.md
…2-1)
D2-1 wires the D1 undo-block resource identity into the GCS data plane's
routing half: two predicates that keep undo resources off the GRD/GCS
hash-master path. Their authority lives at the owning instance, so the
master IS the encoded owner_node (cluster_undo_resid_master), never a
shard hash -- the D1-5 guard already fails closed (53R9Q) on any undo
resid that reaches cluster_grd_lookup_master / cluster_gcs_lookup_master.
- cluster_undo_gcs.{c,h} (new): cluster_undo_block_lookup_master returns
the owner_node; cluster_undo_block_master_is_self is the local
fast-path gate (owner_node == cluster_node_id). The owner-incarnation
epoch self-check that L364 requires before serving from the local fast
path is deferred to the grant/acquire path (D2-3), where the
co-sampled live-authority triple is available, so this routing layer
stays pure (node-id only) and cluster_unit links it standalone.
- src/backend/cluster/Makefile: cluster_undo_gcs.o added to OBJS.
- test_cluster_undo_gcs (new, U1/U2): lookup_master returns owner (not a
hash); master_is_self true for owner==self, false for a foreign owner.
Links cluster_version.o + cluster_undo_resid.o + cluster_undo_gcs.o
with a test-owned cluster_node_id stub.
Zero behaviour change: no consumer routes undo through these predicates
yet (undo_gcs_coherence data plane is D2-3+), so this mirrors D1's
land-then-wire shape.
Spec: spec-5.22b-undo-block-gcs-integration.md
…22b D2-2)
D2-2 gives an undo segment an optional physical home on the shared
cluster_fs root under GCS, breaking the DataDir-local hardcode in the
undo I/O path. The migration is gated on a new GUC
(cluster.undo_gcs_coherence, default off), so with the default the
whole change is inert and every path resolves to the local DataDir
byte-for-byte as before -- mirroring D1/D2-1's land-then-wire shape.
Path intent (the core contract):
- ClusterUndoPathIntent {RUNTIME_SHARED, MATERIALIZED_LOCAL} names which
home a segment resolves to. cluster_undo_intent_for_owner(owner)
derives it -- an own-instance owner is RUNTIME_SHARED, a foreign owner
(in D2 only ever a dead-origin copy recovery rebuilt in the local
DataDir) is MATERIALIZED_LOCAL. Static inline in cluster_undo_alloc.h
so the ~30 call sites add no link dependency.
- cluster_undo_path_uses_shared_root(intent, peer_mode, coherence) is
the single pure decision function: RUNTIME_SHARED resolves under the
shared root only when peer-mode AND coherence are on;
MATERIALIZED_LOCAL always stays local (P1-3 hard contract -- the
dead-origin by-xid resolve path must never move to shared storage).
Threading + migration:
- intent is a new first parameter on cluster_undo_path_resolve and the
four undo smgr APIs (read/write_block, read/write_header_bytes) and is
carried in the get_segment_fd fd-cache key.
- the redo write surface (cluster_undo_xlog.c build_undo_segment_path /
redo_open_segment / redo_stamp_slot) delegates to the SAME
cluster_undo_path_resolve, so runtime and redo cannot split-brain an
own-instance segment onto different roots.
- the shared home resolves via cluster_shared_fs_undo_path_resolve /
_instance_dir_resolve (owner-partitioned instance_<N> under
cluster.shared_data_dir). Undo is outside the RelFileLocator
namespace, so this is a path resolver, not a shared_fs vtable callback.
- ensure-instance-subdir gains a dual branch (shared root -> pg_mkdir_p
on the shared tree; local -> the existing mkdir) so a coherence-on
tree is physically complete.
Supporting:
- new GUC cluster.undo_gcs_coherence (bool, default off, PGC_SIGHUP);
check hook rejects turning it on while cluster.shared_data_dir is
unset (GUC_check_errdetail).
- D4 tripwire: cluster_undo_path_resolve asserts
(intent == RUNTIME_SHARED) == (owner == self), guarding the future
dead-owner foreign-runtime-shared serve case.
- drop a now-redundant local extern of
cluster_undo_segments_max_per_instance (the new cluster_guc.h include
already declares it) to clear a shadowed-declaration finding.
- cluster_undo_alloc.h now includes cluster/cluster_scn.h (latent).
Default path (coherence off) is byte-for-byte the pre-D2-2 behaviour.
Tests: test_cluster_undo_gcs U3-U11 (shared-root branch per mode,
materialized-never-migrates, intent derivation); intent parameter
threaded through test_cluster_undo_buf / test_cluster_tt_durable stubs.
cluster_unit 159/159, PG 219/219, cluster_regress 13/13, and the
undo/recovery/CR TAP set (018/070/213/215/247/248/253) all green with
coherence off.
Spec: spec-5.22b-undo-block-gcs-integration.md
…c-5.22b D2-3)
Add cluster_undo_block_acquire_shared: a peer acquires a coherent shared
view of an owning instance's undo block through owner-as-master routing.
The owner (authority and holder) ships the current image over the reused
undo-TT fetch wire; the requesting peer consumes the shipped image and
never opens the foreign undo file. Admission is fail-closed on two ANDed
dimensions -- segment-generation anti-ABA and owner-incarnation epoch plus
durable coverage -- so any miss/DENIED/doubt leaves the caller's existing
fail-closed visibility boundary untouched.
Structure follows the pure-policy split so cluster_unit links the decision
core standalone:
- cluster_undo_gcs.c gains the pure predicates cluster_undo_grant_armed
(coherence x peer-mode gate), cluster_undo_grant_admissible (reuses
cluster_vis_live_authority_covers_policy and the D1 generation check),
and cluster_undo_grant_reader_pcm_mode/_transition (S via N->S).
- cluster_undo_gcs_grant.c (new) holds the runtime wrapper with its
heavy fetch/epoch/GUC dependencies, registered in the backend OBJS.
Everything is gated behind cluster.undo_gcs_coherence (default off), so the
default path is byte-for-byte unchanged. The master==self local fast path
is fail-closed in this increment pending its owner-incarnation self-check;
the live-owner peer read path is fully implemented.
test_cluster_undo_gcs gains U4/U6/U7/U9 (reader lock contract, epoch and
generation admissibility, coherence gate); its recipe links the pure
live-authority policy object. Runtime wrapper is covered by the later TAP
legs.
Local gates (coherence off): cluster_unit 159/159 (undo_gcs 9/9), PG
219/219, cluster_regress 13/13, undo/CR/recovery TAP 6 files/160 tests,
clang-format-18 clean, comment-headers clean.
Spec: spec-5.22b-undo-block-gcs-integration.md
…ves (spec-5.22b D2-4) Owner-as-master write side, the dual of the D2-3 reader S-grant. - pure core (cluster_undo_gcs.c): writer PCM contract (X via N->X), SCN-only PI-discard coverage judge, and pure PI-holder targeting; all standalone unit-tested (test_cluster_undo_gcs U5, 9 -> 12 cases). - runtime (cluster_undo_gcs_grant.c): acquire_exclusive (the owner takes a local PCM X on its OWN undo block; a peer taking X on a foreign owner's undo fails closed -- single-writer invariant) and invalidate_peers (durable-write PI-discard broadcast keyed by the synthetic undo tag, reusing the shipped PI_DISCARD wire). - cluster_gcs_block.c: extract cluster_gcs_block_send_pi_discard_invalidate so the undo data plane reuses the wire + checksum instead of duplicating it (behaviour-preserving refactor; the gcs_block unit tests stay green). - cluster_undo_record.c: refresh the W3-wall contract comment + errhint (admit set unchanged; documents that a runtime-warm foreign undo read is served by the owner-shipped grant, never a local self-read). All coherence-gated (cluster.undo_gcs_coherence, default off) -> inert at the default. No live S/PI holder yet, so the runtime primitives are skeleton-ahead-of-consumer, forward-covered by the D2-7/D6 TAP. Spec: spec-5.22b-undo-block-gcs-integration.md (D2-4)
Owner-as-master serve-gate: an undo resource's master IS its owner, so the owner-death / remaster window is the undo shard's recovery phase. - pure core (cluster_undo_gcs.c): cluster_undo_serve_allowed(owner_alive, remaster_in_progress) = owner_alive && !remaster_in_progress. The owner-as-master mirror of cluster_gcs_block_phase_for_tag's CSSD-DEAD + recovery-in-progress fence and cluster_hw_serve_allowed's pure shape, keyed on the owner rather than a hash static master. Standalone unit-tested (test_cluster_undo_gcs U8, 12 -> 13 cases). - runtime (cluster_undo_gcs_grant.c): acquire_shared applies the gate on the peer-reads-foreign branch BEFORE the fetch -- a converged-DEAD owner or an in-flight remaster episode denies DENIED_RESOURCE_RECOVERING and fails closed, instead of stalling on a doomed fetch to a dead node. Dead-owner SERVE from shared storage is a later increment; this proves only the fail-closed deny (never a false-resolve). Undo folds into the reconfig episode by reading its state (cluster_cssd_get_peer_state + cluster_grd_recovery_in_progress), never by joining the hash-shard master map (undo never hash-routes). So cluster_grd.c and cluster_gcs_block.c are unchanged. All coherence-gated (cluster.undo_gcs_coherence, default off) -> inert at the default. Spec: spec-5.22b-undo-block-gcs-integration.md (D2-5)
Register three owner-as-master undo-block grant-plane wait events so an operator can see a backend blocked in the undo GCS data plane: UndoBlockGrantWait / UndoBlockInvalidateWait / UndoBlockRemasterWait. Full 4-place wait-event symmetry: the WaitEventCluster enum + name lookup (pg_stat_activity.wait_event), plus the pg_stat_cluster_wait_events roster (registry + count). The Undo class grows 8 -> 11 in the enum and the surfaced roster grows 4 -> 7 (the I/O-internal CR_CONSTRUCT / TT_DURABLE_IO / BUF_FLUSH / EXTENT_CLAIM stay off the roster, as before). The PCM convert/downgrade waits an undo grant blocks on reuse the PCM class (WAIT_EVENT_PCM_BLOCK_CONVERT_WAIT/DOWNGRADE), so no new events for those. The three new events are register-ahead of their report sites, which land with the grant-path consumer (the static registration table convention, cluster_views.h). Update-required snapshot tripwires bumped in lockstep: the per-stage acceptance count snapshots (118 -> 121), the two roster count tests (Undo 4 -> 7, total 118 -> 121), and the enum-range count (Undo 8 -> 11). Spec: spec-5.22b-undo-block-gcs-integration.md (D2-6)
…2b D2-6)
New ClusterUndoGcsShared shmem region (six atomic counters, no LWLock):
grant_shared / grant_exclusive / ship_bytes / invalidate_notify /
remaster_deny / local_fast_path. The owner-as-master grant primitives
bump them at their existing sites; dump_undo (pg_cluster_state,
category='undo') reads them lock-free -> the undo category grows 54 -> 60
rows. The counters are register-ahead of a live consumer (the grant
primitives have no live caller until the grant-path consumer lands), so at
rest every counter reads 0, but the surface is present + queryable now.
Region roster: +"pgrac cluster undo gcs" (t/020 roster + region count).
Also completes the wait-event count baselines the prior commit
(undo-block grant-plane wait events) moved 118 -> 121 but did not update
everywhere: the earlier commit's local check ran only the three view TAP
files, so ~20 other TAP files and the cluster_smoke regress snapshot that
pin the total wait-event count were left stale. Swept and bumped here
(pg_stat_cluster_wait_events total 118 -> 121 across t/012-118, t/203,
t/030, cluster_smoke.{sql,out}).
Unit-test stubs added for the new dump accessors + shmem register
(test_cluster_debug, test_cluster_shmem link cluster_debug.o /
cluster_shmem.o standalone).
Spec: spec-5.22b-undo-block-gcs-integration.md (D2-6)
The pi/cr recovery acceptance test pins per-category dump counter counts. The 'cr' category baseline (41) was stale: 16 more cr counters shipped after the last reconciliation -- spec-6.12i runtime-visibility rtvis_* / cr_server_undo_* / cr_server_verdict_*, spec-6.15 origin verdict -- all unconditional in dump_cr, so the deterministic count is now 57. Update the expected value + document the delta. This drift went undetected in CI because t/254 falls in the un-sharded t/249-272 nightly gap (its earlier stale value was "first caught by a local full run", per the existing comment). Test-only; no C change.
…ne TAP
Add t/255: a 2-node ClusterPair + shared cluster_fs behavioral TAP for the
owner-as-master undo data plane. cluster_unit (test_cluster_undo_gcs,
U1-U10) already covers the pure decision cores; this pins the one thing it
cannot -- the physical migration wired through the real allocator:
- coherence off (default): own runtime undo stays on the local DataDir,
the shared root holds none (inert / regression-safe negative control)
- coherence on: after a forced fresh segment, own runtime undo migrates
to the shared cluster_fs root (owner-as-master, observed end-to-end)
- the migrated shared-root undo write path is functional (DML + rollback)
- grant-plane counters stay 0 at rest -- the peer-really-reads-foreign
-undo consumer is forward-D6; asserted, not faked
Register t/255 in the nightly stage4-wal shard per the "every new t/ file
lands in a shard the same commit" rule. The broader un-sharded t/249-272
gap is pre-existing and tracked separately. Test-only; no C change.
Spec: spec-5.22b-undo-block-gcs-integration.md
Add the five-value cross-node xid->commit_scn verdict taxonomy and its two
pure mapping helpers, the structural layer D3 builds on.
new src/include/cluster/cluster_undo_verdict.h
ClusterUndoVerdictKind {UNKNOWN_FAIL_CLOSED=0, COMMITTED_EXACT=1,
COMMITTED_BOUND=2, ABORTED=3, IN_PROGRESS=4} + ClusterUndoVerdictResult
{kind, commit_scn, wrap}. kind alone distinguishes EXACT vs BOUND
(no is_bound side axis): a switch handling only EXACT lets a BOUND
fall to a fail-closed default, so a horizon bound is never stamped or
cached as an exact commit SCN. The zero value is the fail-closed
sentinel.
mod cluster_runtime_visibility_policy.c
cluster_undo_verdict_from_wire_page(): CP5 verdict page -> taxonomy
(unusable page -> UNKNOWN; EXACT -> EXACT; BELOW_HORIZON -> BOUND;
ABORTED -> ABORTED). The read_scn admissibility of a BOUND needs
scn_time_cmp and is deferred to the consumer, keeping this policy
object dependency-free (as cluster_vis_undo_verdict_page_usable does).
cluster_undo_verdict_from_block_proof(): CP3 proof -> taxonomy
(COMMITTED -> EXACT{scn,wrap}; ABORTED -> ABORTED; NONE -> UNKNOWN).
Both pure: no I/O, shmem, or elog.
new test_cluster_undo_verdict.c (U1-U3, standalone link against the
policy object) + Makefile registration.
Pure taxonomy layer only; not yet wired into the resolve path (D3-2+), so
this increment is zero behaviour change.
Spec: spec-5.22c-ttslot-verdict-shared-undo.md (D3-1, §2.1/§4.1)
…grant
Switch the CP3 single-block TT read in the runtime-visibility resolver from
the 6.12i best-effort authority-less fetch to the D2 owner-as-master coherent
S-grant, gated on cluster.undo_gcs_coherence + peer-mode.
cluster_runtime_visibility_try_resolve_remote (CP3 leg):
coherence on + peer -> cluster_undo_resid_encode(origin, seg, block0, gen)
+ cluster_undo_block_acquire_shared() (owner-as-master routing + D2-5
serve-gate + epoch/generation admissibility; the owner ships the image,
this peer never opens the foreign undo file). The grant result's authority
triple is reconstructed so the version-coverage gate D2 deferred
(admissibility used anchor=Invalid) is applied here: hwm >= this tuple's
anchor_lsn, else fall to the CP5 verdict leg. coherence off keeps the
best-effort fetch byte-for-byte (zero regression); every miss still falls
closed to CP5 and ultimately 53R97.
Amendment-1: the requester has no independent source for the owner's current
segment generation, so it encodes rid.field3 == gen (its own known value).
D2's generation_matches then compares two equal caller values and is
structurally true -- NOT the anti-ABA gate on this path. The real anti-ABA
is the slot-level xid+wrap positive proof (content-based; a recycled segment
changes the slot wrap -> proof NONE -> fail-closed).
test_cluster_undo_verdict.c U4/U6/U11: pin the pure contracts the CP3 wiring
composes (D1 resid encode/master, the version-coverage policy, the D2 armed
gate). The orchestration wiring is backend-heavy and forward-covered by the
D3-7 / D6 TAP; links the D1 + D2 objects with a cluster_node_id stub.
Spec: spec-5.22c-ttslot-verdict-shared-undo.md (D3-2, §2.3/§3.5/§4.1)
…elf local Add the D3 cross-node verdict entry the D6 consumer will call, with both routing legs, the CP5 serve-gate precheck, and the shared origin/self resolve. cluster_undo_verdict_resolve (D3-3, cluster_runtime_visibility.c): the entry point. master==self routes to the local durable resolve; master!=self to the CP3 owner-as-master S-grant + CP5 origin verdict, whose (ok, committed, scn, is_bound) outcome is folded into the taxonomy by the new pure cluster_undo_verdict_from_resolve() (Amendment-2: is_bound collapses into the COMMITTED_BOUND kind, so a consumer never sees the side axis). Every unproven path returns UNKNOWN_FAIL_CLOSED (caller keeps 53R97, Rule 8.A). rtvis_resolve_own_xid (D3-4): the owner resolving its OWN xid uses its own durable TT + own CLOG, never acquire_shared (which is fail-closed-forward-D6 for master==self). It reuses cluster_lms_undo_verdict_fill_page -- extracted from lms_undo_verdict_serve (behavior-preserving; the origin serve still calls it after zeroing the full BLCKSZ wire page and co-sampling authority) -- so a node's self answer and its served answer over one xid can never diverge. A COMMITTED_BOUND then applies the same read_scn admissibility (observe horizon, admit iff read_scn at/after it) the foreign path applies. CP5 serve-gate precheck (Q9, rtvis_try_origin_verdict): under the coherent regime a dead owner / in-flight remaster fails closed early with no doomed verdict request on the wire, symmetric with the D2-5 acquire_shared gate; serve = D4. Off the coherent gate the 6.12i best-effort path is unchanged. test_cluster_undo_verdict.c U13: from_resolve folding (ok/committed/is_bound -> kind; BOUND never EXACT). cr_server_policy 7/7 confirms the extraction is behavior-preserving; the origin/self verdict e2e is D3-7 / D6 TAP. Spec: spec-5.22c-ttslot-verdict-shared-undo.md (D3-3/D3-4, §2.2/§2.4/§3.1)
D3-5 (8.A gate closure): pin the fail-closed boundaries of the verdict layer.
Every unproven / malformed / ambiguous outcome must land UNKNOWN_FAIL_CLOSED,
never a guessed commit.
test_cluster_undo_verdict.c U7/U8/U10/U12:
U7 malformed verdict page (EXACT w/o scn, BELOW_HORIZON w/ stray scn,
unknown kind, wrong magic) -> UNKNOWN.
U8 segment-generation gate is structural, not anti-ABA (Amendment-1):
rid.field3==gen makes generation_matches structurally true; it is a
real comparison (a genuine mismatch fails closed) that D3 makes
always-true by construction, so it is no independent proof.
U10 IN_PROGRESS is reserved (value 4) but never produced -- every mapper
folds an unproven-terminal outcome to UNKNOWN, never IN_PROGRESS.
U12 the slot-wrap positive proof is the real anti-ABA even under a
structurally-true generation gate: a proof NONE -> UNKNOWN. The proof
truth table itself is covered by test_cluster_runtime_visibility (the
pure helper D3-2's CP3 reuses verbatim), not duplicated here.
No new production code: the fail-closed behaviour is by construction in
D3-1..D3-4; these pin it.
D3-6 (observability): the master==self local resolve now counts its terminal
outcome on the same rtvis resolve counters the foreign path uses
(committed / aborted / fail-closed), so both entry legs are observable via the
existing dump. No new shmem region or wait event (reuses D2-6).
cluster_unit test_cluster_undo_verdict 11/11.
Spec: spec-5.22c-ttslot-verdict-shared-undo.md (D3-5/D3-6, §3.2/§4.1/§1.2)
… undo verdict
A fresh joiner reading a catalog tuple that a peer committed before the joiner
came up finds an empty local TT overlay, so the remote-ITL-ref classification
left {REMOTE, UNKNOWN} on the miss and the visibility consumer failed closed
(53R97 -> false-invisible). Route that branch through the shared-undo
cross-node verdict (cluster_undo_verdict_resolve) so the owner-as-master
resolves the tuple's commit outcome.
- cluster_visibility_resolve.c: widen the fresh-remote-ITL-ref branch to ask
the origin verdict when cluster.crossnode_runtime_visibility is on and the
origin is a peer; every unproven leg keeps the unchanged 53R97 fail-closed
boundary.
- cluster_visibility_verdict.c: pure helpers cluster_vis_from_undo_verdict()
and cluster_vis_freshref_origin_decision() (derivable-time integrity check;
an underivable origin still asks the verdict rather than failing closed).
- KIND_UNDO_VERDICT wire: carry an authoritative (physical-binding) flag. A
fresh-ref request whose origin is the page's physical ITL binding lets the
origin serve from its own durable TT + CLOG, skipping the stripe self-check
precheck; derived (recycled) requests keep cluster_xid_is_mine unchanged so
the overlapping-value guard is not relaxed for them. The authoritative path
does not weaken the positive proof (RESOLVED_SCN still requires
TransactionIdDidCommit + wrap-suspect).
- cluster_cr.c / cluster_debug.c: two cr counters
vis_freshref_verdict_{resolved,failclosed}_count (cr category 57 -> 59).
- tests: unit test_cluster_vis_undo_verdict_map (U1-U10); 2-node e2e
t/359_cluster_5_22f_seed_visibility.pl; sync the six cr category-count
assertions 57 -> 59.
Local gates: cluster_unit 161/161, PG 219/219 (cassert), cluster_regress
13/13, cr/verdict/recovery TAP band green. t/347 fails a pre-existing
same-epoch-restart recovery-ownership path on this host (reproduced on the
base commit without these changes, node restart FATAL "crash recovery of a
cluster-coherent page requires a recovery-ownership window"); merge-window
fast-gate/nightly gate it.
Spec: spec-5.22f-shared-catalog-seed-visibility-consumer.md
…22d)
Pure lowest-fresh-survivor election + live-snapshot wrapper for electing a
deterministic temporary serve authority when an undo resource's owner is
dead/absent (spec-5.22d D4-1, Route B dead-owner block0 verdict serve).
- cluster_undo_authority.{c,h}: pure cluster_undo_authority_decide() —
no shmem/GRD/GUC deps; mirrors cluster_grd_master_map_remaster survivor
rule (declared minus non-fresh). Fail-closed edges: epoch mismatch ->
RECOVERING; owner not dead-decided (生死未定) -> UNKNOWN; no fresh
survivor -> RECOVERING; malformed owner -> UNKNOWN. Never elects a
non-(declared AND fresh) node (约束 #2 at derivation layer).
- cluster_undo_authority_snapshot.c: wrapper projects membership SSOT
(cluster_membership_get_state) + L420 heartbeat-freshness
(observed_fresh_alive, NOT record existence) + current reconfig epoch
into the pure decide(). TAP-covered (D4-8).
- test_cluster_undo_authority: 8 unit cases RED->GREEN (U1 lowest-survivor,
U2 owner-live, U7 undecided, epoch-mismatch, no-survivor, skips-lower,
requires-declared, invalid-owner).
Base = D6 tip 36098ec (spec-5.22b-undo-gcs, unpushed F1 window).
Identity (canonical owner, D1 cluster_undo_resid_master) kept separate from serve DESTINATION (owner-live -> owner; dead -> elected authority). Pure cluster_undo_route_decide() maps (owner, epoch, status, authority) -> route with NO branch to a GRD hash-master: a lookup miss fails closed to destination -1, never falls back to hash routing (约束 #2). Heavy glue cluster_undo_serve_authority(resid, epoch) composes D1 owner + D4-1 wrapper + the mapper. 4 route unit cases (U3a-d) RED->GREEN; 12/12 total.
Add a third ClusterUndoPathIntent value, CLUSTER_UNDO_PATH_RUNTIME_SHARED_AUTHORITY_BLOCK0, so a survivor node can resolve a foreign (dead) owner's durable block0 from the shared cluster_fs root, read-only. cluster_undo_path_uses_shared_root() now admits the new intent under the same peer-mode + undo_gcs_coherence arm gate as own RUNTIME_SHARED (both off => inert local DataDir). cluster_undo_path_resolve()'s owner/self assert is relaxed to permit AUTHORITY_BLOCK0 with owner != self only; a plain RUNTIME_SHARED foreign owner and AUTHORITY_BLOCK0 with owner == self are still rejected. The read stays block0-only by call-site contract. Adds U6 (test_undo_gcs_path_authority_block0_mode_branch) to test_cluster_undo_gcs.c: the AUTHORITY_BLOCK0 arm truth table (14/14). Spec: spec-5.22d-undo-dead-owner-verdict-serve.md
…rity serve (spec-5.22d) Wire the dead/absent-owner serve authority into the cross-node undo verdict entry. Under the armed data plane (peer-mode + cluster.undo_gcs_coherence), cluster_undo_verdict_resolve now routes a foreign origin by owner liveness BEFORE asking the owner: - owner live: the CP3 S-grant + CP5 origin-verdict path, byte-for-byte unchanged; - owner dead, self is the elected authority: serve the verdict off the owner's durable shared block0 (AUTHORITY_BLOCK0 read-only intent, not an S-grant), admissible only under the three-way coverage AND -- claimed-at-current-epoch, block0 fully read, slot xid+wrap positive proof; any term false keeps the 53R97 fail-closed boundary; - owner dead with a peer/no authority, or liveness unproven: fail closed. Peer-authority wire serve lands with D4-5/D4-6. Never the native CLOG/hint path. New pure helpers cluster_undo_authority_serve_decide (route -> consumer action) and cluster_undo_authority_coverage_ok (three-term shape pinned by the U5 truth table); 6 new unit tests (18/18). New cr counters undo_authority_serve_hit / undo_authority_fail_closed (dump-exposed; cr category 59 -> 61, baselines reconciled in t/215, t/216, t/254). Gate off => chain inert; t/359 + t/346 stay green (the live-owner path is unchanged under the armed chain). Spec: spec-5.22d-undo-dead-owner-verdict-serve.md
…counter (spec-5.22d) Extract the dead-owner block0 serve into ONE prove core, cluster_undo_authority_block0_prove (heavy snapshot object): input validation, the claimed-at-current-epoch check, the AUTHORITY_BLOCK0 shared read, the slot xid+wrap positive proof, and the coverage three-way AND all live in a single implementation, so the requester self-serve leg (D4-4) and the wire-served LMS authority leg (D4-6) can never diverge over one (owner, segment, xid) -- the same discipline as cluster_lms_undo_verdict_fill_page on the live-owner path. The undo_authority_* counters are owned by the core, so no consumer can forget the observability contract. rtvis_authority_serve_block0 becomes a thin wrapper (route-OK assert + rtvis resolve counter fold); behavior unchanged, t/359 + t/346 stay green. Add the third serve counter, undo_authority_epoch_stale_reject: the epoch axis of a refusal, bumped in addition to fail_closed (which stays the total-refusal count). cr category 61 -> 62; baselines reconciled in t/215, t/216, t/254. Spec: spec-5.22d-undo-dead-owner-verdict-serve.md
A dead undo owner's xid verdict can now be served over the wire by the deterministically elected survivor authority (Route B), activating the D4-4 requester peer leg that until now failed closed. Wire (request zero-ABI, reserved/empty fields only): - reserved_0[6] VALUE 4 = authority verdict request (kind-4); the dead owner rides in the previously-empty tag.relNumber as owner+1 (0 stays absent; kinds 1/2/3 now strictly require an empty carrier). - Reply provenance: ClusterGcsUndoVerdictPage version 2 marks authority-served; the v1 strict ==1 gate refuses it (old requesters fail closed) and the authority leg accepts ONLY version 2, refusing the BELOW_HORIZON kind outright (the block0 prove has no horizon leg). - HELLO capability bit 0x2 (UNDO_AUTHORITY_SERVE_V1), advertised unconditionally as a protocol capability; requesters never route kind 4 to a peer without it and do NOT re-run the election. Serve side (LMS): triple check before a byte is answered -- request epoch == current epoch, re-derived authority for (owner, current) is self, then the shared block0 prove core (the same implementation the D4-4 self leg runs, so wire-served and self answers can never diverge). Requester side: full reply binding -- sender == elected authority AND reply epoch == stamped epoch EXACTLY (transport HC100 >= is only a pre-filter) -- plus the version-2 structural gate/mapper. The two verdict fetch wrappers now share one transport core (single-stamped epoch); acceptance policies stay separate per leg. Tests: serve_decide truth table gains the PEER_BLOCK0 arm; new unit truth tables for the kind-4 flag, owner tag roundtrip, version-2 usable/fill/mapper closure, and the reply binding (wrong-sender and epoch±1 negatives). cluster_unit 162/162; TAP regression band t/359 t/346 t/255 t/215 t/216 t/254 all green (live-owner path byte-identical: verdict_served=8/denied=0 unchanged). Spec: spec-5.22d-undo-dead-owner-verdict-serve.md (D4-5 rest + D4-6)
… (spec-5.22d) The dead-owner authority verdict prove is widened from a single ref-segment block0 read to the owner's COMPLETE durable shared segment set — the CP5 complete-scan contract translated to durable state. D4-8's first true e2e proved the TT-slot cursor and the undo-record cursor roll independently, so a slot living outside the ref's segment made every such verdict fail closed forever (availability gap; fail-closed direction was already safe). Prove core (one implementation, self + kind-4 wire legs both covered): - enumerate <shared_data_dir>/pg_undo/instance_<owner-1>/ — the single namespace the shared undo resolver ever writes; complete-or-refuse: any enumeration break, non-canonical entry name (strict parse + re-render equality; seg_0 included, aliases and over-range ids poison), unreadable or unparseable block0 refuses the WHOLE verdict, never skip-and-continue; - set-wide UNIQUE terminal match or refuse: 0-match and non-terminal keep the fail-closed shape, >=2 matches anywhere void the kept evidence; - claim epoch checked at entry AND re-checked after the scan window (a mid-scan reconfig voids the enumerated set's writer-exclusion argument); - writer exclusion is structural: the dead owner is dead-decided + write- fenced (revival forces an epoch bump), survivors' only foreign-owner intent is the read-only AUTHORITY_BLOCK0, and no undo unlink/rename path exists (the future D5 reclaimer must inherit this invariant). Pure layers: cluster_vis_tt_block_xid_scan core under the byte-identical positive-proof wrapper (nmatch reported, POISONED distinct), and the scan-fold aggregate with its admissibility predicate — both truth-table unit-pinned. Two new attribution counters (scan_incomplete_reject / multi_match_reject; cr 62->64, baselines updated) plus the D4-8 L3 injection point cluster-undo-authority-block0-prove (registry 158->159). TAP t/369 (renumbered from the workspace t/360 per the collision ruling): L1 committed rows served post-mortem with serve_hit attribution and materialized==0; L2a regular-abort xid stays in-doubt fail-closed (no durable ABORTED stamp exists outside 2PC/own-recovery); L2b 2PC-aborted row never serves visible (the positive ABORTED-serve e2e is unreachable here — ROLLBACK PREPARED reverts synchronously — and is forwarded to the D7 crash-window injection); L3/L3b injection-forced prove refusal with counter delta + unarmed control. Spec: spec-5.22d-undo-dead-owner-verdict-serve.md (Amendment A1)
…ap fix (spec-5.22e)
The pure decision core of the cluster-wide undo retention brake: fold the
local spec-3.12 horizon and every required MEMBER peer's report view into
a {scn, epoch} recycle floor. A floor is only proven when each required
peer's report is capable (current-connection UNDO_HORIZON_V1), present,
seqlock-stable, current-epoch, well-formed (valid SCN + interval within
[100,60000]ms), monotone (a same-epoch regression latch stalls: the peer
contradicted its accepted lower bound, so the older higher value must not
be consumed), and fresh (3 x max(sender, local) interval, receiver-local
clock, future recv treated stale). Anything unproven STALLS with
deterministic first-fail attribution (reason + blame node) -- deliberately
NO fallback to local-horizon recycling on any edge: a required peer
without capability may be an old binary still holding old snapshots.
Empty required set folds to the local horizon (single node / cold
formation stays on today's path). All SCN comparisons go through
scn_time_cmp.
Unit truth table U1-U16 (19 tests) pins every arm, including ghost-slot
ignore, DEAD/REMOVED drop, blame ordering, and the reason-name table.
Also closes the SetTransactionSnapshot publish gap found by the spec's
S3.0 audit: the imported FirstXactSnapshot was registered without an
immediate cluster_recompute_proc_read_scn(), so the exporter could
release its pin before the importer's first push published the inherited
read_scn -- a window where this node's retention floor (and its future
horizon report) sat above a live imported snapshot. Mirror the
GetTransactionSnapshot register path.
Spec: spec-5.22e-undo-cluster-retention-horizon.md (D5-1)
…ish + seqlock slots (spec-5.22e)
The report plane of the cluster-wide undo retention brake:
- New PGRAC_IC_MSG_UNDO_HORIZON (36; LMON-only producer, per-peer p2p,
never broadcast) carrying {epoch, horizon_scn, sender_interval_ms}.
The LMON tick samples the spec-3.12 own-instance floor bounded by the
node's minimum future-snapshot origin (consistent_scn when MRP-active:
the no-reader clock fallback would over-report an ADG standby whose
future snapshots read far below the clock) and sends it to every
declared peer whose CURRENT connection advertised the new HELLO
capability UNDO_HORIZON_V1 (0x4) -- an old peer treats an unregistered
msg_type as a peer-level failure and closes the connection, so no
capability means no send, never a reconnect storm.
- Capability state is now connection-bound (Q1' amend): the tier1 close
funnel clears the peer's advertised HELLO capabilities
(cluster_sf_note_peer_disconnected), so no consumer -- horizon
sender/fold, D4 authority routing, smart fusion -- trusts a stale
capability across a reconnect window. Clearing is uniformly the
conservative direction for every existing consumer.
- The receive handler validates strictly before publishing (exact
length, declared non-self sender, valid SCN, interval within
[100,60000]ms) and enforces same-epoch report monotonicity: a
regressing report is rejected AND latches a violation flag -- the
previously accepted higher value must not be consumed once the peer
has contradicted it (the fold stalls on REGRESSION until a conforming
report clears the latch). Invalid frames bump
undo_horizon_wire_reject and are never published: the slot ages into
a fold stall.
- Reports live in a new shmem region 'pgrac cluster undo horizon' of
per-peer seqlock slots (single LMON writer: odd/even seq + write
barriers; cleaner-side sampling double-reads with bounded retries,
torn => the fold's TORN stall). cluster.shmem_max_regions default
raised 80 -> 96 (region +1 with headroom).
- Two D5-7 injection points registered (registry 159 -> 161):
cluster-undo-horizon-report-drop (stall leg) and
cluster-undo-horizon-epoch-fence (F-D2 fence leg, consumed by D5-3).
- Baseline sweep (L462): t/015 (161 rows + name list), t/020 (region
count/list + L12/L13 96 + L15 161), t/021-023 (region count + L11).
t/021-023's L11 was still asserting 158 -- a latent stale baseline
from the D4 bump that only swept t/015; trued up to 161 here.
Band t/015+t/020-023 rerun: 100/100 PASS.
Spec: spec-5.22e-undo-cluster-retention-horizon.md (D5-2)
…nce (spec-5.22e)
The undo cleaner pass input changes from the bare spec-3.12 local horizon
to the cluster floor {scn, epoch}: sample the per-peer seqlock report
views + the required MEMBER set (epoch-consistent walk) and run the pure
fold. An empty required set folds to the local value (single node / cold
formation = today's path, byte-equal behavior). Any unproven required
peer STALLS the pass: the whole recycle stage -- shmem TT slot GC
included (Q8: one horizon input) -- is skipped, with reason/blame
attribution (undo_horizon_stall / undo_horizon_peer_stale counters +
LOG-once mirroring the pinned-horizon pattern, re-armed on recovery).
There is deliberately no fallback to local-horizon recycling on any
stall reason (Q3'').
F-D2 epoch fence: the folded floor's epoch is re-verified at every
recycle mutation point --
- cluster_tt_slot_gc_current_pass: before each slot CTS_FREE (returns
false = pass aborts);
- cluster_undo_segment_advance_recyclable: INSIDE the lifecycle_lock,
before the WAL/pwrite/fsync three-step (new result
CLUSTER_SEG_RECYCLE_EPOCH_CHANGED; the fence sits in the caller's
locked region rather than widening try_mark_recyclable's signature
-- same guarantee, smaller ABI ripple).
A mid-pass reconfig epoch bump (a join) voids the folded floor's member
coverage, so a tripped fence aborts the WHOLE pass immediately
(undo_horizon_pass_abort counter), never just waits for the next pass.
The fence consults the cluster-undo-horizon-epoch-fence injection point
so the t/370 L6 leg can force the tripped arm deterministically.
Unit ripple: gc pass callers updated (test_cluster_tt_slot_allocator
+ fence/register/tick link stubs for standalone binaries). Full
cluster_unit suite + PG219 regression pass.
Spec: spec-5.22e-undo-cluster-retention-horizon.md (D5-3)
…ity hard gate (spec-5.22e)
Closes the S3.0 condition (4) hole (F-C2/F-D3): a snapshot must either be
covered by the retention fold or be unable to reference foreign undo at
all. JOINING only blocks writes (xact.c INV-J9), reconfig interrupts
absorb xid-less read-only transactions, and snapshots follow the storage
gate -- so a not-yet-admitted node COULD run cross-node reads that no
owner's required set covers, and a mere at-consumption MEMBER check would
re-admit RR snapshots taken during JOINING.
cluster_undo_horizon_read_admission_enforce, called on the FOREIGN arms
of every cross-node undo consumption entry (cluster_undo_verdict_resolve
origin!=self; try_resolve_remote entry -- which also covers the single
acquire_shared caller, contract-noted there):
- self not MEMBER -> 53R60 (retry-safe; local reads ungated)
- snapshot read_epoch below self admission epoch -> 53R60 with a
new-snapshot hint (pre-join snapshots are refused FOREVER, F-D3(3))
- any MEMBER peer without the current-connection UNDO_HORIZON_V1
capability -> false (mixed-version cluster: an old-binary owner has
no brake, so a new consumer must not trust any owner retention);
the caller keeps its UNKNOWN/53R97 fail-closed shape, LOG-once +
admission counter here. Upgrade-window contract self-enforcing on
new binaries (S3.3 matrix; old-old outside guarantee scope).
Admission epoch captured exactly at the cluster_membership_set_state
choke (self, ->MEMBER, prev!=MEMBER): bootstrap formation, online join
and rejoin all funnel through it, so a founding member's post-admission
snapshots are never mis-refused after unrelated epoch bumps (L470).
Snapshot identity cross-checked against the resolver read_scn; any
unprovable shape refuses conservatively.
HELLO capability baseline sweep: wire[36] 0x02 -> 0x06 and the
capability-gate truth table gain the unconditional 0x4 bit
(test_cluster_ic). Link stubs for standalone membership/reconfig unit
binaries. Full unit suite + PG219 pass.
Spec: spec-5.22e-undo-cluster-retention-horizon.md (D5-8)
…5 brake observability (spec-5.22e) D5-4: the undo path intent/ownership pairing (plain RUNTIME_SHARED iff own instance; AUTHORITY_BLOCK0 iff foreign) was Assert-only -- a release build would silently resolve a violating pair into the shared tree. It is a writer-exclusion pillar for both the D4 complete-scan prove and the D5 retention brake, so it now fails closed in production too (ereport ERROR, DATA_CORRUPTED; resolvers run outside critical sections per the 53R9D discipline). The D4 prove-core inscription is updated: spec-5.22e RULED retention stays own-instance (dead-owner segments frozen); a future cross-node reclaimer (D7+) must revise A1.1-bis first. D5-5: pg_cluster_state 'undo' category grows 8 rows -- the five brake counters (horizon_stall / horizon_peer_stale / horizon_pass_abort / horizon_wire_reject / horizon_admission_refuse), the last proven floor gauge (horizon_last_floor_scn), a per-peer report summary line (horizon_peer_reports: valid/cap/epoch/scn/age/interval/regression per declared peer), and the previously accessor-less cleaner_header_tt_slots_below_horizon inventory counter. No TAP asserts an 'undo'-category row count (key-specific asserts only), so no baseline sweep beyond the test_cluster_debug link stubs. Spec: spec-5.22e-undo-cluster-retention-horizon.md (D5-4/D5-5)
…ed cap reset; D5-7 t/370 (WIP, substrate-gated) (spec-5.22e) Two real fixes uncovered while wiring the D5 2-node brake TAP: 1. cluster_sf_dep sized the WHOLE region to zero under the default cluster.smart_fusion=off -- but the header carries the HELLO capability store that the undo-horizon sender/fold, the D4 authority-routing gate AND smart fusion all read. A zero-size region left ClusterSfDep == NULL, so every capability query silently returned false. The header (peer_capabilities + lock + counters) now exists in every cluster build; only the dep-vector slot array stays smart-fusion-gated. (D4's peer-authority query has the same latent dependency but its 2-node e2e only covers the self-authority leg, so it never noticed.) 2. cluster_ic_tier1_close_peer now clears the peer's HELLO capabilities only when tearing down an ESTABLISHED fd (Q1' amend), scoped inside the fd guard so a failed-dial / tie-break close does not wipe a surviving connection's capabilities. t/370 (D5-7) lands as WIP: legs L2 (stall), L6 (epoch fence), L3 (member-drop), L5 (read admission) exercise machinery that works on the current one-directional-HELLO substrate, but L0/L1 (bidirectional brake) need each node to know its PEER's capability -- and the tier1 mesh only learns capabilities one-directionally (the acceptor learns the dialer's; the active dialer never receives the peer's HELLO -- "No HELLO_ACK", cluster_ic_tier1.c). Making the brake work bidirectionally requires a symmetric HELLO exchange (spec-2.2 substrate); that lands in a separate focused change (user-approved direction) before t/370 is un-WIP'd. Unit suite 163/163 + PG219 219/219 stay green (the substrate gap only surfaces in the 2-node e2e). Spec: spec-5.22e-undo-cluster-retention-horizon.md (D5-7 WIP)
…CAPS_REPLY
The tier1 mesh performs a one-way HELLO (dialer -> acceptor), so only
the acceptor ever learned the peer's capability word; the dialer's view
of every peer it dialed stayed permanently empty, starving every
capability consumer on the dialing side. Add an additive, capability-
gated reply so both directions learn, with zero change to the HELLO
64-byte wire format and zero change to the active handshake sequence
(send HELLO -> CONNECTED, no wait state).
- HELLO capability bit 0x8 (CAPS_REPLY_V1): meta capability advertised
unconditionally, meaning this binary registers PEER_CAPS_REPLY and
can receive it.
- msg_type 37 (PEER_CAPS_REPLY, LMON-only, p2p): after verifying a
dialer's HELLO the acceptor replies with its own standard 64-byte
HELLO -- only when the dialer advertised the meta bit, so an old
binary is never sent a msg_type it would reject by closing the
connection. A missing reply just leaves the peer's capabilities
UNKNOWN, which every consumer treats as fail-closed; it never
triggers reconnect or membership change.
- Dialer-side handler revalidates the embedded HELLO (length, parse,
versions, cluster_name, sender identity) and drops + counts on any
mismatch -- never a close.
- Capability records become generation-bound ({bits, generation,
valid}; generation = the peer's reconnect_count while the connection
was established). The close funnel only invalidates the record whose
generation matches the closing connection, so defensive closes of
failed dials / duplicate-connection tie-breaks can no longer wipe the
surviving connection's record. RDMA passes generation 0: its CM
lifecycle never consumed the tier1 clear, and RDMA HELLO exchange is
already bidirectional via CONNECT_REQUEST/ESTABLISHED private_data.
- Epoch-recheck resend: a rejoining acceptor's one-shot reply is
stamped with its still-stale epoch and gets stale-epoch dropped by
the survivor with no retry anywhere. Remember per-peer that a reply
was sent and at which epoch; the recv drain resends after every local
epoch advance (idempotent on the receiving store, no-op at equal
epochs).
- Test-only GUC cluster.ic_suppress_caps_reply (PGC_SIGHUP, default
off) simulates a pre-amendment binary on both sides for the rolling-
upgrade compat legs.
- Observability: pg_cluster_state ic/peer_capabilities (per-peer
bits/gen/validity) + ic/caps_reply_reject_count.
Tests: unit hello meta-bit gate + reference bytes 0x0E + ClusterSfPeerCap
generation matrix; t/386 3-node directed capability matrix + old/new
compat both directions + generation invalidation across reconnects
(29/29); t/370 L0/L1 bidirectional capability legs now pass.
Local gates: cluster_unit suite green; cluster_regress 13/13; PG
regress 219/219; TAP t/012 014 072 075-079 331 337 339 386 green.
Pre-existing t/369 and t/370 tail failures reproduce identically with
this change stashed (A/B verified); they belong to the D5 lane.
Spec: spec-2.2-interconnect-tcp-listener-lmon-phase1.md (additive
amendment); spec-5.22e-undo-cluster-retention-horizon.md (D5 prereq)
…ation is not the never-admitted sentinel The D5-8 read-admission gate stored the epoch at which this node last became MEMBER directly, with 0 doubling as the never-admitted sentinel. Cold formation admits at the initial epoch -- which IS 0 -- so on any freshly formed cluster the gate conflated an admitted member with a never-admitted node and refused every cross-node undo consumption with "snapshot predates this node's cluster admission" (all seven t/369 dead-owner authority-serve legs regressed: the positive serve leg, both 53R97 fail-closed shape asserts, their prove counters, and the unarmed control leg; t/370's post-rejoin admission leg failed the same way). Store (epoch + 1) instead: 0 stays the conservative never-admitted refusal, N+1 means admitted at epoch N, and the pre-join snapshot check compares read_epoch against the unbiased value. Verified: t/369 27/27 green again; t/370 L5 admission legs green. Spec: spec-5.22e-undo-cluster-retention-horizon.md (D5-8)
added 2 commits
July 11, 2026 13:18
Three test-side defects kept the L2-L6 tail red after the substrate and
admission fixes landed:
- Injection disarm used a bare ALTER SYSTEM RESET, but colon-typed arms
(':skip') survive the not-in-list reclaim -- the report-drop stayed
armed in LMON forever, so the L2 self-heal leg (and everything after
it) only recovered when L3 removed node1 from the required set ~100s
later. Disarm now re-arms the point to ':none' explicitly before
clearing the GUC (t/338 precedent).
- L3 churned before sampling floor_pre while the dead peer's report was
still inside its freshness window, so the sampled floor already
contained the churn-advanced clock and the post-drop self-only fold
pinned at exactly floor_pre forever. Sample first, churn after.
- Floor comparisons used raw gauge values, but the SCN wire encoding is
(8-bit node_id || 56-bit local_scn) and the fold's scn_time_cmp min
flips between node encodings across samples -- numeric comparison
across encodings is meaningless (and the L1 legs passed only while
the fold happened to stay on one encoding). Compare the 56-bit time
component via the new scn_time() helper.
Spec: spec-5.22e-undo-cluster-retention-horizon.md (D5-7)
sqlrush
force-pushed
the
stage7-rootcause1-integration
branch
from
7192f0b to
9607bd1
Compare
added 8 commits
July 11, 2026 18:54
authority_scan_owner_segments detected a mid-enumeration directory read break via errno after ReadDirExtended(dir, path, LOG). But ReadDirExtended calls ereport(LOG) before returning NULL on a readdir error, and the LOG output path can clobber errno -- so an EIO-truncated scan could read errno == 0, be judged "complete", and serve a definite (false-unique) verdict on a partial segment set. PostgreSQL's own ReadDirExtended comment notes a LOG-level error looks like end-of-directory, tolerable only for best-effort cleanup scans. Switch to ReadDir (ERROR elevel) so a readdir break throws instead of masquerading as EOF, wrap the enumeration loop in PG_TRY/PG_CATCH that folds any throw into the existing scan_incomplete coverage failure, and move FreeDir past PG_END_TRY so the AllocateDir handle is freed on every path (no leak on the LMS drain's error-swallowing envelope). Both the self reader leg and the wire LMS leg now fail closed identically; drop the unreliable errno probe. New injection point cluster-undo-authority-scan models the readdir fault; t/369 L3c arms it and pins the scan-incomplete fail-closed with a counter delta (53R97), distinct from the head refusal. Injection registry 167 -> 168. Spec: spec-5.22d-undo-dead-owner-verdict-serve.md (Hardening v1.0.1)
…n builds The undo resid derived primitives guarded their invariants with Assert only: decode / master / generation_matches asserted the class byte, and encode asserted the owner range. Under --disable-cassert these are no-ops, so a production build could decode / route a non-undo-class resid, or encode an out-of-range owner (truncated into the uint16 owner field), and silently misroute the undo authority. The consuming side (cluster_grd_lookup_master) already fails closed with an ereport; the producing side did not -- an asymmetry that only bites production. Replace the four no-op Asserts with real runtime ereport(ERROR) guards that fire in both cassert and production builds, using the new SQLSTATE 53R9R ERRCODE_CLUSTER_UNDO_RESID_INVALID (53R9Q hash_routed covers only the reverse misroute). All current callers pass validated undo-class resids and in-range owners, so the guards are defense-in-depth and never fire on a valid path. test_cluster_undo_resid gains a setjmp trampoline and four guard-fire cases requiring the ereport arm (rc 1), so the guard is exercised at runtime even in the cassert unit build; the two sibling tests that link the object gain elog link stubs. Spec: spec-5.22a-undo-block-resource-identity.md (Hardening v1.0.1)
…dict" guard A single-block CP3 positive proof classifies COMMITTED by matching on xid and reporting the slot's CURRENT wrap/scn -- it does not compare against the ref's expected wrap, so a recycled slot reused by the same raw xid (bumped wrap, an unrelated COMMITTED stamp) still reads COMMITTED with the recycled commit's wrap/scn. A single-block COMMITTED proof is therefore evidence, not a verdict; the anti-ABA belongs downstream (the D4 dead-owner complete-scan uniqueness, or the CP5 origin verdict leg's own-CLOG cross-check + wrap-suspect gate), and the fetch / fresh-ref fast leg breaks to the verdict leg (C1b) rather than concluding visible. Harden the ungated cluster_undo_verdict_from_block_proof banner to state it applies no anti-ABA of its own and must only be reached behind a caller that already did, and add test_ttproof_recycled_same_xid_reports_bumped_wrap_not_gated pinning the report-not-gate contract for the same-xid/bumped-wrap case (distinct from the recycled different-xid -> NONE case already covered). No behaviour change: the routing invariant is already locked e2e by the verdict-wire counter assertions in the 7.1a and 5.22f TAP legs. Spec: spec-5.22f-shared-catalog-seed-visibility-consumer.md (Hardening v1.0.1)
…ster_undo_horizon.c cluster_undo_horizon_cluster_floor guards its out-params with Assert(out_floor != NULL && out_reason != NULL && out_blame_node != NULL) then writes through them. PG's Assert has no noreturn attribute on the failure path, so cppcheck reads the four post-Assert out-param writes as redundant null checks (nullPointerRedundantCheck). Same false positive already suppressed for cluster_shmem.c / cluster_dl_lock.c / cluster_oid_lease.c; add the same per-file suppression to keep the policy consistent. Verified: run-cppcheck.sh reports 0 findings (matches baseline).
The D5 undo-horizon lane updated the injection-registry count in t/015 / t/017 / t/020-023 / t/030 to reflect D4-8 (+1 authority-block0-prove) and D5-7 (+2 horizon) but missed t/018 and t/024, which stayed at 164; and it added 8 undo-category brake-observability keys (horizon_stall/peer_stale/pass_abort/ wire_reject/admission_refuse/last_floor_scn/peer_reports + cleaner below-horizon) without refreshing t/214's undo-category row count (60 -> 68). These are the nightly foundation-core and stage3-core reds. Refresh all three to the current registry (168 injection points incl. this branch's cluster-undo-authority-scan; 68 undo rows). Verified locally: t/018/t/024 168 PASS, t/214 68 PASS.
…e value (was colliding with multi-verdict) spec-5.22f D6-7 introduced the AUTHORITATIVE single-verdict request sub-kind by reusing reserved_0[6] value 3 -- the value already assigned to the spec-7.1 D3-b undo-MULTI-verdict request. GcsBlockForwardPayloadIsUndoVerdictRequest then matched (2 || 3), so it accepted a multi-verdict request, and the block-forward handler's single-verdict branch (which runs before the multi branch) stole it. A cross-node multixact member serve was routed to the single-xid verdict path, which cannot resolve the widened MXID, so the requester fail-closed with 53R97 and never converged (t/359_mxid G5: green on main, red on the branch). Move the authoritative single verdict to the next free value 5; multi keeps its shipped value 3. Byte legend is now 1=undo-TT fetch, 2=derived verdict, 3=MULTI verdict, 4=dead-owner authority verdict, 5=authoritative single verdict -- all five request kinds mutually exclusive across every Is* predicate. Only this sub-kind (unshipped on main) moves; the wire value for every kind that main already ships is unchanged. Add test_forward_payload_undo_verdict_kinds_no_collision pinning the mutual exclusion (the pre-existing kind-4 test never checked the multi predicate, so it missed the collision). Verified: t/359_mxid 19/19, and the authoritative-flag consumers t/365 / t/369 / t/370 / t/359_seed all pass. Spec: spec-5.22f-shared-catalog-seed-visibility-consumer.md
…TA-plane reset does not wipe them Peer HELLO capabilities live in the shared ClusterSfDep store but were recorded AND cleared by whichever tier1 plane verified the HELLO or closed the fd. Since spec-7.2 tier1 is multi-plane (CONTROL = LMON, DATA = LMS workers), a transient DATA-plane worker reconnect participated in the caps lifecycle: a same-epoch DATA-plane conn-reset cleared the CONTROL-established capabilities, and the DATA reconnect could not re-send the CONTROL-only PEER_CAPS_REPLY (and the still-up CONTROL plane had no reason to), so the peer read NOCAP forever. The spec-5.22e D5-8 admission gate then fail-closed every cross-node undo access, so a cross-node write after a DATA-plane reset failed permanently (t/360 L5.5). Make the capability lifecycle a CONTROL-plane property: only the CONTROL plane records the peer's HELLO caps / sends PEER_CAPS_REPLY, and only a CONTROL-plane close clears the shared record. A DATA-plane worker only READS the store. A genuine peer loss still tears down the CONTROL connection, which clears as before, so the fail-closed safety is preserved. Verified: t/360 38/38 (RED->GREEN), and the caps / D5 / membership band t/386 / t/369 / t/370 / t/359_mxid / t/359_seed / t/365 / t/364 / t/367 all pass. Spec: spec-5.22e-undo-cluster-retention-horizon.md
…use-#1 integration lane Main advanced past the PR #40 base (b015369) with spec-7.4 D0/D1/D4 (commit-scn freshness + durable frontier + BOC observability, PRs #39/#41), so the 5.22 integration branch no longer merged cleanly. Bring current main in and reconcile: - injection registry: both lanes added points (5.22: authority-block0-prove + 2 horizon + this branch's cluster-undo-authority-scan; 7.4: cluster-boc-event -publish), so the merged total is 169. Resolved the count assertions in t/015/017/018/020/021/022/023/024/030 to 169 (M5 derived 336 -> 338) and merged cluster-boc-event-publish into the t/015 sorted name list. - src/test/cluster_unit/Makefile SIMPLE_TESTS: union of both lanes' custom-rule test exclusions (5.22 undo tests + 7.4's test_cluster_scn_frontier). Verified on the merged tree: registry band t/015-030 + t/068 pass, 7.4's t/387 durable-frontier passes, both root-cause-#1 fixes (t/359_mxid, t/360) still pass, and cluster_unit is 175/175 clean.
This was referenced Jul 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Integrates the root-cause-#1 shared-undo physical data plane chain (D1–D6) that enables cross-node undo block reads and transaction-verdict resolution under GCS. This is the last code gap for the 4-node
shared_catalogcold-formation write path.Consolidated into a single integration PR by maintainer direction; per-deliverable commit boundaries are preserved (no squash) so each deliverable remains independently reviewable. Merge target tree is byte-identical to the locally validated integration branch.
Deliverables (in dependency order)
spec-5.22a-undo-block-resource-identity.mdspec-5.22b-undo-block-gcs-integration.mdspec-5.22c-ttslot-verdict-shared-undo.mdverdict_resolveover shared undo (owner-as-master S-grant block0)spec-5.22f-shared-catalog-seed-visibility-consumer.mdspec-5.22d-undo-dead-owner-verdict-serve.mdspec-5.22e-undo-cluster-retention-horizon.mdSafety / compatibility
cluster.undo_gcs_coherence,cluster.crossnode_runtime_visibility, …). With defaults, the visibility/undo path is byte-identical to prior behavior — no change for single-node or PG-ecosystem use.Test plan
linkdb fast-gate CIall jobs green (PR gate)linkdb nightly full CIall matrix jobs green (ship gate)stage5-undo-retentionnightly shard green (t/369, t/370, t/386)Local pre-check on the integration tip (macOS, 4 surfaces): cluster_unit full set, cluster_regress 13/13, PG219 219/219, TAP incl. t/369 27/27 · t/370 22/22 · t/386 29/29.
Supersedes