[WIP] CAS draft (adopting to CI/CD, not for review / merge)#2073
Draft
filimonov wants to merge 2415 commits into
Draft
[WIP] CAS draft (adopting to CI/CD, not for review / merge)#2073filimonov wants to merge 2415 commits into
filimonov wants to merge 2415 commits into
Conversation
installKeeper now passes the pool CasRequestBudget lease_safety_margin_ms into MountLeaseKeeper, so the production mount-lease keeper actually rides out transient renewal blips instead of fencing on the first one (fix #37 phase 1). The background-thread gtests proving the loop retries-and-recovers on a transient fault without fencing, and still fences immediately on a confirmed foreign-owner mismatch, were already added alongside Task 1 (gtest_cas_heartbeat.cpp, previous commit). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
#37 phase 2, part 1) Single helper the escaping retry-later CAS write throws will route through (next commits), replacing an ABORTED that ReplicatedMergeMutateTaskBase silently swallows (no backoff, no last_exception) with NETWORK_ERROR, which the merge queue's existing exponential backoff already treats as retryable. See the helper's doc comment for the full rationale and the one-line escape hatch to a dedicated error code if NETWORK_ERROR ever proves too coarse. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…eRetryLater (fix #37 phase 2) 9 sites (recovery-restart cap, recovery seal PUT failure, shutdown, mount-lost, superseded-by- remount x2, wedge-Unresolved x2, DefiniteFailure) that threw ABORTED for a retry-later condition now throw/carry NETWORK_ERROR via the shared helper. This was every ABORTED usage in the file, so the now-dead ABORTED extern declaration is removed. Existing tests pinning ABORTED at these sites are fixed in the next task. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…p retry-later reroute (fix #37 phase 2) Updates every assertion in this file that pinned ABORTED at one of the 9 CasRefLedger.cpp sites rerouted through throwCasWriteRetryLater/makeCasWriteRetryLaterExceptionPtr (previous commit) to NETWORK_ERROR: the wedge-creating dropRef/dropNamespace/listRefs calls across RefWriterAppendLane, RefWriterRemount, RefTableCacheEviction, CasAnomalyPolicy, RefWriterNamespaceRemoval, and RefWriterRecoverySeal. Confirmed via a pre-reroute rebuild that the one remaining failure in this file (RefWriterRecoverySeal.SealPutConflictThrowPropagatesAndDoesNotWedgeRecovery, a zstd decompression error unrelated to any ABORTED/NETWORK_ERROR assertion) is pre-existing and NOT caused by this reroute -- left untouched, flagged separately. The three assertions in this file that reach CasPartWriteTxn.cpp's requireAlive/precommitAdd (not yet rerouted -- one of them is still under discussion, see the fix #37 phase 2 scope question) are intentionally left as ABORTED for now and will be updated alongside that file's reroute commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…rowCasWriteRetryLater (fix #37 phase 2) 9 sites (requireAlive x2 -- cancelled + superseded epoch, uploadFromSource-Unresolved, stageManifest x2, promote x5 -- body-absent/RefMatchesBody/ManifestNamespaceMatches/precommit-no-longer-live-owner/ unique-ref-invariant) that threw ABORTED for a retry-later condition now throw NETWORK_ERROR via the shared helper, so the merge queue's exponential backoff engages instead of the silent ABORTED no-backoff path. One intended behavior change: putBlob's 8-round condemned-churn retry loop only catches ABORTED, so uploadFromSource's NETWORK_ERROR now escapes to the caller (merge backoff) on the first attempt instead of being re-driven locally 8 times -- desirable (no point hammering a lost fence locally). Existing tests pinning ABORTED at these sites are fixed in the next commit. DEFERRED: precommitAdd's "namespace removed, exact _cleanup completion marker not yet observed" throw (spec §Namespace Birth) is NOT rerouted here -- it is not in the plan's enumerated 10-site list, and while it reads like the same retry-later class, main has not yet confirmed the scope for this specific site. Left as ABORTED pending that answer; the 9 unambiguous sites (matching the plan exactly) proceed now so Task 7/8's known test fallout does not keep piling up while waiting. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…etry-later reroute Full CAS gtest sweep after the CasPartWriteTxn.cpp/CasRefLedger.cpp reroute (previous commits) turned up every remaining assertion pinning ABORTED at one of the 18 rerouted call sites (all except the one precommitAdd site still pending confirmation): - gtest_cas_promote_republish.cpp: PromoteOverDifferentCommittedRefFailsClosed (unique-ref invariant). RepublishReDriveOverDifferentContentDstFailsClosed is deliberately left untouched -- it reaches PartFolderAccess::republishRef's own genuine-content-conflict guard, not one of the rerouted sites (verified by reading that guard directly). - gtest_cas_part_write.cpp: PromoteFailsClosedWhenPrecommitNoLongerLiveOwner, PromoteRepointsCommittedRef (unique-ref invariant), and both budget-exhaustion tests: stageManifest's (renamed ...MapsToNetworkError) and putBlob's (renamed ...MapsToNetworkErrorAndEscapesImmediately -- the one genuine behavior change: putBlob's 8-round condemned-churn loop only catches ABORTED, so a NETWORK_ERROR now escapes on the FIRST attempt instead of being re-driven 8 times). - gtest_ca_wiring.cpp: PromoteWithoutLivePrecommitAbortsWithoutResurrect (precommit-no-longer-live-owner). - gtest_cas_pool.cpp: OldEpochBuildFailsClosedAfterRemount (requireAlive superseded-epoch), plus the three wedge-creating dropRef assertions this sweep's first pass had missed (CasPoolShutdown, CasRemountTmat x2) -- these hit CasRefLedger.cpp's append-wedge path from an earlier commit. - gtest_cas_ref_writer.cpp: DropNamespaceCancelsInFlightBuildAndNextOpThrows (requireAlive cancelled -- both its stageManifest and promote assertions). IMPORTANT verification-scope finding: this plan's own gate command (--gtest_filter='Cas*:CA*') silently excludes every RefWriter*/RefTableCacheEviction/RefSnapshotCodec suite in gtest_cas_ref_writer.cpp and gtest_cas_pool.cpp, none of which match "Cas*" or "CA*" (e.g. RefWriterAppendLane, RefWriterRemount, RefWriterNamespaceRemoval, RefWriterRecoverySeal, RefTableCacheEviction) -- about 60 tests, several of which this very reroute broke. Discovered by enumerating every TEST()/TEST_F() suite name across src/Disks/tests/gtest_ca*.cpp and diffing against the plan's filter. The corrected comprehensive filter used from here on is: Cas*:CA*:ContentAddressedLog*:CountingBackendShape*:RefSnapshotCodec*:RefTableCacheEviction*:RefWriter* Full sweep under the corrected filter: 907/908 pass; the one remaining failure (RefWriterRecoverySeal.SealPutConflictThrowPropagatesAndDoesNotWedgeRecovery) is a pre-existing zstd decompression bug confirmed unrelated to this reroute (reproduced identically by rebuilding CasRefLedger.cpp from its pre-reroute commit and rerunning just that test) -- left untouched, flagged separately for main. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…se 3) throwCasWriteRetryLater/makeCasWriteRetryLaterExceptionPtr now also emit a LogSeriesLimiter- throttled Warning-level CAS log line, independent of whatever the caller (e.g. ReplicatedMergeMutateTaskBase) separately does with the exception. system.replication_queue visibility (last_exception/last_exception_time) is already automatic from phase 2's NETWORK_ERROR reroute -- this only adds direct CAS-log visibility for operators not looking at that table.
…-alive S3 (fix #37 regression) Closes the chaos-coverage gap identified in the #37 post-mortem: prior soak chaos only faulted nodes, never a degraded-but-alive object store. Runs on docker-compose-s3faultproxy.yml, the same compose S22 already proved out (compose_variant = "s3faultproxy" needed no new plumbing). Two legs against the mount lease's compiled-default 30s TTL: - Short fault (< TTL): asserts fix #37 phase 1 directly via SingleWriterSlot::backgroundLoop's own log lines -- at least one transient-retry line (the fault was actually exercised, not vacuous) and zero fence-trip lines (the lease must survive). A post-disarm INSERT must succeed immediately. - Long fault (> TTL): the fence SHOULD trip (correct fail-closed) -- asserts a fence-trip line now appears, and that system.replication_queue.last_exception is populated when anything queued during the outage (fixes #37 phases 2/3: the old ABORTED mapping was invisible here). The system must recover cleanly afterward: a final INSERT succeeds and fsck is clean at quiescence. Not run by this commit -- the controller owns scenario runs (docker collision avoidance); this lands the card only. Verified: py_compile clean, imports and registers under the scenarios package without executing run(). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…se 2, scope closed) Closes the one deferred throw site from the #37 reroute (PartWriteTxn::precommitAdd's "namespace removed, exact _cleanup completion marker not yet observed" check, spec Namespace Birth). Not in the plan's original 10-site inventory for this file -- confirmed by main as a plan-inventory omission (the spec's line numbers predate the source-layout/R1 refactors), not a deliberate exclusion: - It ESCAPES to the merge/insert caller: precommitAdd sits between stageManifest and promoteBuild in PartFolderAccess::publishEntries, both already rerouted to NETWORK_ERROR. Leaving this one ABORTED would let the middle step in the same commit sequence defeat the merge backoff -- the exact Fix-2 defect. - It is genuinely retry-later: the condition resolves once GC's namespace-cleanup item publishes the marker, matching the spec's own "namespace being dropped" retry-later reason. - republishRef's own genuine content-conflict ABORTED (PartFolderAccess.cpp:359) is a DIFFERENT, terminal condition and stays untouched, confirmed by direct inspection. CasPartWriteTxn.cpp's ABORTED inventory is now closed: 11 escaping retry-later throws rerouted to NETWORK_ERROR total (the plan's original 10 -- requireAlive x2, uploadFromSource-Unresolved, stageManifest x2, promote x5 -- landed in an earlier commit whose message miscounted them as "9"; they were all 10, confirmed by re-grepping this file's throw sites -- plus this 11th, precommitAdd) + 2 internal-signal ABORTEDs correctly left alone (observeAndAdmit's condemned-token throw, reviveObserve's vanish-during-revival throw). Updates the one test pinning the old ABORTED (RefWriterNamespaceBirth.BirthFromRemovedRejectedWithoutMarkerAcceptedWithMarker, gtest_cas_ref_writer.cpp) to NETWORK_ERROR. Verified: unit_tests_dbms builds clean; comprehensive CAS gtest sweep (Cas*:CA*:ContentAddressedLog*:CountingBackendShape*:RefSnapshotCodec*:RefTableCacheEviction*:RefWriter*) 907/908 pass, the one remaining failure is the pre-existing, unrelated RefWriterRecoverySeal.SealPutConflictThrowPropagatesAndDoesNotWedgeRecovery zstd bug flagged separately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…* tests) + pre-existing RecoverySeal zstd fragility Two test-health findings from R3: (1) the Cas*:CA* gate filter silently excludes RefWriter*/ RefTableCacheEviction/RefSnapshotCodec/ContentAddressedLog suites (~60 tests); corrected filter recorded for future gates. Verified R1/R2 broke nothing in the excluded set. (2) RefWriterRecoverySeal .SealPutConflictThrow fails on a pre-existing zstd/corrupt-seal-adoption fragility (F3-1a side-finding), not an R1/R2/R3 regression (R1 diff provably does not touch the seal/decompress/recovery path). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
The S39 lease-fault-tolerance card (fix #37 regression) declared @register + name="S39" but cards/__init__.py uses an explicit per-module import list and s39_lease_fault_tolerance was not added, so @register never fired and 'run --scenario S39' matched nothing. Add the import. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…e (fix false fence-trip) Leg A ran the best-effort INSERT inline while the fault was armed, then slept short_s, THEN cleared the fault. A blocking insert retries under the fault for its whole CAS budget (~20s), so the armed window was insert-duration + short_s (>30s), past the lease TTL -- the renewer then correctly fenced (observed: retry@:00, retry@:10, fence@:20), failing leg A's "short fault must NOT fence" assertion and cascading into the post-disarm INSERT (uncaught). Run the write in a daemon thread so the armed window is EXACTLY short_s; add an assert that short_s < renew_period so the window faults at most one beat (one transient retry, no deadline breach, no fence -- by construction). Fix #37 phase 1 is correct (confirmed by review + the retry-then-fence log); this was a scenario-card timing bug. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
`adoptPartFromManifest` (fetch-by-relink receiver) wraps publishEntries, which calls stageManifest, precommitAdd, and promote. #37 rerouted their retry-later throw sites from `ABORTED` to `NETWORK_ERROR` so the merge backoff engages, but this catch still keyed only on `ABORTED`, so every routine retry-later relink now fell into the "unexpected error" branch and logged a misleading WARNING (both branches already `return false`, so no correctness break — this is an observability fix only). Match `NETWORK_ERROR` alongside `ABORTED` on the expected retry-later branch, declare `ErrorCodes::NETWORK_ERROR` in this file, and update the two nearby comments that still said the retryable class maps to `ABORTED` only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
All 4 S39 verdicts passed (short-fault-no-fence, long-fault-fences, last_exception populated) but the leg crashed on the trailing post-recovery INSERT: it threw the expected retry-later NETWORK_ERROR while the self-remount recovery (~16s + queue backoff) was still in flight, exceeding settle_s under load. Replace the bare INSERT with a poll-until-success loop (90s budget) -- a retry-later error means "not recovered yet", not a failure; the verdict now asserts writes RESUME within the budget, which is the actual #37 recovery contract. No product change; binary unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…ment check Replica-agreement failed with UNKNOWN_TABLE on node2: the table was created only on node1 (cl.nodes()[0]), but a ReplicatedMergeTree materializes per-replica, so node2 never had it. Create it on every node (matching S36/S37), and SYNC REPLICA on all nodes before the agreement check so it compares converged state rather than a mid-catch-up snapshot after leg B's long fault (the check itself only polls ~8s). All fault/write legs still drive node1 only -- single-writer mount-lease test. The five #37-fix verdicts already pass (short-fault-no-fence, long-fault-fences, last_exception populated, writes-resume-in-budget, not-vacuous); this closes the last, unrelated setup-gap verdict. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…uest retry after R3 (#37) stageManifest Unresolved reroute R4 soak (seed=42, chaos) confirmed a release-blocker: a client-retried sync INSERT after R3's stageManifest Unresolved->NETWORK_ERROR "retry-later" can return server success while its rows never land (9 ops, sum=1118 = exact soak deficit; both replicas agree; fsck CLEAN so the CAS oracle misses it). Airtight acked-vs-errored proof: model applies on submit + driver re-raises on retry-exhaustion + run ended CHECKPOINT (not WORKLOAD) failure => every insert got HTTP-200 success => the 9 absent ones were false dedup-no-ops. Mechanism: cross-request retry dedups against an orphan RMT block-dedup token left by the failed first attempt (violates the driver's B138 "failed sync insert leaves no token"), because R3 moved the ambiguous-PUT abort across the HTTP-request boundary. Fix needs RMT-dedup-token vs CAS-commit atomicity ordering (hard decision). R3/#37 NOT ship-ready until fixed. Full evidence + mechanism + fix directions + repro in the backlog entry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…data-loss empirically stands, mechanism now OPEN Independent source trace: CAS commit precedes the atomic Keeper dedup-token multi (ReplicatedMergeTreeSink.cpp:976 vs :985), so a failed CAS write creates NO token -> the stated orphan-token mechanism is wrong. Empirical acked-then-lost (9 ops, exact 1118) still confirmed; root cause now open (reroute-across-replicas + kill, or client-retry-classification hypotheses; needs instrumented repro). R3 still not ship-ready. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…lica dedup (orphan-token theory refuted) dl_probe.py reproduces deterministically (pause rustfs>budget + kill ch2 during continuous sync inserts): 1314 acked, 198 acked-but-absent. text_log smoking gun: byte-identical R3-NETWORK_ERROR client retry dedups against a Keeper block_id/part-znode (all_NNNN_NNNN_0) whose CAS data never durably landed (rustfs paused) on a then-killed replica = PHANTOM part -> ignoring it -> acked, lost. fsck clean (nothing referenced). Full trace + fix directions + TODO in the report. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…b_storage_log (112 failed uploads); refined root = block_id znode outlives CAS-data durability Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…s CAS-blob durability); pointer to report Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…ing correct); defect = znode OUTLIVES part data (1258 znodes vs 1116 rows); fix = tie znode lifetime to CA part-data / verify-on-dedup Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…mitted, local txn rolled back under fault); text_log proves in-zk-true/locally-false phantom -> NO_REPLICA_HAS_PART Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfRKU1Tt1CnaiFYh3kue27
…TER the Keeper multi(:985), not at renameParts; genuine split-commit window, earlier DECISIVE section superseded Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PeC1dg2dXBh55xkaZ63U1P
…rrected split-commit mechanism, durability order inverted on CA) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PeC1dg2dXBh55xkaZ63U1P
…d) + structural two-txn defect + agreed fix model (disk txn closes at renameParts; rollback = new txn) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PeC1dg2dXBh55xkaZ63U1P
…-dedup too (crash window, merge-base-verified); recommendation flips to generic renameParts disk-txn close Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PeC1dg2dXBh55xkaZ63U1P
… designed for this point) + renameParts throw-trace all sites; :9503 false alarm; ready for spec Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PeC1dg2dXBh55xkaZ63U1P
…enameParts); fixes acked-then-lost INSERT loss, generic + upstream candidate Report: docs/superpowers/reports/2026-07-17-dataloss-traced-root-cause.md Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PeC1dg2dXBh55xkaZ63U1P
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PeC1dg2dXBh55xkaZ63U1P
…meParts disk-txn close + failpoint test + S40 + gates) Spec: docs/superpowers/specs/2026-07-17-part-durability-before-keeper-commit-design.md Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PeC1dg2dXBh55xkaZ63U1P
…EBUG) Task 3 follow-up. The recompute-and-compare debug check (debugAssertBodyCounters) and its call site were guarded with `#ifndef NDEBUG`, which compiles them OUT of sanitizer builds (RelWithDebInfo-based, NDEBUG defined). Switch to `#ifdef DEBUG_OR_SANITIZER_BUILD` -- the idiomatic ClickHouse guard (base/base/ sanitizer_defs.h), the same condition `chassert` fires under -- so the O(N) drift check runs in debug AND all sanitizer builds. Verified under ASan (build_asan): CasRef* 169/169 pass with the check active on every applyRefLogTxn, zero chassert aborts, no ASan errors -- byte-exactness of the incremental body counters proven across the whole suite. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds `encodedSnapshotBudgetSize` / `encodedRemovalBudgetSize`, which read `framing + running body-byte total` instead of re-encoding the whole table, and rewrites `admits()` to use them -- O(touched rows) instead of O(table size). Property-tests the accessors against the real encoders across randomized states (`CasRefBudgetSize.AccessorsEqualFullEncodeRandomized`); all existing byte-exact `Admits*` boundary tests stay green unmodified. `admits()`'s debug drift check keeps the `DEBUG_OR_SANITIZER_BUILD` guard (not `NDEBUG`) so ASan/TSan runs exercise it too, matching the guard `debugAssertBodyCounters` is now compiled under. `buildHypotheticalRemovalTxn` (the full-encode helper `admits()` used to call) has no remaining callers after this rewrite and is removed; a few doc comments that named it are reworded to describe the hypothetical removal-transaction shape directly instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Final-review polish: the recompute-and-compare chassert reasserts equality on every applied transaction (via applyRefLogTxn) and every admits preview, not literally per-op; and it is debug/sanitizer-only, not debug-only. Comment-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… failing the load Layer 1 of the stuck-table-load fix. A transient NETWORK_ERROR during CasRefLedger::ensureRefTableRecovered (LIST/GET/seal-PUT) is now retried with capped-exponential backoff up to cas_request_budget.recovery_retry_budget_ms (default 120s) instead of propagating and failing the table's async load permanently. Non-transient errors (CORRUPTED_DATA etc.) still fail fast, and the deliberately-terminal vanish-race brake is latched so the outer loop never re-drives it. The backoff sleep runs with state_mutex released and is interruptible via the mount fence. New ProfileEvent CasRefRecoveryRetries. The existing SealPutFailureFailsRecoveryClosed test asserted the old fail-closed-on-transient behavior; renamed to SealPutTransientFailureIsRetried ThenSeals and updated to the new single-touch retry contract. Four new RefWriterRecoveryRetry gtests cover retry-then-succeed, budget exhaustion, non-retryable fail-fast, and vanish-brake terminality. Full CA gtest gate green (1045/1045).
Layer 2 of the stuck-table-load fix (configuration only, no C++). The soak's ca_stress table now lives in a dedicated 'ca_soak' Atomic database created with lazy_load_tables=1, so a transient S3 error during table startup is retried on next access (StorageTableProxy) instead of caching a permanent AsyncLoader FAILED job. The table name stays bare; the workload connection's default database (Node/Cluster gain a `database` kwarg, always sent as a URL param) is set to ca_soak, so system.parts filters and the zk path are unchanged. Both the setup_cluster_and_table and phase-1 bring-up paths create the lazy database first over a default-scoped bootstrap cluster. Documents the same recommendation for CAS deployments. Soak unit suite green (218/218).
…ETACH catch-22 Deliverable 3 of the stuck-table-load fix. A CAS-free description of the generic ClickHouse problem: a table whose async load job fails is permanently stuck until restart, and DETACH cannot recover it because table resolution (DatabaseCatalog/DatabasesCommon waitTableStarted) waits on the failed load job before reaching the state-erasing detach path. References lazy_load_tables (ClickHouse#96283) and related issues ClickHouse#88934/ClickHouse#67521. Draft only; not filed.
…nt S3 outage Layer 2 verification (with_rustfs, single node). Reproduces the soak v11 scenario's essence: restart so the CAS table re-attaches as a lazy StorageTableProxy, pause rustfs and touch the table (best-effort bounded probe while the object store is unreachable), then unpause and assert the table becomes usable again on a later access WITHOUT a server restart -- the guarantee a non-lazy FAILED AsyncLoader load job cannot offer. Note on behaviour observed while writing this test: the lazy first-access build does not fail fast when S3 is down; it blocks server-side until S3 returns (the mount lease stops advancing at ~TTL but loadDataParts keeps waiting), so a DETACH cannot interrupt an in-flight build. The test asserts the actual guarantee (self-heal without restart), not a fast-fail or DETACH-while-down. Passes in ~58s.
…age (observed during Layer 2 test) Records a robustness observation from the stuck-table-load Layer 2 integration test: a lazy CAS table's first-access build (loadDataParts over the CA disk) blocks server-side for the entire S3 outage instead of fail-closing when the mount fence is lost, so DETACH cannot interrupt it. Not the stuck-table bug (no permanent FAILED state; self-heals without restart when S3 returns), but a possible UX/robustness follow-up worth confirming.
… remount/shift/sleep (3-model review) Three independent adversarial reviews (opus/fable/codex-sol) converged on real defects in the just-landed recovery-retry loop. This fixes them: - F1 (severe): the retry only covered the seal PUT. Recovery LIST/GET call the backend directly and surface a raw S3_ERROR (or a non-DB Poco timeout the old `catch (const Exception&)` didn't even see), NOT the NETWORK_ERROR the seal controller mints -- so a blip on the LIST/GET leg (the path the motivating stuck-load incident actually hit) was NOT retried. Now `catch (...)` + getCurrentExceptionCode + an isTransientRecoveryError classifier (NETWORK_ERROR/S3_ERROR/POCO_EXCEPTION/SOCKET_TIMEOUT/CANNOT_READ_FROM_SOCKET/ TIMEOUT_EXCEEDED). New gtest faults the LIST leg; comment corrected. - F3/F4: after the unlocked backoff sleep, re-check budget/fence AND rt.superseded_by_remount before the next attempt (mirrors appendRefOps' gate), so recovery can't complete on an orphaned runtime after a remount-during- backoff (a possible hang) or run past the budget under a lost fence. - F2: saturating shift for the backoff (mirrors CasRequestController:: backoffBeforeAttempt) -- no UB/wrap at high retry counts / large budgets. - F5: re-acquire state_mutex before an exception from the sleep fn can unwind, so the SCOPE_EXIT clearing recovery_in_progress never runs unlocked. New gtest proves a throwing sleep does not wedge the recovery lane. CA gtest gate 1047/1047.
…T, not loadDataParts (review finding) Per the 3-model review (codex-sol): an earlier draft attributed the lazy first-access block to loadDataParts, but the server log places it in the CAS ref-namespace LIST inside ensureRefTableRecovered (the disk S3 client's ~500- attempt transparent retry rides the outage there; loadDataParts only runs, in ms, AFTER the unpause). This is the same leg as Layer 1 review finding F1. Also scoped the claim honestly (the integration test proves no-permanent-FAILED via the original build continuing on unpause, not a proxy retry of a thrown build) and pinned the fix-direction to a fence-keyed (not bare-deadline) fail-fast so it cannot regress the short blips Layer 1 rides out.
… outage, scope the claim honestly Per the 3-model review (fable/codex-sol): the test was vacuity-prone (wait_for_paused=False lets the probe race the freeze; every probe outcome was accepted) and over-claimed. Now it asserts the during-outage probe actually raised (the multi-round-trip build reliably catches the freezer, confirmed over two runs), and the comments state honestly what is proven: no permanently-cached AsyncLoader FAILED state, via the original in-flight build completing on unpause (block-until-recovered) -- NOT a proxy retry of a thrown build, which this path cannot exercise. Passes in ~54s.
…mcpy serialization) Design for the BACKLOG item 'ref-ledger JSON encoding writes byte-by-byte instead of bulk-copying safe runs': replace WriteBuffer in all CAS format encode paths with a flat bulk-append writer, byte-identical output, memcpy-floor benchmark acceptance gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
11 tasks: byte pins first, writer core + differential-fuzzed escaping, vocabulary overloads, per-codec migration, WriteBuffer vocabulary retirement, memcpy-floor benchmark acceptance gate, BACKLOG close-out. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…edge run Pre-change golden corpus for the CasJsonWriter bulk-encoding migration (docs/superpowers/specs/2026-07-20-cas-json-writer-bulk-encoding-design.md). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…CT_EQ) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ming) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nst writeJSONString Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…to the benchmark-gated contingency) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ial-tested against the WriteBuffer set Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…per-record allocations) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…asJsonWriter Migrates the four GC-side codec encoders from WriteBuffer to CasJsonWriter, same mechanical transformation as the ref codecs: encode entry points swap WriteBufferFromOwnString for CasJsonWriter and return std::move(out).take(); file-local write-side helpers (writeEntryRecord, writeRun, writeSortedRuns) take CasJsonWriter & instead of WriteBuffer &, bodies unchanged since the CasJsonWriter vocabulary overloads already exist. Decode paths untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…CasJsonWriter Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ne bulk write per record Memory stays bounded by the largest line, never by record count. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…y CAS text writer writeKey/writeStringValue/writeHex128Value/writeU64StringValue/writeBoolValue/closeObject/ writeHeaderLine/writeTrailerLine in CasTextFormat, and writeTokenFields/writeBlobRefFields/ writeManifestRefFields in CasWireVocab, no longer have WriteBuffer overloads: every CAS encoder (ref log, ref snapshot, part manifest, gc state, gc outcomes, fold seal, pool meta, blob meta, blob envelope, server-root, cas_run) already builds its object through a CasJsonWriter before handing bytes to the underlying WriteBuffer. The old WriteBuffer implementation survives verbatim inside gtest_cas_json_writer.cpp as a test-local reference_vocab namespace, so CasJsonWriterVocab.MatchesReferenceVocabulary keeps an independent oracle instead of comparing CasJsonWriter against itself. The header/trailer/ manifest/token/blob-ref differential sub-tests are dropped (their WriteBuffer counterparts no longer exist); the per-format goldens and pins already gate those paths end to end. gtest_cas_wire_vocab.cpp and gtest_cas_text_format.cpp, which exercised the WriteBuffer vocabulary directly, are migrated to CasJsonWriter the same way the production codecs were in Tasks 5-8. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…mbers Adds BM_MemcpyTxnBytes as the near-memcpy floor for BM_EncodeRefLogTxn: reassembles the same encoded bytes from precomputed 16-byte fragments via plain String appends. Measured BM_EncodeRefLogTxn = 333 ns (down from the 753 ns pre-CasJsonWriter baseline) vs a 30.7 ns floor -- a ~10.8x ratio, above the 3x acceptance gate. Applied contingency rung 1: CasJsonWriter::keyLiteral merges the separator-store and rendered key text into one append, replacing the three appends key() does to compose the same bytes. Rewired the fixed unprefixed keys in writeOp (CasRefLogFormat.cpp) and writeCommittedRow (CasRefSnapshotFormat.cpp) to use it. CasEncodingPins.* stayed byte-identical, but the ratio barely moved (325 ns / 30.9 ns ~= 10.5x), so the dominant cost is elsewhere (likely string-value escaping scans, not key/separator composition). Rung 2 was not attempted per the contingency ladder -- it trades readability and needs a human decision. Reported as DONE_WITH_CONCERNS; see .superpowers/sdd/task-10-report.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…(CasJsonWriter, measured 2.26x, honest floor analysis) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… remove no-op keyLiteral, reconcile perf numbers Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rework the two arm-only content-addressed stateless-test jobs into a broader matrix. For `content_addressed s3 storage` (the CA-over-S3 / RustFS MergeTree default) add coverage on amd across the plain binary and all three amd sanitizer builds (`amd_asan_ubsan`, `amd_tsan`, `amd_msan`), keep an `arm_binary` s3 run for cross-arch, and move the local (non-S3) `content_addressed storage` job to `amd_binary`. Runner labels and `_GH` build artifacts follow the existing amd stateless jobs. No runtime change to `ci/jobs/functional_tests.py` is needed: the `amd_*` tokens, the `content_addressed s3 storage` -> `--content-addressed-s3-storage` / RustFS handling, and the `amd_tsan --timeout 1200` special-case already exist. Regenerated the committed workflow YAML with `praktika yaml`. Release Builds keeps only the non-sanitizer CA jobs (it does not build sanitizers, so the sanitizer jobs are pruned automatically). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
content addressable storage - draft PR
Documentation entry for user-facing changes
TBD.
Exclude tests:
Regression jobs to run: