CAS: orphan-manifest sweep deletes live part manifests → `fsck` reports `dangling=2` after basic CAS replication smoke

[alsugiliazova]

Commit: 99a6e1bec662f49b9491022d08a3d7207df03898
Binary: locally built /root/workspace-alsu/ClickHouse/build/programs/clickhouse → ClickHouse local version 26.6.1.1.
Harness: utils/ca-soak/docker-compose.yml (default RustFS topology)
Severity: HIGH — violates INV-NO-LOSS / INV-NO-DANGLE (live refs name missing manifest bodies)
Component (suspected): CasOrphanManifestSweep.cpp — orphan-manifest sweep classifies committed manifests as eligible+unowned and deletes them while refs still point at ManifestId 1:1:1.

---

Summary

After a minimal two-replica CAS smoke test (create table → insert one row on ch1 → sync replica on ch2), SQL replication appears to work (SELECT returns 123 on both nodes), but clickhouse disks fsck fails with dangling=2: two part-manifest object keys referenced by live refs are absent from the object store.

Forensics show the manifest bodies were physically deleted ~51 seconds after insert by the orphan-manifest sweep (manifest_delete events in system.content_addressed_log), not by a normal GC round reclaim of unreferenced blobs.

The pool listing (mc ls --recursive) confirms zero objects under cas/manifests/ while ref-log objects under cas/refs/.../_log/ remain.

---

Environment

Item	Value
Branch	cas-gc-rebuild
HEAD	99a6e1bec66 — cas: insulate reads from piggybacked stale-precommit sweep failures
Built binary	build/programs/clickhouse (mounted into containers per compose)
Compose file	utils/ca-soak/docker-compose.yml
Object store	RustFS rustfs/rustfs:1.0.0-beta.8 (RUSTFS_SCANNER_ENABLED=false, RUSTFS_HEAL_ENABLED=false)
Pool prefix	s3://test/soak_pool/ (endpoint http://rustfs1:11121/test/soak_pool/)
ch1 HTTP	localhost:8123
ch2 HTTP	localhost:8124
ch1 server_root_id	ca_soak_ch1 (configs/storage_conf_ch1.xml)
ch2 server_root_id	ca_soak_ch2 (configs/storage_conf_ch2.xml)
GC	enabled, gc_interval_sec=10 on both writable disks
fsck disk	ca_ro in configs/fsck_only_ca.xml (NOT in server config.d — intentional)

---

Prerequisites for reproduction

1. Build ClickHouse with CAS on branch cas-gc-rebuild:

   cd /root/workspace-alsu/ClickHouse
   # (assumes cmake build already done)
   ls -l build/programs/clickhouse
   build/programs/clickhouse --version
   # Expected: 26.6.1.1 or your local CAS build

2. Docker available with permission to pull:

   • clickhouse/clickhouse-server:25.8 (runtime shell only — binary is bind-mounted)
   • rustfs/rustfs:1.0.0-beta.8
   • minio/mc:RELEASE.2025-04-16T18-13-26Z

3. Working directory for all harness commands:

   cd /root/workspace-alsu/ClickHouse/utils/ca-soak
   mkdir -p logs/ch1 logs/ch2 tmp

---

Reproduction steps (exact commands run by reporter)

Step 1 — Start the CA soak cluster

cd /root/workspace-alsu/ClickHouse/utils/ca-soak
docker compose up -d

Cluster was left running ~14+ minutes before fsck; the bug manifests within ~1 minute of insert (see timeline).

---

Step 2 — Basic replication smoke (manual)

Define HTTP query helpers:

Q1() { curl -sf "http://localhost:8123/" --data-binary "$1"; }
Q2() { curl -sf "http://localhost:8124/" --data-binary "$1"; }

Create the same replicated table on both replicas with CAS storage policy:

DDL="CREATE TABLE smoke (a UInt64) ENGINE=ReplicatedMergeTree('/clickhouse/tables/smoke','{replica}') ORDER BY a SETTINGS storage_policy='ca'"

Q1 "$DDL"
Q2 "$DDL"

Insert on ch1 only, sync ch2, read on ch2:

Q1 "INSERT INTO smoke VALUES (123)"
Q2 "SYSTEM SYNC REPLICA smoke"
Q2 "SELECT a FROM smoke"

Observed at Step 2: 123 — replication smoke passes.

Verify table DDL:

Q1 "SHOW CREATE TABLE smoke"
# Contains: SETTINGS storage_policy = 'ca'
Q1 "SELECT count() FROM smoke"   # 1
Q2 "SELECT count() FROM smoke"   # 1

---

Step 3 — Run fsck (pool integrity check)

docker exec ca-soak-ch1-1 clickhouse disks \
  --config-file /etc/clickhouse-server/fsck-only.xml \
  --disk ca_ro \
  --query fsck

Observed output (summary + error):

fsck: listing blobs — 2 objects, 1 pages
fsck: listing manifests — 0 objects, 1 pages
fsck: listing manifests — 0 objects, 1 pages
reachable=0 dangling=2 unreachable=1 pending_gc=0 awaiting_gc=0 unaccounted=1 physical_bytes=290 referenced_logical_bytes=0 distinct_blobs=0 total_blob_refs=0 dedup_ratio=0
note: 1 object(s) are outside the current GC view — normal only as a transient (created+dropped between GC rounds); re-run fsck after the next round and investigate any that persist

Error: Code: 36. DB::Exception: fsck: 2 reachable object(s) MISSING (INV-NO-LOSS violation). (BAD_ARGUMENTS)

With detail:

docker exec ca-soak-ch1-1 clickhouse disks \
  --config-file /etc/clickhouse-server/fsck-only.xml \
  --disk ca_ro \
  --query "fsck --detail"

Observed detail rows:

dangling  soak_pool/cas/manifests/ca_soak_ch2/store/d66/d6604ccb-f7a4-43c1-9441-36fcd76a7148@cas@/0000000000000001-0000000000000002/000001.proto  0  ca_soak_ch2/store/d66/d6604ccb-f7a4-43c1-9441-36fcd76a7148@cas@/all_0_0_0
dangling  soak_pool/cas/manifests/ca_soak_ch1/store/f21/f21587d3-e176-4ec0-828b-a47d4038e119@cas@/0000000000000001-0000000000000001/000001.proto  0  ca_soak_ch1/store/f21/f21587d3-e176-4ec0-828b-a47d4038e119@cas@/all_0_0_0
unaccounted  soak_pool/blobs/ch128/2f/2f4d381cc5f0f81e7c113d36b9cbe52f  290  not in the current GC view — transient for a fast create+drop between rounds; PERSISTENT occurrences violate INV-2 (reachability-before-content), investigate

Expected (per 08-testing-and-soak.md): dangling=0, exit code 0, reachable > 0 after a successful insert.

---

Evidence collected after failure

A. SQL still returns data (misleading “green” signal)

Even after fsck failure, both replicas still answer:

Q1 "SELECT a FROM smoke"   # 123
Q2 "SELECT a FROM smoke"   # 123

Hypothesis for developers: part metadata may still be cached in-process; cold read / restart may fail once manifests are gone (not verified in this session — worth testing).

---

B. CAS event timeline (system.content_addressed_log)

Query:

SELECT event_time, event_type, ref_name, object_kind, object_hash, outcome, reason
FROM system.content_addressed_log
WHERE disk_name = 'ca' AND event_type NOT LIKE 'gc_%'
ORDER BY event_time;

Observed sequence (TabSeparated):

event_time	event_type	ref_name / notes	object_hash	reason
15:53:53	build_start	tmp insert build on ch1	—	startBuild: build in-flight
15:53:53	manifest_put		1:1:1	stageManifest: part-manifest body written
15:53:53	precommit	all_0_0_0	—	precommitAdd: build-intent owner add...
15:53:53	blob_put		ch128:2f4d381cc5f0f81e7c113d36b9cbe52f	uploadFromSource: fresh incarnation...
15:53:53	build_publish	all_0_0_0	1:1:1	promote: atomic owner move precommit -> ref...
15:53:53–59	ref_resolve ×5	all_0_0_0	1:1:1	read-side resolve (queries work)
15:54:44	manifest_delete	ch1 manifest path	full S3 key	orphan-manifest sweep: exact-token delete of an eligible+unowned build-prefix body
15:54:44	manifest_delete	ch2 manifest path (seq 1)	full S3 key	same reason
15:54:44	manifest_delete	ch2 manifest path (seq 2)	full S3 key	same reason
15:58:24	ref_resolve ×2	all_0_0_0	1:1:1	reads still “resolve” after delete

Key timing: insert/publish at 15:53:53, manifest deletes at 15:54:44 (~51 seconds later).

Manifest delete query:

SELECT event_time, event_type, object_hash, reason
FROM system.content_addressed_log
WHERE event_type = 'manifest_delete'
ORDER BY event_time;

---

C. GC round log (system.content_addressed_garbage_collection_log)

SELECT event_time, round, outcome, objects_deleted, manifests_deleted, error
FROM system.content_addressed_garbage_collection_log
WHERE disk_name = 'ca'
ORDER BY event_time DESC
LIMIT 10;

Observed: rounds completing with outcome = Success, but manifests_deleted = 0 on Finish rows — suggesting the manifest_delete audit events may come from orphan-manifest sweep outside the counted GC round metric, or accounting gap. Worth verifying in code.

---

D. Physical pool inventory (RustFS via mc)

From a one-shot container on the compose network:

docker run --rm --network ca-soak_default --entrypoint /bin/sh \
  minio/mc:RELEASE.2025-04-16T18-13-26Z -c \
  'mc alias set carustfs http://rustfs1:11121 clickhouse clickhouse && \
   mc ls --recursive carustfs/test/soak_pool/'

Observed objects (2026-07-15, after failure):

Key	Size	Notes
_pool_meta	39B	pool identity
blobs/ch128/2f/2f4d381cc5f0f81e7c113d36b9cbe52f	290B	column blob from insert
blobs/ch128/2f/...meta	22B	blob meta sidecar
cas/refs/ca_soak_ch1/.../_log/0000000000000001-0000000000000001	129B	ref log (new layout)
cas/refs/ca_soak_ch1/.../_log/0000000000000001-0000000000000002	241B	ref log
cas/refs/ca_soak_ch2/.../_log/0000000000000001-0000000000000001 … _000005	various	ch2 ref log entries (replica relink path)
gc/state, gc/hb, gc/gen/*/fold_seal, gc/server-roots/*/mount	—	GC coordination
roots/ca_soak_ch1/.../_files/format_version.txt	1B	verbatim table file
roots/ca_soak_ch2/.../_files/format_version.txt	1B	verbatim table file

Absent: any key under cas/manifests/ — confirms manifests were deleted from the store.

---

Expected vs actual behavior

Check	Expected	Actual
Step 2 SELECT on ch2	123	PASS — 123
fsck exit code	0	FAIL — exit 36 (BAD_ARGUMENTS)
fsck dangling	0	FAIL — 2
Manifest bodies in pool while ref live	present	FAIL — deleted
system.content_addressed_log error classes	none in positive scenario	no read_missing yet, but manifest_delete on live manifests
INV-NO-LOSS	no ref → missing object	violated

---

Hypothesis for root cause (for assignee)

1. Branch context: cas-gc-rebuild introduces ref snapshot + log layout (cas/refs/<server_root_id>/.../_log/<epoch>-<seq> objects visible in pool listing) and writer snapshot persistence changes (commits on branch: 093b1f83cd8, b271bf65a02, etc.).

2. Orphan-manifest sweep (src/Disks/.../CasOrphanManifestSweep.cpp) lists cas/manifests/ prefixes and deletes bodies classified as eligible+unowned via prefixEligible() / mount watermark floor / sealed fold cursor logic.

3. Failure mode: sweep does not recognize committed ownership for manifests whose refs now live in the _log ref layout, so it deletes manifest bodies that fsck still resolves as reachable via store.resolveRef(..., allow_stale=false) (see CasFsck.cpp owned-manifest walk).

4. Why SQL still works: ref_resolve events continue after delete — possibly stale snapshot/cache path, or resolve does not re-HEAD the manifest body on every read. Restart test recommended.

---

Suggested investigation paths

1. Reproduce on clean pool with immediate fsck (before ~51s sweep window):

   docker compose down -v && docker compose up -d
   # ... Step 2 ...
   docker exec ca-soak-ch1-1 clickhouse disks \
     --config-file /etc/clickhouse-server/fsck-only.xml \
     --disk ca_ro --query fsck

   If dangling=0 immediately but dangling=2 after ~1 minute → confirms sweep regression.

2. Trace orphan sweep eligibility for namespace ca_soak_ch1/store/f21/... and ca_soak_ch2/store/d66/... at writer_epoch=1, build_sequence=1/2 — why marked unowned after build_publish.

3. Align ownership detection between:

   • CasFsck.cpp (owned_manifest_keys via resolveRef)
   • CasOrphanManifestSweep.cpp (prefixEligible, sealedFoldCursor, mount floor)
   • new ref log / snapshot reader (ref snapshot+log Phase 1 on branch)

4. Restart test:

   docker restart ca-soak-ch1-1 ca-soak-ch2-1
   # wait for healthy, then:
   Q1 "SELECT a FROM smoke"
   Q2 "SELECT a FROM smoke"

   Report whether reads fail after restart with missing manifest.

5. Unit test gap: add gtest covering published ref + manifest body → orphan sweep must not delete under ref-log layout.

---

Related documentation / code pointers

Resource	Location
fsck spec	docs/superpowers/cas/08-testing-and-soak.md §1.2
fsck-only config	utils/ca-soak/configs/fsck_only_ca.xml
fsck implementation	src/Disks/.../CasFsck.cpp
orphan sweep	src/Disks/.../CasOrphanManifestSweep.cpp
soak harness compose	utils/ca-soak/docker-compose.yml
smoke script (equivalent Step 2)	utils/ca-soak/scripts/smoke_bringup.sh
Branch GC rebuild commits	093b1f83cd8, b271bf65a02, 26ed47963d8, 99a6e1bec66

---

Workaround (for testers, not a fix)

To learn fsck on a clean pool without hitting the sweep window:

docker compose down -v
docker compose up -d
# Step 2 insert + sync
# Run fsck within seconds of publish

Disabling GC (<gc_enabled>0</gc_enabled>) may not help if orphan sweep runs on a separate scheduler path — not verified.

---

Files created during this session (not part of the bug)

Reporter / assistant created runtime dirs only (not in git):

• utils/ca-soak/logs/ch1/, utils/ca-soak/logs/ch2/ — docker log mounts
• utils/ca-soak/tmp/ — harness temp dir

All harness YAML/XML/scripts were already present in the repo (dated 2026-07-14 checkout).

---

Checklist for developer sign-off

• Reproduce on 99a6e1bec66 with steps above
• Confirm manifest_delete events precede fsck failure
• Confirm zero cas/manifests/ keys in RustFS after failure
• Identify why orphan sweep marks published manifests unowned under ref-log layout
• Add regression test
• Verify fix: fsck → dangling=0 persists ≥5 minutes after insert with gc_enabled=1

[filimonov]

Thanks @alsugiliazova — this is an exceptionally thorough first report: the event-log timeline, the mc pool inventory, the fsck detail rows, and the root-cause hypothesis all line up, and you correctly fingered the ref-log ownership seam. That made triage fast.

Short version: this looks like a real defect that was specific to a mid-refactor window on cas-gc-rebuild, and it does not reproduce on the current branch head. Your build 99a6e1bec66 (2026-07-12) is now ~316 commits behind head, and it landed in the middle of the ref snapshot + log (refsnaplog) cutover — the new cas/refs/<root>/.../_log/ layout was in place, but the GC/orphan-manifest-sweep ref-intake and ownership detection had not yet been switched over to read published refs from that new layout. So the sweep classified freshly-published manifests as eligible + unowned and exact-token-deleted them (your manifest_delete … orphan-manifest sweep: exact-token delete of an eligible+unowned build-prefix body events), while fsck — which resolves refs — still saw them as reachable → dangling=2. That is exactly your Hypothesis #3.

The GC/sweep side of the cutover was completed shortly after your build (e.g. 318291fe5e5 "GC ref intake on snapshot+log", 1e95ca6bb6a "read-only consumers … for ref snapshot+log", plus follow-up GC fix-rounds).

On the current head (e3a165dfa15) a fresh 20-minute soak on the same default utils/ca-soak harness (two replicas, RustFS, GC enabled) reports dangling=0 at every checkpoint — including the quiesced GC checkpoint and after restarts and fault injection — well past the ~51 s sweep window that bit your run. [If PENDING step 1 confirms:] I also re-ran your exact two-replica smoke (insert → sync → wait >60 s → fsck) on head and got dangling=0, exit 0.

So: nothing to fix on head for this specific report — it was a transient during the ref-log migration. A couple of follow-ups from your excellent write-up that we're taking regardless:

• Regression coverage: your "published ref + manifest body → orphan sweep must NOT delete, under the ref-log layout" is a good explicit gate; we'll make sure that invariant is covered (there is gtest_cas_orphan_manifest_sweep + TLA models CaManifestSweepWindow in this area — we'll confirm they pin the ref-log-ownership case specifically).
• Your "restart test" instinct is right and we exercise it in the soak: reads after restart do not survive missing manifests (fail-closed), so the "SQL still returns 123" was the in-process part cache, as you suspected — not durable.

Could you retry on the current cas-gc-rebuild head (or a build after 318291fe5e5)? I expect a clean fsck. If you can still reproduce dangling>0 on a recent head, please reopen with the head SHA — that would be a different, live bug and we'd want it. Thanks again for the detail; this is exactly the kind of report that makes the branch better.