cloud-db-lab

A single-node private-cloud storage stack: Ceph (via Rook) providing block storage to PostgreSQL (via CloudNativePG), on k3s, on one Hetzner CPX32.

Built to answer one question with evidence rather than assertion: when the Postgres primary dies, what actually happens to the data underneath it?

Answer, measured twice: promotion in 3 seconds, zero rows lost, and the dead instance rejoins as a streaming replica and catches up — including writes that happened after it died.

---

The stack

Hetzner CPX32  (4 vCPU, 7.6 GB RAM, Ubuntu 26.04)
└── Hetzner Cloud Volume, 10 GB, raw          <- real block device, real udev
    └── k3s v1.36.2+k3s1  (single node)
        └── Rook v1.19 / Ceph 20.2.2 tentacle
            └── OSD 0  ->  pool replicapool  ->  StorageClass rook-ceph-block
                └── CloudNativePG v1.30.0
                    └── cluster pg-lab: 1 primary + 2 replicas
                        each with its own 1Gi RBD-backed PVC

Every byte Postgres writes lands on Ceph. No local-path provisioner is installed — k3s ships with one and it is deliberately disabled, so there is no way for a PVC to quietly bind to host disk and make the test meaningless.

Results

$ ./scripts/06-failover-test.sh
primary before : pg-lab-2
rows before    : 1001

11:45:02  deleting primary pod pg-lab-2
11:45:05  promoted pg-lab-1  --  FAILOVER TOOK 3s

new primary in recovery : f      (a real primary, not a replica)
rows after              : 1001   (no loss)
write to new primary    : ok (1002 rows now)

=== pg-lab-2 state ===
 is_replica | rows
------------+------
 t          | 1002    <- rejoined, and caught up past where it died

run	primary killed	promoted	failover	rows before → after	rejoin
1	pg-lab-1	pg-lab-2	3s	1000 → 1000	~25s
2	pg-lab-2	pg-lab-1	3s	1001 → 1001	~25s

Ceph stayed HEALTH_OK throughout both runs.

What broke, and why it's the interesting part

Two failures cost real time. Both are written up properly because the debugging is more useful than the fix:

• Why kind could never work — ceph-volume aborts its entire device inventory if any one device lacks udev metadata. kind nodes have an empty /run/udev/data, so enumeration dies on /dev/sda before reaching any other device. No device added inside a kind node can ever be found. Includes the source-level trace.
• Choosing the storage backing — k3s vs kind and loop devices vs a real cloud volume, including the undocumented ROOK_CEPH_ALLOW_LOOP_DEVICES gate that makes loop devices fail with a symptom identical to the udev bug.
• Failover results — full method and output.
• CSI VolumeSnapshots on Ceph RBD — snapshot, overwrite the original, restore, verify. Includes the copy-on-write chain visible at the Ceph layer, a snapshot of a live PostgreSQL volume, and an honest account of why crash-consistent is not the same as a backup.

An upstream bug found along the way is tracked in docs/01-why-kind-failed.md: Rook builds a malformed /dev//dev/sdb path when shelling out to udevadm.

Reproducing it

Needs: a host with real systemd/udev (not a container), a raw block device, and a non-root user in the docker/sudo groups.

./scripts/00-preflight.sh /dev/disk/by-id/YOUR_DEVICE   # read-only, exits 1 if blocked
sudo ./scripts/02-install-k3s.sh
./scripts/03-install-rook.sh /dev/disk/by-id/YOUR_DEVICE
./scripts/04-verify.sh                                  # want HEALTH_OK, 1 osd up/in

kubectl apply --server-side -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.30/releases/cnpg-1.30.0.yaml
kubectl -n cnpg-system rollout status deploy/cnpg-controller-manager
kubectl apply -f manifests/postgres/pg-cluster.yaml

./scripts/06-failover-test.sh

./scripts/07-install-snapshotter.sh   # CRDs + snapshot-controller + snapshot class
./scripts/07-snapshot-test.sh         # snapshot, modify, restore, verify

00-preflight.sh is the important one. It refuses to proceed on a device that has a filesystem, partitions, a mount, or no udev record — every failure mode that produces Rook's silent "skipping OSD configuration as no devices matched".

01-teardown-kind.sh and make-loop-device.sh are situational: the first removes a prior kind cluster, the second creates a loop-device fallback if you don't want to pay for a cloud volume.

Notes worth keeping

• The Hetzner Volume must be created with Manual mounting. The console offers only EXT4 or XFS and no "none" option; Automatic formats the volume, and Rook silently skips any device with a non-empty FSTYPE.
• Use /dev/disk/by-id/ paths, not /dev/sdb — kernel names can reorder.
• CloudNativePG's field is spec.storage.storageClass, not storageClassName. The wrong one fails with strict decoding error.
• On a single node, CNPG needs podAntiAffinityType: preferred; the default required leaves replicas Pending with nowhere to schedule.
• Rook manifest order is crds → common → csi-operator → operator. Skipping csi-operator gives no matches for kind "CephConnection".
• Single OSD means replica 1 (failureDomain: osd). Enough to demonstrate Postgres-layer failover; it is not a story about Ceph surviving disk loss.

Honest limitations

• One OSD, one mon, one node. This exercises Postgres failover on Ceph-backed storage, not Ceph's own redundancy. Killing the OSD would take the cluster down.
• Replica 1 means no data protection at the Ceph layer.
• Hetzner Volumes are network-attached, so this measures a lab, not bare-metal latency.
• The 3s failover figure is for an unloaded 1000-row table. It is a floor, not a number to quote under production write load.

Next: multiple OSDs with failureDomain: osd so the Ceph layer can be failed independently of the Postgres layer.

License

Apache-2.0, matching the ecosystem this targets (Kubernetes, Rook, Ceph, CloudNativePG).