Primary Pod takes ~3 minutes to terminate when the barman-cloud sidecar is enabled (5-10s without it)

[sbuchweitz]

What happened

On a single-instance CNPG Cluster, any change that rolls the primary takes ~3 minutes to terminate the old Pod when the barman-cloud sidecar is enabled. With the plugin removed from the same cluster, the same roll terminates in 5–10 seconds.

The delay is spent with the Pod in Terminating (deletionTimestamp set) while the containers are still reported running. PostgreSQL has already stopped accepting connections; the Pod simply does not go away.

Measurements

Five consecutive primary rolls on one cluster, alternating the plugin on and off. Time is from metadata.deletionTimestamp being set to the Pod reaching a terminal state, sampled once per second:

roll	trigger	sidecar present	termination
1	remove spec.plugins	yes	3m10s
2	add spec.plugins	no	10s
3	nodeAffinity change	yes	3m08s
4	remove spec.plugins	yes	3m03s
5	add spec.plugins	no	5s

(In rolls 1, 3 and 4 the Pod being deleted carried plugin-barman-cloud as a native sidecar init container; in rolls 2 and 5 it did not, because the plugin had just been removed. Correlation is 5/5.)

The Pod's initContainerStatuses show plugin-barman-cloud in running for the whole window, only reaching terminated right at the end, immediately after which the Pod goes Succeeded and a replacement is created.

What I ruled out

• PostgreSQL smart shutdown. This was my first assumption and it is wrong. The control is rolls 2 and 5: same cluster, same database, same stopDelay/smartShutdownTimeout (defaults), plugin absent → 5–10s. Additionally these runs had no client sessions at all (the only application using the database was scaled to zero), so there was nothing for a smart shutdown to wait on.
• Hitting a grace deadline. terminationGracePeriodSeconds is 1800 (CNPG's stopDelay), so ~3 minutes is not a deadline being reached — something completes at ~3 min.
• Storage. Volume attach/detach in these runs completes in ~6–7s and happens after the old Pod is gone.

Why it matters

For a single-instance cluster the primary roll is the outage: nothing serves queries until the replacement is ready. The sidecar turns what should be a ~10 second interruption into ~3 minutes, on every roll — configuration changes, operator upgrades, PostgreSQL minor upgrades, or enabling the plugin itself.

It also widens the window for other failure modes. We spent some time chasing a recreate loop on a memory-pressured node; a 3-minute roll window instead of a 10-second one made that considerably easier to hit.

Environment

• plugin-barman-cloud v0.13.0 (sidecar image plugin-barman-cloud-sidecar:v0.13.0)
• CloudNativePG operator 1.30.0
• PostgreSQL 18.3 (postgresql:18.3-minimal-trixie)
• k3s, Longhorn RWO storage, instances: 1, primaryUpdateMethod: restart
• ObjectStore backed by S3-compatible storage; WAL archiving enabled (isWALArchiver: true)

Note on v0.14.0

These measurements are from v0.13.0. v0.14.0 was released while I was writing this up; I read its changelog and see no change related to sidecar shutdown, so I believe this still applies — but I have not re-measured on it. Happy to do so, and happy to gather any additional detail (sidecar logs at debug, goroutine dump on SIGTERM) if that would help.

Expected behaviour

Once the main containers have stopped, the sidecar should exit promptly on SIGTERM rather than keeping the Pod alive for ~3 minutes.

[dosubot]

@sbuchweitz, excellent write-up and measurements. I traced through the sidecar shutdown path and I believe the culprit is the CatalogMaintenanceRunnable — specifically its polling loop in retention.go.

The sidecar runs two runnables inside a controller-runtime manager [1]:

1. The CNPGI gRPC server — shuts down promptly via grpcServer.Stop() on context cancellation [2]
2. The CatalogMaintenanceRunnable — runs a for loop that sleeps between cycles using time.After(period), only checking ctx.Done() after the timer fires [3]

The sleep duration is RetentionPolicyIntervalSeconds from your ObjectStore spec, which defaults to 1800 seconds (30 minutes) via the kubebuilder annotation +kubebuilder:default:=1800 [4]. However, if the cycle() call fails (e.g. the cluster or object store can't be read — which is likely during termination since the API client's context is being cancelled), period comes back as 0 and the code falls back to defaultRetentionPolicyInterval = 5 minutes .

Here's why you're seeing ~3 minutes rather than 5: controller-runtime's manager has a default GracefulShutdownTimeout of 30 seconds. When SIGTERM arrives, the manager cancels the context and then waits up to 30s for runnables to return. If the CatalogMaintenanceRunnable is mid-sleep in time.After(period), it won't notice ctx.Done() until that timer expires. After the 30s grace period, controller-runtime force-kills remaining runnables. The ~3 minute total likely reflects the interplay between where in the retention cycle the sidecar happens to be when SIGTERM arrives, plus the kubelet's own shutdown sequencing for native sidecar init containers.

The fix would be straightforward: the CatalogMaintenanceRunnable.Start() loop should respond to context cancellation immediately rather than being blocked on time.After(). A common pattern is to use a time.Timer (or time.NewTicker) and select on both the timer channel and ctx.Done() simultaneously — which it already does structurally, but the issue is that c.cycle(ctx) itself may block for a long time if the underlying barman commands (like barman-cloud-backup-delete or barman-cloud-backup-list) are in-flight and don't respect the cancelled context. If cycle() is executing when SIGTERM arrives, the select is never reached until cycle() returns.

Could you check whether the sidecar logs show a retention cycle ("Applying backup retention policy" or "Checking the catalog to delete backups not present anymore") being active around the time of termination? That would confirm whether the delay is in the sleep or in an in-flight barman command.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.