Changes to ObjectStore spec do not get applied until after the cluster is healthy

[dancollins]

We had an issue yesterday where we updated QNAP QuObjects, causing some change to the way S3 is handled. The resultant error from barman cloud was:

ERROR: Barman cloud WAL archiver exception: An error occurred (InvalidDigest) 
when calling the PutObject operation: The Content-MD5 or checksum value that 
you specified is not valid.

The fix is already mentioned in the documentation, where we add the following to the object store:

---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
spec:
  configuration:
     ...
  instanceSidecarConfiguration:
      env:
        - name: AWS_REQUEST_CHECKSUM_CALCULATION
          value: when_required
        - name: AWS_RESPONSE_CHECKSUM_VALIDATION
          value: when_required

and I can confirm this works perfectly. However. Because one of the pods is stuck (1/2 Ready, blocked on the error above) the update never gets applied. What I had to do was apply the fix and then manually delete that one stuck pod (3-node cluster, this pod was a replica).

I think this is a bug, just because this configuration change was specifically required to resolve the issue I was having (S3 semantics changed). It's a little more complicated, here, because it seems like I had a primary switchover (perhaps due to similar S3-related errors) which left a bunch of WALs waiting to get uploaded. Would be good to see if there's a way this could be improved such that the barman-cloud-plugin operator is able to detect the S3-related failure and apply outstanding configuration.

[dosubot]

Hi @dancollins! I'm Dosu and I’m helping the plugin-barman-cloud team.

This is a confirmed behavioral gap in the plugin. ObjectStore configuration (including instanceSidecarConfiguration.env) is only read at pod creation time via the CNPG-I lifecycle hooks — there's no mechanism to dynamically reload it into running sidecars [1]. Additionally, the ObjectStoreReconciler that could potentially trigger automatic pod restarts on spec changes is currently unimplemented [2].

So the chicken-and-egg scenario you experienced is expected (unfortunately): the config fix can't take effect because the pod won't be recreated, and the pod is stuck because it doesn't have the config fix.

Current workaround: After applying the ObjectStore change, force a rolling restart:

kubectl cnpg restart <cluster-name>

This recreates the pods, which picks up the new environment variables.

A similar issue has been reported with resource limit changes (e.g., OOM kills) where the same manual restart is needed [3]. The underlying improvement would be for the plugin to detect ObjectStore spec drift and trigger pod restarts automatically — but that doesn't exist yet.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[Utkarsh-sharma47]

I spent some time tracing the code path and the reported behavior appears valid.

From what I found:

• ObjectStoreReconciler (internal/controller/objectstore_controller.go) only reconciles RBAC Role rules and does not update pods or enqueue Cluster reconciliation.
• instanceSidecarConfiguration is consumed through the CNPG-I lifecycle hooks (collectObjectStoreEnvs -> reconcilePodSpec) when pod specs are created/evaluated.
• There does not appear to be a runtime reload path for instanceSidecarConfiguration.env, so environment changes require pod recreation.
• CNPG's Cluster controller does not appear to watch ObjectStore resources directly.
• Additionally, isInstanceNeedingRollout() skips rollout evaluation for pods that are not Ready, which seems relevant to the reported 1/2 Ready failure scenario.

This matches the reproduction described in the issue: updating the ObjectStore alone does not propagate the new sidecar environment to the failing pod, while deleting/restarting the pod causes the updated configuration to be picked up during recreation.

Unless I'm missing another reconciliation path, this looks like a real propagation/reconciliation gap rather than user error.