Deployment/StatefulSet pods don't restart when secret.rustfs.access_key/secret_key (or secret.existingSecret) is rotated

[alex-mabrouk]

Title

Helm chart: Deployment/StatefulSet pods don't restart when secret.rustfs.access_key/secret_key (or secret.existingSecret) is rotated

Body

Chart: rustfs (helm/rustfs), version 0.8.0

Problem

templates/deployment.yaml (and templates/statefulset.yaml) load credentials into the
container purely via:

envFrom:
  - secretRef:
      name: {{ include "rustfs.secretName" . }}

Kubernetes only reads envFrom/env values when a Pod is created — it never live-updates a
running container's environment when the referenced Secret's contents change. So if a user
changes secret.rustfs.access_key / secret.rustfs.secret_key (or updates the Secret named by
secret.existingSecret) and runs helm upgrade, Helm patches the Secret object, but the
existing Deployment/StatefulSet pod template hash doesn't change (nothing in spec.template
references the secret content), so no rollout is triggered and the running pod keeps using
the old credentials indefinitely.

values.yaml already declares a podAnnotations: {} value (line ~150), but it's never consumed
by templates/deployment.yaml or templates/statefulset.yaml, so there's no way to wire in the
standard checksum/secret annotation pattern from chart values alone.

Impact

Any credential rotation via helm upgrade silently no-ops on the running pod. In our case this
surfaced as a helm upgrade ... --wait timeout: our own post-upgrade provisioning hook polls the
RustFS S3 API with the new credentials, but the still-running pod only accepts the old ones,
so the hook spins until the wait timeout is hit and the release is reported failed — even though
from RustFS's perspective nothing is actually broken, it's just serving stale credentials.

Suggested fix

Wire .Values.podAnnotations into the pod template in both deployment.yaml and
statefulset.yaml (as most charts do), and/or add a checksum/secret annotation on the pod
template computed from the rendered secret.yaml, e.g.:

template:
  metadata:
    annotations:
      checksum/secret: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }}
      {{- with .Values.podAnnotations }}
      {{- toYaml . | nindent 8 }}
      {{- end }}

Either would let users trigger a rollout on credential rotation without having to manually
delete the pod.

To reproduce

1. helm install rf rustfs/rustfs --set secret.rustfs.access_key=... --set secret.rustfs.secret_key=...
2. helm upgrade rf rustfs/rustfs --set secret.rustfs.access_key=... --set secret.rustfs.secret_key=<new value>
3. kubectl exec into the running pod and check RUSTFS_SECRET_KEY — still the old value, pod age unchanged.