Policies applied but Tenant is shown as blocked with Live RustFS policy changed since the operator last applied it

[mskitroot]

New Tenant applied. Bucket, Policies and Users are created.

Operator shows "Live RustFS policy changed since the operator last applied it"

Tenant Configuration:

apiVersion: rustfs.com/v1alpha1
kind: Tenant
metadata:
  name: rfsd01
  labels:
    app: rfsd01
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  image: rustfs/rustfs:latest
  podManagementPolicy: Parallel
  tls:
    mode: certManager
    mountPath: /var/run/rustfs/tls
    rotationStrategy: Rollout
    enableInternodeHttps: true
    requireSanMatch: true
    certManager:
      manageCertificate: true
      certificateName: rfsd01-server-cert
      secretName: rfsd01-server-tls
      issuerRef:
        group: cert-manager.io
        kind: ClusterIssuer
        name: cluster-issuer
      includeGeneratedDnsNames: true
      dnsNames:
        - rfsd01-io.rfsd01.svc
        - rfsd01-io.rfsd01.svc.cluster.local
        - rfsd01-console.rfsd01.svc
        - rfsd01-console.rfsd01.svc.cluster.local
        - rfsd01-hl.rfsd01.svc
        - rfsd01-hl.rfsd01.svc.cluster.local
      duration: 4380h
      renewBefore: 730h
      usages:
        - server auth
      caTrust:
        source: CertificateSecretCa
        trustSystemCa: false
        trustLeafCertificateAsCa: false
  credsSecret:
    name: rfsd01-admin-creds
  env:
    - name: RUST_LOG
      value: info
  pools:
    - name: pool-0
      servers: 1
      persistence:
        volumesPerServer: 4
        volumeClaimTemplate:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 12Gi
      resources:
        requests:
          cpu: 500m
          memory: 1Gi
        limits:
          memory: 4Gi
  policies:
    - name: rfsd01-rw
      document:
        configMapKeyRef:
          name: rfsd01-readwrite-policy
          key: policy.json
    - name: rfsd01-ro
      document:
        configMapKeyRef:
          name: rfsd01-readonly-policy
          key: policy.json
  users:
    - name: rfsd01-rw-user
      policies:
        - rfsd01-rw
    - name: rfsd01-ro-user
      policies:
        - rfsd01-ro
  buckets:
    - name: rfsd01-data

Policy Example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: rfsd01-readwrite-policy
  labels:
    rustfs.tenant: rfsd01
  annotations:
    argocd.argoproj.io/sync-wave: "0"
data:
  policy.json: |
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "s3:ListBucket",
            "s3:GetObject",
            "s3:DeleteObject",
            "s3:PutObject"
          ],
          "Resource": [
            "arn:aws:s3:::rfsd01-data",
            "arn:aws:s3:::rfsd01-data/*"
          ]
        }
      ]
    }

[mskitroot]

Root cause

Provisioning drift detection hashes the policy JSON from the Tenant ConfigMap and compares it to the live canned policy returned by RustFS (list_canned_policies / get_canned_policy).

RustFS does not round-trip the document verbatim. Stored policies include extra envelope fields and formatting differences, for example:

• "ID": ""
• "Sid": "" on each statement
• "Condition": {} on each statement
• Different ordering of Action array entries

The operator hashed raw JSON, so semantically identical policies produced different SHA-256 values. After the first successful apply, a subsequent reconcile saw lastAppliedHash != liveHash (or liveHash != desiredHash) and reported:

Live RustFS policy changed since the operator last applied it

…even though nothing had been changed externally and the policy on the server matched the spec.

Fix

Normalize policy documents before hashing:

• Strip empty RustFS-only fields (ID, empty Sid, empty Condition)
• Sort Action / Resource arrays and statements for stable comparison
• Treat live policies that match spec as Ready when status tracking hashes are stale after upgrades

Fix PR: #146

Verification

Tested on a live cluster (rfsd01): Tenant Ready=True, all spec policies Ready, policy recreated after manual deletion when reconcile was triggered.