fix(sts): rotate TLS certificates

Cargo.toml

@@ -11,7 +11,7 @@ homepage = "https://rustfs.com"
 chrono = { version = "0.4", features = ["serde"] }
 const-str = "1.0.0"
 serde = { version = "1.0.228", features = ["derive"] }
-tokio = { version = "1.49.0", features = ["rt", "rt-multi-thread", "macros", "fs", "io-std", "io-util"] }
+tokio = { version = "1.49.0", features = ["rt", "rt-multi-thread", "macros", "fs", "io-std", "io-util", "sync"] }
 tokio-rustls = "0.26"
 tokio-stream = { version = "0.1", features = ["sync"] }
 tokio-util = { version = "0.7", features = ["io", "compat", "rt"] }
@@ -30,6 +30,8 @@ rustls = { version = "0.23", default-features = false, features = ["ring"] }
 rustls-pemfile = "2.2.0"
 webpki = { package = "rustls-webpki", version = "0.103" }
 rcgen = "0.13"
+x509-parser = "0.18.1"
+time = "0.3.44"
 sha2 = "0.10"
 hmac = "0.12"
 hex = "0.4"

deploy/k8s-dev/operator-rbac.yaml

@@ -92,8 +92,8 @@ subjects:
     name: rustfs-operator
     namespace: rustfs-system
 ---
-# STS auto TLS needs to create one Secret and replace it only when a previously generated
-# Secret becomes invalid. Keep write access in the operator namespace.
+# STS auto TLS needs to create one Secret and replace it when a previously generated Secret
+# becomes invalid, uses the legacy policy, or approaches expiry. Keep write access namespaced.
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:

deploy/rustfs-operator/README.md

@@ -123,7 +123,7 @@ manifests remain consistent.
 | `sts.audience` | Kubernetes TokenReview audience expected by the operator STS endpoint | `sts.rustfs.com` |
 | `sts.port` | Operator container port for STS | `4223` |
 | `sts.tls.enabled` | Serve the operator STS endpoint over TLS | `true` |
-| `sts.tls.auto` | Create and replace an invalid Operator-managed STS TLS Secret with namespaced write RBAC | `true` |
+| `sts.tls.auto` | Create and rotate an invalid, legacy, or soon-to-expire Operator-managed STS TLS Secret with namespaced write RBAC | `true` |
 | `sts.service.type` | Kubernetes Service type for STS | `ClusterIP` |
 | `sts.service.port` | Kubernetes Service port for STS | `4223` |
 
@@ -135,7 +135,9 @@ POST /sts/{tenantNamespace}/{tenantName}
 
 This differs from MinIO Operator's namespace-only route. A `PolicyBinding` still lives in the Tenant namespace, but the workload must call STS with both the Tenant namespace and the Tenant name.
 
-The STS service is HTTPS by default. When `sts.tls.auto=true`, the operator creates the fixed `sts-tls` Secret in the operator namespace with `tls.crt`, `tls.key`, and `ca.crt`. With `rbac.create=true`, the chart creates a namespaced Role that can create Secrets and update only `sts-tls`; the ClusterRole keeps all Secret and ConfigMap access read-only. If `rbac.create=false`, you must provide an equivalent Role and RoleBinding for the operator ServiceAccount: namespaced Secret `create`, plus `get` and `update` restricted to the `sts-tls` resource name. Workloads must trust that CA. To use an externally issued certificate, pre-create `sts-tls` with a certificate signed by a CA already trusted by the workload and set `sts.tls.auto=false`; the chart then omits the namespaced Secret write Role.
+The STS service is HTTPS by default. When `sts.tls.auto=true`, the operator creates the fixed `sts-tls` Secret in the operator namespace with `tls.crt`, `tls.key`, and `ca.crt`. Generated certificates are valid for one year and are replaced 30 days before expiry. The operator checks the Secret every five minutes and hot-loads a valid replacement for new TLS connections while retaining the last valid configuration on refresh failures. Existing Operator-managed Secrets created with the legacy long-lived policy are replaced once after upgrade; refresh every STS client's trusted `ca.crt` as part of that upgrade. With `rbac.create=true`, the chart creates a namespaced Role that can create Secrets and update only `sts-tls`; the ClusterRole keeps all Secret and ConfigMap access read-only. If `rbac.create=false`, you must provide an equivalent Role and RoleBinding for the operator ServiceAccount: namespaced Secret `create`, plus `get` and `update` restricted to the `sts-tls` resource name. Workloads must trust that CA. To use an externally issued certificate, pre-create `sts-tls` with a certificate signed by a CA already trusted by the workload and set `sts.tls.auto=false`; update that Secret to rotate it manually, and the operator hot-loads the valid replacement within five minutes. The chart then omits the namespaced Secret write Role.
+
+Monitor `rustfs_operator_sts_tls_certificate_expiry_timestamp_seconds` and `rustfs_operator_sts_tls_ca_expiry_timestamp_seconds` and alert before either timestamp is reached.
 
 STS only issues credentials for TLS-enabled Tenants. For Tenant upstream calls, the operator selects the Tenant HTTPS service endpoint and trusts the CA recorded in `status.certificates.tls.caSecretRef`.
 

deploy/rustfs-operator/values.yaml

@@ -128,9 +128,9 @@ sts:
     # Operator STS listens with TLS by default and uses this Secret for server certs.
     enabled: true
     # When true, the chart grants namespaced create access plus update access to the fixed
-    # sts-tls Secret. The operator creates a missing Secret and replaces an invalid Secret only
-    # when it is marked as Operator-managed. If rbac.create=false, provide an equivalent Role and
-    # RoleBinding in the operator namespace.
+    # sts-tls Secret. The operator creates a missing Secret and replaces an invalid, legacy, or
+    # soon-to-expire Secret only when it is marked as Operator-managed. If rbac.create=false,
+    # provide an equivalent Role and RoleBinding in the operator namespace.
     auto: true
   service:
     type: ClusterIP

docs/operator-user-guide.md

@@ -327,7 +327,8 @@ Notes:
 - `operator.leaderElect` can be unset. The chart enables leader election automatically when `operator.replicas > 1`.
 - Keep `console.jwtSecret` stable when running multiple Console replicas. If unset, the chart generates or reuses a Secret.
 - Keep `CONSOLE_COOKIE_SECURE` enabled for production HTTPS. Only disable it for local HTTP testing.
-- `sts.tls.auto=true` lets the operator create or repair `sts-tls`; with `rbac.create=true`, the chart isolates write access in a namespaced Role while keeping cluster-wide Secret and ConfigMap access read-only. With `rbac.create=false`, provide an equivalent Role and RoleBinding for the operator ServiceAccount: namespaced Secret `create`, plus `get` and `update` restricted to the `sts-tls` resource name.
+- `sts.tls.auto=true` lets the operator create or repair `sts-tls`. Generated certificates are valid for one year and rotate 30 days before expiry. The operator checks every five minutes and hot-loads valid changes for new connections while retaining the last valid configuration on errors. A legacy Operator-managed Secret is replaced once after upgrade, so refresh clients that trust its `ca.crt`. With `rbac.create=true`, the chart isolates write access in a namespaced Role while keeping cluster-wide Secret and ConfigMap access read-only. With `rbac.create=false`, provide an equivalent Role and RoleBinding for the operator ServiceAccount: namespaced Secret `create`, plus `get` and `update` restricted to the `sts-tls` resource name.
+- With `sts.tls.auto=false`, replace `sts-tls` to rotate an externally issued certificate manually; a valid replacement is hot-loaded within five minutes. Monitor `rustfs_operator_sts_tls_certificate_expiry_timestamp_seconds` and `rustfs_operator_sts_tls_ca_expiry_timestamp_seconds`.
 
 ## 6. Create a Tenant
 

docs/operator-user-guide.zh-CN.md

@@ -304,7 +304,8 @@ sts:
 - `operator.leaderElect` 可以不配置；当 `operator.replicas > 1` 时 Chart 会自动启用 leader election。
 - 多副本 Console 部署需要保持 `console.jwtSecret` 稳定；不设置时 Chart 会生成或复用已有 Secret。
 - 生产环境应使用 HTTPS 并保持 `CONSOLE_COOKIE_SECURE` 启用。仅本地 HTTP 调试时才关闭。
-- `sts.tls.auto=true` 时，Operator 会创建或修复 `sts-tls`；当 `rbac.create=true` 时，Chart 通过命名空间级 Role 隔离写权限，全集群 Secret 和 ConfigMap 权限保持只读。当 `rbac.create=false` 时，必须为 Operator ServiceAccount 自行提供等效的 Role 和 RoleBinding：允许在 Operator namespace 内 `create` Secret，并将 `get`、`update` 限定到名为 `sts-tls` 的资源。
+- `sts.tls.auto=true` 时，Operator 会创建或修复 `sts-tls`。自动生成的证书有效期为一年，并在到期前 30 天轮换。Operator 每五分钟检查一次 Secret，对新连接热加载有效变更；检查失败时继续使用最后一个有效配置。升级后，旧策略生成的 Operator 托管 Secret 会被替换一次，因此需要同步刷新客户端信任的 `ca.crt`。当 `rbac.create=true` 时，Chart 通过命名空间级 Role 隔离写权限，全集群 Secret 和 ConfigMap 权限保持只读。当 `rbac.create=false` 时，必须为 Operator ServiceAccount 自行提供等效的 Role 和 RoleBinding：允许在 Operator namespace 内 `create` Secret，并将 `get`、`update` 限定到名为 `sts-tls` 的资源。
+- `sts.tls.auto=false` 时，可通过替换 `sts-tls` 手动轮换外部签发证书；有效的新证书会在五分钟内热加载。请监控 `rustfs_operator_sts_tls_certificate_expiry_timestamp_seconds` 和 `rustfs_operator_sts_tls_ca_expiry_timestamp_seconds`。
 
 ## 6. 创建 Tenant