Scope kube informer caches to operator-generated objects; make informer resync configurable

[gregakinman]

Problem

The operator's informer memory footprint scales with total cluster size instead of with the number of ClickHouse installations it manages. On a large cluster with many ephemeral namespaces this OOM-kills the operator.

GetInformerNamespace() returns meta.NamespaceAll whenever watchNamespaces contains anything other than exactly one literal DNS label, so any regexp watch config forces cluster-wide informers on Pods, Services, EndpointSlices, ConfigMaps and StatefulSets. Filtering then happens client-side in isTrackedObject, i.e. after every object in the cluster has already been transferred, decoded and queued in the DeltaFIFO.

Compounding it, defaultInformerFactoryResyncPeriod was hardcoded with no flag or env var, so the entire cluster-wide cache replayed through the DeltaFIFO as sync deltas every minute with no way to tune it.

Changes

1. Push the label filter server-side. kubeinformers.WithTweakListOptions sets opts.LabelSelector on the kube informer factory. The selector is built from the same labeler that IsCHOPGeneratedObject() resolves through, so the server-side selector and the client-side predicate cannot drift apart. CHOp custom resource informers (CHI/CHIT/CHOpConfig) are intentionally left unfiltered — they are cluster-wide by necessity and small.

2. Make the resync periods configurable. --kube-informer-resync-period / OPERATOR_KUBE_INFORMER_RESYNC_PERIOD and --chop-informer-resync-period / OPERATOR_CHOP_INFORMER_RESYNC_PERIOD. Precedence is flag > env > default.

Defaults are deliberately left at 60s, so this change is purely additive — no existing deployment changes behavior. Scoping the cache in (1) is what actually bounds resync cost, and resync is cheaper in this operator than it first appears: the ConfigMap and StatefulSet UpdateFuncs only log, EndpointSlice is gated on isUpdatedEndpointSlice and resync feeds old == new so it never enqueues, and CHI resync enqueues but the worker exits early on isGenerationTheSame. Only the Pod handler enqueues unconditionally, and after (1) that is bounded by the number of operator-managed pods. Raising the period trades drift-detection latency for DeltaFIFO churn, so it is exposed as a knob rather than repriced for everyone.

3. Scope the keeper manager's StatefulSet cache. Owns(&apps.StatefulSet{}) starts a StatefulSet informer unconditionally, for every operator, whether or not any CHK exists — and GetCacheNamespaces() falls back to NamespaceAll under the same regexp condition, so every operator cached every StatefulSet in the cluster, full pod template spec each.

4. Remove the CHK watcher's private resync constant. It watches a custom resource, so it now receives the CHOp custom resource resync period and honors the flag and env var from (2). Its effective default is unchanged.

Equivalence

The label selector is intended to be exactly equivalent to the pre-existing client-side filtering. Verified:

• All handler funcs registered on kubeInformerFactory already gate on isTrackedObject, so no handler regresses.
• There are no Lister() calls and no GetStore()/GetIndexer() reads on the kube informers anywhere in the tree — the caches are pure event sources, so nothing can read an object the selector excludes.
• The metrics exporter reads through direct API calls, not the informer cache.
• The CHK watcher builds its own dynamic informer, independent of the factory.

Two things worth reviewer attention:

• CHI and CHK use different label keys. clickhouse.altinity.com/app vs clickhouse-keeper.altinity.com/app, and a k8s label selector cannot express a disjunction over two keys. This is fine because isTrackedObject uses the CHI labeler exclusively, so CHK-generated objects were already dropped client-side; and the keeper cache gets its own separate selector. Tests assert the two selectors are mutually non-matching so this can't silently regress.
• EndpointSlices are populated by Kubernetes, not by the operator. They inherit the chop label from the chop-created Service (the endpointslice controller mirrors Service labels onto slices, excluding reserved prefixes). Verified on two live clusters: 179/179 and 19/19 chop Services had a correspondingly labelled EndpointSlice, zero misses. The Endpoints informer is never constructed — that handler is commented out — so legacy Endpoints are moot either way.

For the keeper cache, only StatefulSets are scoped. Point Gets on StatefulSets go through the non-cached APIReader, and the only cached-client StatefulSet reads are Lists that already carry this very selector, built from the same labeler in the CHK discoverer. The other types the CHK adapter reads through the cached client (ConfigMap, Service, Secret, PVC, PDB, Pod) are cached lazily, only once a CHK is actually reconciled, and are deliberately left alone here — I had no cluster with CHKs deployed to exercise those read paths against.

Testing

• go build ./... clean.
• go vet produces an identical finding set before and after the change (compared in the same tree, same build cache) — no new findings.
• Existing tests pass. Note go test currently requires -vet=off on master due to pre-existing vet failures, and TestEnforceVerifiedLegacyTLS fails on macOS independently of this change.
• New tests pin the literal selector strings, assert the CHI selector agrees with IsCHOPGeneratedObject() across a table of label sets, and assert the CHI and CHK selectors never match each other's objects.
• Operator image builds; the new flags are present in the built binary and the env var overrides take effect inside the container.

Measured on one production-shaped cluster with a regexp watch config, cached objects across the five watched types drop from 22,194 to 761.

🤖 Generated with Claude Code