Add watch.labelSelector for label-scoped operator instances

[27rohan]

Problem

Today a single operator instance manages every ClickHouseInstallation (CHI) and ClickHouseKeeperInstallation (CHK) in its watched namespaces. watch.namespaces is the only way to split a fleet across operator instances, which does not help when many CHIs live in the same namespaces. At fleet scale this makes one operator both a throughput bottleneck (one slow reconcile delays every other CR) and a blast-radius problem (an operator bug or a bad operator upgrade hits the whole fleet at once, and a canary rollout of a new operator version is impossible).

Proposal

A new watch.labelSelector setting lets several operator instances run side by side, each managing a disjoint, label-defined subset of CRs:

watch:
  # Standard Kubernetes label selector syntax. This operator instance
  # only manages CHI/CHK resources whose labels match.
  labelSelector: "example.com/clickhouse-shard=shard-a"

  # Abort startup when labelSelector is empty — recommended on scoped
  # deployments so a lost/typo'd selector fails loudly instead of the
  # operator silently taking over the whole fleet.
  requireLabelSelector: true

A typical layout: one operator per shard value, plus a legacy catch-all with labelSelector: "!example.com/clickhouse-shard" for unlabeled CRs. With one value per shard and an absence-based catch-all, every possible label state matches exactly one operator. This enables gradual operator upgrades (canary one shard), fault isolation, and horizontal scaling of reconcile throughput.

Behavior

• Standard selector syntax — anything k8s.io/apimachinery's labels.Parse accepts: equality, set-based, presence/absence, conjunctions. An invalid selector aborts startup.
• Env overrides — WATCH_LABEL_SELECTOR and WATCH_LABEL_SELECTOR_REQUIRED. An empty WATCH_LABEL_SELECTOR env var does not clear a file-configured selector (same semantics as WATCH_NAMESPACES), so an accidentally-unset env var cannot silently widen an operator's scope.
• Filtering is enforced in depth: informer event handlers, plus a guard inside reconcile (requests originating from owned StatefulSets bypass predicates), plus a live-state ownership re-check before any write (finalizer install, child reconcile, deletion protocol) so two operators never fight over a CR mid-flip.
• Status writes are fenced against flips: statusUpdateProcess (CHI and CHK) validates ownership on the same object snapshot whose resourceVersion fences the update — a flip landing after the read causes a conflict, and the retry re-runs the check. This centrally covers every CR .status path (reconcile start, abort, host progress, completion) with no extra API reads for the guard itself; unsharded operators are unaffected. Verified by a race-path test that injects the flip between the snapshot read and the write and asserts the conflicted retry skips.
• Label flip-away is an unwatch, not a delete — the losing operator stops metrics scraping and drops in-memory state but leaves the CR and all child objects intact for the operator that now matches.
• Zero pod restarts on re-assignment — every label key referenced by the selector is automatically appended to label.exclude, so shard labels never propagate to child objects and flipping a CR's label hands it over without touching running ClickHouse pods.
• Metrics-exporter follows the same scope — each exporter only scrapes the CHIs its operator manages.
• Observability — skipped CRs are counted in a new clickhouse_operator_cr_skipped_by_label_selector metric and logged.

Known limitations

Two windows remain open by design, both pre-existing classes of handoff race that also apply to watch.namespaces changes today, not regressions introduced here:

• A label flip cannot stop an already-running reconcile from mutating child resources (StatefulSets, ConfigMaps, Services). Closing this fully needs per-mutation ownership checks or lease-based fencing — happy to open a follow-up discussion if there's interest.
• The CHI chi-storage-* auxiliary status ConfigMap is written behind the snapshot ownership check but is a separate object not fenced by the CHI resourceVersion, so a flip landing after the snapshot read can produce one stale ConfigMap write (the CR .status update itself still conflicts and is skipped). It is rewritten by the new owner's next status update; reordering it after the fenced CR write would invert the intentional aux-resources-first ordering, so it is left as-is.

In both cases the effect is transient and self-healing: the operator that owns the CR after the flip overwrites the stale data on its next reconcile.

Backward compatibility

Fully opt-in. labelSelector defaults to empty and requireLabelSelector to false, preserving today's watch-everything behavior. No API/CRD changes to CHI/CHK themselves; the chopconf CRD schema gains the two optional fields. ClickHouseInstallationTemplates are intentionally not filtered — templates stay visible to every operator instance.

Changes

• watch.labelSelector + watch.requireLabelSelector config fields, env overrides, startup validation
• CHI/CHK controllers: event filtering, reconcile guards, live-ownership confirmation, flip-away unwatch handling
• automatic label.exclude derivation from selector keys
• metrics-exporter discovery filtering
• new skip metric
• docs (docs/operator_configuration.md), chopconf CRD schema, config templates, install manifests, helm chart values + schema

Testing

• Unit tests across config parsing/merging/env-override semantics, selector disjointness, CHI/CHK controller filtering, flip-away handling, keeper thread guards, and exporter filtering (~900 lines of new tests)
• go build ./...
• go test -race -count=1 -vet=off ./pkg/apis/clickhouse.altinity.com/v1/ ./pkg/controller/chi/ ./pkg/controller/chk/ ./pkg/metrics/... ./pkg/chop/... ./cmd/operator/app/ — all pass (-vet=off because go test vet flags pre-existing non-constant-format-string calls in untouched files on master)
• The feature has been running in a production multi-operator deployment (two label-scoped operators plus an absence-selector legacy operator) including live shard re-assignment with zero ClickHouse pod restarts

Note: manifests were edited by hand in lockstep with deploy/builder/templates-config because the builder scripts are not runnable on macOS (GNU tooling assumptions); happy to regenerate if CI or maintainers run the builders.