fix-1828: started work

[wind57]

No description provided.

[wind57]

@ryanjbaxter can you trigger copilot in here please?

[wind57]

in the new IT that I added, there is a need for two replicas, to really test the HA set-up

[wind57]

two properties are needed to enable HA

[wind57]

I've added a single IT, that goes through a cycle of leader / no leader / leader

[wind57]

this is the definition of the resource version store. Methods in this one are executed only by the leader

[wind57]

if HA is enabled and this is the first call, this will create an empty lease. Otherwise, it will read whatever is stored there.

We store in the spring.cloud.kubernetes.configuration.watcher/configmap-resource-version annotation, something like : "default=123,prod=456", so each namespace tracks its own resource version checkpoint.

We need such a store because during a downtime when there is no leader at all, resource version can progress ( meaning configmap is updated ), but since there is no leader, events can get lost. As such, we always increment and "store" ( via this implementation ) the most recent resource version we have observed. This happens in the handlers onAdd / onDelete / onUpdate.

So for example:

• we are now the leader and the resourceVersion is at 1.
• we lose leadership, so our store stays at 1.
• configmap progresses to resourceVersion=2
• another leader is elected, it reads the store, sees that it holds 1
• starts the informer at resourceVersion=1, so it can replay events it has not seen

[wind57]

these events are triggered only when the instance became the leader

[wind57]

after every update that we receive in the handler, write the resource version to the store

[wind57]

With HA enabled:

• @PostConstruct does nothing, so the informers are not started.

• The native leader-election callback emits StartLeadingEvent.

• ConfigurationWatcherHACoordinator receives that event.

• It calls stateStore.readOrCreate()

  • If the HA state Lease does not exist, it creates it and returns an empty state.
  • Otherwise, it reads the stored ConfigMap and Secret resource versions.

• The coordinator calls start(...) on the available detectors, passing:

  • the stored resource versions
  • a writer that persists future resource versions

• Each detector creates its InformerResourceVersionResolver.

• For each watched namespace, the informer sends its initial LIST request:

  • if a stored checkpoint exists, the request starts from that resource version
  • otherwise, it starts normally

• The informer then starts its WATCH.

• Resource events are handled normally:

  • ConfigMap or Secret change is processed
  • refresh or bus notification is triggered
  • the processed resource version is written to the HA state Lease

• Later LIST/WATCH requests use the resource version supplied by the informer, not the stored checkpoint.

• If the leader loses leadership, StopLeadingEvent is emitted.

• The coordinator stops both detectors.

• The next leader reads the persisted state and repeats the process from the stored checkpoints.

[wind57]

TL;DR

This PR adds high-availability support to the Kubernetes client-based Configuration Watcher.

The Configuration Watcher can now run with multiple replicas while ensuring that only one replica actively watches ConfigMaps and Secrets at a time. If the leader fails, another replica acquires leadership, restores the persisted informer checkpoints, and continues processing changes.

How It Works

HA requires both properties to be enabled:

spring:
  cloud:
    kubernetes:
      leader:
        election:
          enabled: true
      configuration:
        watcher:
          ha:
            enabled: true

The two properties have separate responsibilities:

• spring.cloud.kubernetes.leader.election.enabled enables native Kubernetes leader election.
• spring.cloud.kubernetes.configuration.watcher.ha.enabled makes the Configuration Watcher start and stop its informers according to leadership.

When HA is disabled, the existing behavior is unchanged. ConfigMap and Secret informers start during normal bean initialization.

When HA is enabled:

• Informers are not started during initialization.
• The leader-election callback notifies the Configuration Watcher that leadership was acquired.
• The Configuration Watcher reads its persisted state.
• ConfigMap and Secret informers start using the stored resource-version checkpoints.
• When leadership is lost, both informers are stopped.
• The next leader restores the checkpoints and resumes watching.

The HA coordinator is initialized before the leader-election callbacks so that it is ready to receive leadership events.

Persistent State

The Configuration Watcher stores its state in a Kubernetes Lease. The leader-election lock and the Configuration Watcher state lease are separate resources.

The state lease stores the last processed resource version independently for:

• each watched namespace
• ConfigMaps
• Secrets

The values are stored in annotations on the lease. For example:

spring.cloud.kubernetes.configuration.watcher/configmap-resource-version:
  default=123,other-namespace=456

The default configuration is:

spring:
  cloud:
    kubernetes:
      configuration:
        watcher:
          ha:
            enabled: false
            lease-name: configuration-watcher-ha
            lease-namespace: default

The lease is created automatically when the first leader starts, if it does not already exist.

Resource-Version Replay

• When a new leader starts, the persisted resource version is used only for the first informer request for each namespace. This request restores the informer from the exact stored checkpoint.
• After that first request, the informer controls its own resource-version progression. Each subsequent list or watch request uses the resource version supplied by the informer.
• ConfigMap and Secret namespaces are tracked independently, so different namespaces can progress at different resource versions.
• Resource versions are persisted after the corresponding resource event has been handled. This provides at-least-once event processing:
  • If a leader processes an event and persists the checkpoint successfully, the next leader continues after that event.
  • If the leader fails after processing the event but before persisting the checkpoint, the replacement leader may process the event again.

Repeated refresh notifications must therefore be tolerated by refresh targets.

• The watcher does not refresh every ConfigMap or Secret after a leadership change. It only replays events for the configured informer scopes.

• At least two Configuration Watcher replicas are required for failover:

spec:
  replicas: 2