[KYUUBI #7556] Rearm the ZooKeeper engine deregistration watcher after reconnect

[Yangxuhao123]

Why are the changes needed?

Closes #7556.

ZooKeeper watches are one-shot. After a Kyuubi engine's ZooKeeper session expires and reconnects, its service node is restored, but the DeRegisterWatcher is not automatically rearmed. A subsequent node deletion may therefore be missed, preventing graceful shutdown and allowing an unhealthy engine to remain discoverable.

This patch rearms the watcher asynchronously after RECONNECTED, retries when the service node is temporarily unavailable, handles node recreation, and invalidates stale or pending rewatch attempts after a newer reconnect or service deregistration.

How was this patch tested?

Added embedded ZooKeeper test coverage for:

• Rearming the service watcher after reconnect.
• Triggering graceful shutdown after a real RECONNECTED event followed by node deletion.
• Rearming after an absent node is recreated.
• Cancelling pending rewatch attempts during deregistration.
• Rearming binary and HTTP discovery clients independently.

The following test suite was executed:

./build/mvn test -pl kyuubi-ha -am \
  -Dtest=none \
  -DwildcardSuites=org.apache.kyuubi.ha.client.zookeeper.EmbeddedZookeeperDiscoveryClientSuite

All 16 tests passed.

Was this patch authored or co-authored using generative AI tooling?

Assisted-by: OpenAI Codex (GPT-5)