fix(pd): keep KvClient watches alive after reconnect failures

[contrueCT]

Purpose of the PR

• close [Bug] PD KvClient watch can permanently stop after reconnect failure #3152

KvClient previously retried a failed watch only once. If that reconnect also failed, the
watch stopped permanently, and unexpected stream completion did not trigger recovery.

The reconnect path also exposed a lifecycle issue: AbstractClient.resetStub() invoked the
virtual close() method, so transport initialization on a KvClient dispatched to
KvClient.close() and marked the whole client closed.

Reconnect Lifecycle

Main Changes

• Separate AbstractClient transport cleanup from the overridable client lifecycle close.
• Track each exact-key or prefix watch as an independent subscription with its current observer.
• Retry onError, Leader_Changed, and unexpected onCompleted with a fixed delay until the
  watch recovers or the client closes.
• Deduplicate reconnect scheduling for the same observer and ignore stale observer callbacks.
• Stop the reconnect executor and invalidate active observers when KvClient.close() is called.
• Add deterministic regression coverage for repeated failures followed by recovery, leader
  changes, completion, deduplication, stale observers, prefix semantics, and close behavior.

This PR does not add event replay, protocol revisions, PD Server changes, or the Server-side
reconciliation tracked by #3151. Events emitted while a watch is disconnected are still not
replayed.

Verifying these changes

• Trivial rework / code cleanup without any test coverage. (No Need)
• Already covered by existing tests, such as (please modify tests here).
• Need tests and can be verified as follows:
  • mvn -q -o test -pl hugegraph-pd/hg-pd-test -am -Dtest=KvClientTest -DfailIfNoTests=false -Drat.skip=true -Djacoco.skip=true (12 tests, 0 failures/errors)
  • mvn -q -o package -pl hugegraph-pd -am -DskipTests -Dmaven.javadoc.skip=true -Drat.skip=true -Djacoco.skip=true
  • mvn -q -o -f hugegraph-pd/hg-pd-client/pom.xml apache-rat:check
  • mvn -q -o -f hugegraph-pd/hg-pd-test/pom.xml apache-rat:check

The full local PD suite was also attempted. The common suite passed 83/83 and the core suite
passed 89/91 with 2 skipped. The client suite could not complete without a local PD service at
127.0.0.1:8686 (4 failures and 5 errors reported connection refused), which prevented the rest
suite from executing in that Maven run.

Repository-wide RAT is not a valid signal in this checkout because pre-existing ignored
.upgrade-artifacts files produce 68 unrelated unapproved-license entries; RAT passes for both
changed modules.

Does this PR potentially affect the following parts?

• Dependencies (add/update license info & regenerate_known_dependencies.sh)
• Modify configurations
• The public API
• Other affects: PD KV watch reconnect lifecycle
• Nope

Documentation Status

• Doc - TODO
• Doc - Done
• Doc - No Need

Visual summary

[imbajin]

Blocking: yes. Summary: The reconnect implementation has critical identity and failure-propagation risks that can silently stop watches and invalidate existing locks; permanent callback errors also retry without termination. Evidence: static review of KvClient.java and AbstractClient.java at head 772b3a5, with exact server lock/watch handling cross-checked.

[codecov]

Codecov Report

❌ Patch coverage is 72.41379% with 104 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.35%. Comparing base (7bb624b) to head (3124fae).
⚠️ Report is 18 commits behind head on master.

Files with missing lines	Patch %	Lines
.../java/org/apache/hugegraph/pd/client/KvClient.java	76.70%	43 Missing and 22 partials ⚠️
...org/apache/hugegraph/pd/client/AbstractClient.java	60.20%	26 Missing and 13 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3157      +/-   ##
============================================
+ Coverage     32.68%   41.35%   +8.66%     
- Complexity     5516     7294    +1778     
============================================
  Files           789      802      +13     
  Lines         67750    69659    +1909     
  Branches       8953     9285     +332     
============================================
+ Hits          22147    28805    +6658     
+ Misses        42990    37569    -5421     
- Partials       2613     3285     +672     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[imbajin]

Blocking: yes. Summary: The current head addresses the previously reported watch reconnect, lock identity, terminal-error retry, and stale-observer issues; focused validation passes, but coverage checks remain failed and independent review evidence is incomplete. Evidence: mvn -pl hugegraph-pd/hg-pd-test -am -Dtest=KvClientTest -DfailIfNoTests=false test (18/18 passed), git diff --check passed; codecov/project and codecov/patch failed.

[bitflicker64]

Blocking: no. Summary: Splitting the lock and watch client IDs, tracking each watch as an independent subscription, and propagating the terminal streaming failure are all solid, and the regression suite is genuinely deterministic; two gaps remain in the reconnect loop itself, plus one visibility nit. Evidence: read of KvClient.java, AbstractClient.java and KvClientTest.java at 3c2f4df; repo greps for streamingCall/listen callers; PD server watch path in KvWatchSubject.notifyClientChangeLeader and KvServiceGrpcImpl.clientWatch; gh -R apache/hugegraph pr checks 3157 (17 pass, only codecov/patch and codecov/project fail).

[Copilot]

Pull request overview

This PR fixes PD KvClient watch reliability by ensuring watches continue to reconnect after repeated failures, leader changes, and unexpected stream completion, while also addressing a lifecycle bug where transport resets could inadvertently invoke subclass close() behavior.

Changes:

• Refactors AbstractClient to separate transport cleanup (closeConnections()) from the overridable close() lifecycle.
• Reworks KvClient watch handling to track independent watch subscriptions and keep reconnecting with deduped scheduled retries until recovery or client close.
• Adds regression tests covering repeated failures/recovery, leader changes, completion recovery, deduplication, stale observers, prefix semantics, and close behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
hugegraph-pd/hg-pd-client/src/main/java/org/apache/hugegraph/pd/client/AbstractClient.java	Avoids invoking subclass lifecycle close() during transport reset; improves streaming-call retry failure signaling.
hugegraph-pd/hg-pd-client/src/main/java/org/apache/hugegraph/pd/client/KvClient.java	Introduces per-watch subscription tracking and scheduled reconnect loop to prevent watches permanently stopping.
hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/client/KvClientTest.java	Adds deterministic coverage for reconnect behaviors and the transport initialization lifecycle regression.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[imbajin]

Blocking: yes. The latest update fixes the reconnect executor parking reported in comment 3885719895, and the focused KvClientTest passes 19/19, but four current-head races can still silently strand watches. The hstore check is red on VertexCoreTest.testQueryByJointIndexesWithSearchAndTwoRangeIndexesAndWithin; exact base 7bb624b fails the same assertion, so it appears baseline-related, but visible CI remains a merge gate.

[imbajin]

Blocking: yes. The focused KvClientTest passes 19/19, but four current-head lifecycle and failure-handling gaps can still strand watches. The hstore check is red on VertexCoreTest.testQueryByJointIndexesWithSearchAndTwoRangeIndexesAndWithin; exact base 7bb624b fails the same assertion, so it appears baseline-related, but visible CI remains a merge gate.

[imbajin]

Blocking: yes. Summary: the latest head fixes several earlier global-state and lifecycle issues, but four production correctness gaps remain. Evidence: exact head 4202f26, CI is green, and the focused tests pass 33/33; a separate design comment proposes consolidating the watch state machine instead of adding more guards.

[imbajin]

Blocking: yes. Summary: The single-thread reconnect scheduler can serialize all watches behind a 60-second peer-discovery attempt, delaying recovery and close. Evidence: exact head 0a77c55; PDConfig defaults grpcTimeOut to 60000 ms; reconnectExecutor is single-threaded; startWatch synchronously reaches AbstractClient.resetStub() and blocking getMembers().

[bitflicker64]

Blocking: yes. Summary: The AbstractClient transport/lifecycle split and the per-subscription WatchSubscription model are the right fixes, and the new watch tests genuinely run and pass, but the SchemaDriverTest added here fails CI at this head, and two new lifecycle paths hold a monitor across an unbounded channel shutdown loop. Evidence: exact head 0a77c55ac6fa0baaf0e97d98eb0d15a1c28a1531 read via gh api -X GET repos/apache/hugegraph/pulls/3157/files --paginate and gh api -X GET "repos/apache/hugegraph/contents/<path>?ref=0a77c55a" -H 'Accept: application/vnd.github.v3.raw'; CI via gh api -X GET repos/apache/hugegraph/commits/0a77c55a/check-runs (22 success, 2 failure) with logs from gh api -X GET repos/apache/hugegraph/actions/jobs/99501401894/logs; PDClientSuiteTest runs 45 tests on #3164 and 77 here, matching the 3 to 35 growth in KvClientTest.

Three smaller notes, not worth inline comments. The description says -Dtest=KvClientTest ran "12 tests, 0 failures/errors", but this head has 35 @Test methods in that class, so that figure looks like it came from the first of the six commits on the branch. "Main Changes" does not mention the hugegraph-struct and SchemaDriver work, which is two of the five changed files. And the hstore failure on VertexCoreTest.testQueryByJointIndexesWithSearchAndTwoRangeIndexesAndWithin is joint and range index behaviour with no path to the PD watch client or SchemaDriver, and that job passes on #3164, so I do not attribute it to this PR, though I did not reproduce it.

I did not restate the review comments already open at this head on watch readiness, unconditional reconnect, shouldRotateWatchTransport coverage, semaphore ownership, WatchRequest placement, stopped watch visibility, or the stale Started test gap. The three below are distinct from those. No local build backs these findings: fetching the head into a checkout was not available in this session, so this is static review of the exact head sources plus CI logs.

[bitflicker64]

Blocking: yes. Summary: the per-subscription watch model, the AbstractClient transport/lifecycle split and the 40-test regression suite are the right fixes for #3152 and all pass locally, but KvClient.stubResetTimeoutMillis() applies the 5 s watch-start budget to every blocking KV and lock call, cutting the honoured pd.grpc_timeout from a measured 21.3 s to a pinned 5.0 s on the same peers; SchemaDriver.destroy() also publishes the driver while its KvClient is already closed. Evidence: exact head f99b6bd vs merge base 7bb624b; mvn -o -pl hugegraph-pd/hg-pd-test -am test -Dtest=KvClientTest 40/40 and mvn -o -pl hugegraph-struct -am test -Dtest=SchemaDriverTest 5/5 pass; upstream CI green 24/24; timing probe over three unreachable peers with identical PDConfig gives KvClient.get() 4984/5213/5212 ms at head against 21328/35403/6511 ms at base, while MetaClient.getGraphs() on the same peers stays at 25071 ms.

[SebastianGruza]

Live before/after for this branch (head f99b6bd) on a real cluster: PD + 3 store nodes, two HugeGraph servers built from the same tree sharing the PD in PD-meta mode (usePD=true), server A on :8080 and server B on :8082. Server distributions built from master 98477f0 (before) and from this head (after); PD and stores unchanged.

Observable that lazy loading cannot compensate: in PD-meta mode GraphManager.graph() loads an unknown graph from PD meta on first access, so "B sees a graph created on A" proves nothing about the watch. Removal is different — B keeps a graph it has already constructed until the GRAPH/REMOVE event arrives. So each probe is: A creates g → B constructs it on first access (HTTP 200) → A drops g → poll B's GET …/graphs/g/schema/propertykeys until it stops answering 200. Between probes PD is stopped for 30 s (SIGTERM via stop-hugegraph-pd.sh, then kill -9 of the port owner) and restarted; the stores stayed up throughout.

Step	master 98477f0	this branch
baseline: create → drop	B dropped g1 after 1 s (Accept graph add signal seen once)	B dropped g1 after 1 s
PD down 30 s, back, settle 15 s → create → drop	B still serves the dropped g2 after 60 s (HTTP 200); no add signal in B's log → watch dead	B dropped g2 after 1 s; add signal seen
second PD outage → create → drop	B still serves g3 after 60 s; watch dead	B dropped g3 after 1 s; add signal seen

B's log on this branch during each outage: Failed to reconnect watch for key HUGEGRAPH/hg/EVENT/GRAPH/{ADD,REMOVE,UPDATE,CLEAR,SCHEMA/CLEAR} about once per second per subscription, then AbstractClient connect to host = … success and five fresh set watch client id lines within a second of PD listening again. On master B logs nothing after the first set watch client id — the single re-listen() from onError never comes back once PD is down.

So the failure described in #3152 reproduces on master with a plain 30 s PD restart, and this branch recovers every metadata watch after it, twice in a row.

Two small remarks, neither blocking:

• The retry path logs a WARN with a full stack trace for every attempt: with the 4 graph-event subscriptions plus one schema-cache-clear subscription per open graph, server B wrote ~210 Failed to reconnect watch for key … warnings (each with a trace) per 30 s outage. A one-line WARN, or the trace only on the first failure of a subscription, would keep this readable during a longer PD outage.
• Unrelated to this PR but hit while setting the test up: with usePD=true the server does not start on a JVM whose default locale uses a decimal comma (GraphSpace.info() does Float.parseFloat(String.format("%.2f", …)), pl_PL → "0,00" → NumberFormatException). I'll file it separately.

Scripts and full logs (both sides, both variants): https://github.com/SebastianGruza/hugegraph-oracle-suite — cluster/pd_watch_exp2.sh, results/pd_watch_3152_*.log. Happy to re-run on the next head — ping me.

[imbajin]

Pushed eeb7041 to this branch to fix the two confirmed P1 issues: blocking KV/lock discovery keeps its configured timeout, and SchemaDriver is unpublished before resource cleanup while concurrent init remains rejected. Both regression checks failed before the fix and pass now.

Five independent review lanes, including an adversarial review, found no remaining P0/P1 blocker after the fixes; the final incremental diff was independently checked. Validation: KvClientTest 41/41, SchemaDriverTest 5/5, editorconfig:format, full clean compile and diff check passed. Review score: 8.5/10; merge readiness still depends on CI for this new head.

The legacy four-argument streamingCall override concern in discussion_r3901065273 remains open as a compatibility follow-up. There is no production override in this repository; replacing both overloads with a private delegate would bypass the existing override dispatch, so I left that contract change for discussion. This PR restores watch reconnection; missed-event reconciliation remains tracked separately in #3151.

[bitflicker64]

Blocking: no. Summary: the AbstractClient transport/lifecycle split and the per-subscription watch model are the right fix for #3152, CI is green at this head, and the earlier P0/P1 reports are addressed; three regressions introduced by the hardening itself remain open (two in the new SchemaDriver lifecycle, one in AbstractClient locking) plus one piece of dead protected API. Evidence: full exact-head diff at eeb7041 compared against origin/master baselines of AbstractClient, KvClient and SchemaDriver; call-site reads of SchemaGraph.schemaDriverInit(), PdMetaDriver, AbstractClientStubProxy and PDException; gh pr checks 3157 reports 24/24 green.

[imbajin]

+1. Score: 9/10 for the agreed watch-recovery scope. Five independent review lanes, including an adversarial review, and final whole-diff verification are complete. KvClientTest (41) and SchemaDriverTest (5), formatting, full clean compile and all 22 current-head CI checks pass. No remaining blockers within this scope.

Accepted boundaries: shared-transport recovery may briefly delay ordinary KV/lock requests; callers coordinate SchemaDriver initialization and destruction; existing streaming override dispatch is preserved. The remaining threads are resolved on these explicit scope decisions, not on a claim that the tradeoffs were eliminated.

Next: #3151 for missed-event graph metadata reconciliation and discussion #3205 for schema cache reconciliation. This PR restores future notifications for #3152; it does not replay missed events or resolve those convergence workstreams.