fix(store): refresh gRPC channels after address changes - #3130
fix(store): refresh gRPC channels after address changes#3130bitflicker64 wants to merge 3 commits into
Conversation
dosubot
Bot
added
size:XL
218b681 to
26218cb
Compare
There was a problem hiding this comment.
Pull request overview
This PR updates the HStore gRPC client channel/stub lifecycle so that when a stable Store target (e.g., DNS name) resolves to a different address set, the client discards the prior channel pool and rebuilds related stub pools to avoid getting stuck on failed transports (issue #3124).
Changes:
- Add per-target “resolved address set” fingerprinting and retire/replace cached channel pools when the fingerprint changes (or when a previously-unresolved target first resolves).
- Rebuild blocking and async stub pools when they no longer correspond to the current channel pool, with concurrency ordering to prevent stale work from reintroducing retired channels.
- Add regression tests covering address changes and concurrent refresh/stub-build interleavings.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| hugegraph-store/hg-store-client/src/main/java/org/apache/hugegraph/store/client/grpc/AbstractGrpcClient.java | Adds resolved-address fingerprinting, refresh/retire logic for channel pools, and stub-pool rebuild safeguards under concurrent refresh. |
| hugegraph-store/hg-store-test/src/main/java/org/apache/hugegraph/store/client/grpc/AbstractGrpcClientTest.java | Adds refresh-focused tests validating channel replacement, stub-pool rebuild, and concurrency ordering behavior. |
| hugegraph-store/hg-store-test/src/main/java/org/apache/hugegraph/store/client/ClientSuiteTest.java | Introduces a small suite to run the refresh regression tests together. |
Comments suppressed due to low confidence (1)
hugegraph-store/hg-store-client/src/main/java/org/apache/hugegraph/store/client/grpc/AbstractGrpcClient.java:172
- When (re)building the async stub pool, the loop always selects
targetChannels[index]for every slot, so all cached async stubs share a single channel. This defeats the channel pool and undermines concurrency/failover across channels. Bind each stub to its corresponding channel (or at least distribute across the pool) by using the loop index.
IntStream.range(0, concurrency).parallel().forEach(i -> {
ManagedChannel channel = targetChannels[index];
AbstractAsyncStub stub = getAsyncStub(channel);
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
imbajin
left a comment
imbajin
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: Address refresh can abort active RPCs, adds synchronous DNS resolution to every request path, and does not preserve the existing gRPC target contract; the async refresh race also lacks equivalent coverage. Evidence: static review across six independent lanes; mvn test -pl hugegraph-store/hg-store-test -am -Dtest=AbstractGrpcClientTest -DfailIfNoTests=false passed 4 tests.
dosubot
Bot
added
size:XXL
Codecov Report✅ All modified and coverable lines are covered by tests.
Additional details and impacted files@@ Coverage Diff @@
## master #3130 +/- ##
============================================
- Coverage 39.19% 1.54% -37.65%
+ Complexity 264 21 -243
============================================
Files 770 748 -22
Lines 65779 63264 -2515
Branches 8726 8278 -448
============================================
- Hits 25779 975 -24804
- Misses 37247 62205 +24958
+ Partials 2753 84 -2669 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Refresh cached channel pools when a Store target resolves to a new address. Rebuild stale blocking and async stub pools, and guard concurrent resolution and publication races. Fixes apache#3124
- move graceful retirement to a cleanup scheduler - force-close partial pools after creation failures - validate cached stubs against channels by index - cover saturation, interruption, and drain deadlines
a08ac4b to
ddeef7a
Compare
imbajin
left a comment
imbajin
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: DNS refresh can fail under the default security policy, and QueryV2 can race channel retirement. Evidence: six independent exact-head lanes; 14/14 focused Java tests and git diff --check passed; all visible exact-head checks passed.
| return ""; | ||
| } | ||
| try { | ||
| return Arrays.stream(this.resolveHost(host)) |
There was a problem hiding this comment.
getChannels(), but InetAddress.getAllByName() invokes SecurityManager.checkConnect(host, -1). The default launcher enables HugeSecurityManager, whose checkConnect() rejects Gremlin worker stacks and has no HStore/gRPC exemption; only UnknownHostException is caught here. An HStore-backed Gremlin request can therefore throw SecurityException on initial resolution or a periodic refresh instead of using the healthy pool. Please move resolution to a trusted bounded background path (preserving the current pool on failure), or add the narrowest safe HStore policy exemption, and cover a real Gremlin-worker + HugeSecurityManager lookup regression.
| } | ||
|
|
||
| if (replaced) { | ||
| this.retireChannels(staleChannels); |
There was a problem hiding this comment.
tryLock() can still receive staleChannels; after this swap, retirement immediately calls shutdown() on that pool. QueryV2Client#getManagedChannel() uses getChannels() directly and creates its async stub only afterwards, bypassing the identity/retry loops added to getAsyncStub() and getBlockingStub(), so a refresh race can hand QueryV2 a channel that rejects the new RPC. Please route QueryV2 through the guarded async-stub path or provide an atomic channel/stub lease, and add an interleaving test for this production consumer.
Purpose of the PR
HStore caches gRPC channels and blocking/asynchronous stubs by Store target.
When a stable target resolves to a different address, those pools can retain the
failed transport indefinitely even though the replacement Store is reachable.
This is the channel-lifecycle half of #3124. Complete recovery behind a stable
DNS name still requires the finite positive DNS TTL from #3126; this patch does
not bypass an indefinitely stale JVM DNS cache.
Main Changes
channel.
performs a slow DNS check while other callers keep using the cached healthy
pool.
channels with
shutdown()first and use bounded delayedshutdownNow()onlyif they do not drain.
belong to the current pool, and spread cached stubs across the channel pool.
host:port,dns:///host:port, and bracketed IPv6 targets forfingerprinting; unsupported resolver schemes are left to gRPC and skipped by
the DNS fingerprint refresh path.
cluster.
Verifying these changes
JAVA_HOME=$(/usr/libexec/java_home -v 11) mvn test -pl hugegraph-store/hg-store-test -am -Dtest=AbstractGrpcClientTest -Dsurefire.failIfNoSpecifiedTests=false:AbstractGrpcClientTestran 9 tests with 0 failures, errors, or skips on Java 11.JAVA_HOME=$(/usr/libexec/java_home -v 11) mvn test -pl hugegraph-store/hg-store-test -am -P store-client-test -Dsurefire.failIfNoSpecifiedTests=false:ClientSuiteTestran 9 tests with 0 failures, errors, or skips on Java 11.JAVA_HOME=$(/usr/libexec/java_home -v 11) mvn editorconfig:format: passed with 0 files changed.JAVA_HOME=$(/usr/libexec/java_home -v 11) mvn clean compile -Dmaven.javadoc.skip=true: passed across all 38 reactor modules on Java 11.Does this PR potentially affect the following parts?
Documentation Status
Doc - TODODoc - DoneDoc - No Need