Skip to content

PHOENIX-7566 HAGroupStore state machine: ANISTS non-blocking, DS->STANDBY_TO_ACTIVE #2518

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ritegarg wants to merge 1 commit into
base: PHOENIX-7562-feature-new
Choose a base branch
from
+19 −6
Open

PHOENIX-7566 HAGroupStore state machine: ANISTS non-blocking, DS->STANDBY_TO_ACTIVE #2518

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions phoenix-core-client/src/main/java/org/apache/phoenix/jdbc/HAGroupStoreRecord.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,11 +76,11 @@ public ClusterRoleRecord.ClusterRole getClusterRole() {
case ABORT_TO_ACTIVE_NOT_IN_SYNC:
case ACTIVE_IN_SYNC:
case ACTIVE_NOT_IN_SYNC:
case ACTIVE_NOT_IN_SYNC_TO_STANDBY:

@apurtell apurtell Jun 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Reduces unavailability during failover. The safety argument shifts a bit, but the transition tables do not allow the peer to promote, so we still have only a single writer in all states.

case ACTIVE_NOT_IN_SYNC_WITH_OFFLINE_PEER:
case ACTIVE_WITH_OFFLINE_PEER:
return ClusterRoleRecord.ClusterRole.ACTIVE;
case ACTIVE_IN_SYNC_TO_STANDBY:
case ACTIVE_NOT_IN_SYNC_TO_STANDBY:
return ClusterRoleRecord.ClusterRole.ACTIVE_TO_STANDBY;
case ABORT_TO_STANDBY:
case DEGRADED_STANDBY:
Expand Down Expand Up @@ -114,7 +114,7 @@ public ClusterRoleRecord.ClusterRole getClusterRole() {
ACTIVE_IN_SYNC_TO_STANDBY.allowedTransitions =
ImmutableSet.of(ABORT_TO_ACTIVE_IN_SYNC, STANDBY);
STANDBY_TO_ACTIVE.allowedTransitions = ImmutableSet.of(ABORT_TO_STANDBY, ACTIVE_IN_SYNC);
DEGRADED_STANDBY.allowedTransitions = ImmutableSet.of(STANDBY);
DEGRADED_STANDBY.allowedTransitions = ImmutableSet.of(STANDBY, STANDBY_TO_ACTIVE);

@apurtell apurtell Jun 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Fixes a DS side deadlock at peer AISTS.

ACTIVE_WITH_OFFLINE_PEER.allowedTransitions = ImmutableSet.of(ACTIVE_NOT_IN_SYNC);
ABORT_TO_ACTIVE_IN_SYNC.allowedTransitions = ImmutableSet.of(ACTIVE_IN_SYNC);
ABORT_TO_ACTIVE_NOT_IN_SYNC.allowedTransitions = ImmutableSet.of(ACTIVE_NOT_IN_SYNC);
Expand Down
7 changes: 4 additions & 3 deletions phoenix-core/src/it/java/org/apache/phoenix/jdbc/HAGroupStoreManagerIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,16 +145,17 @@ public void testMutationBlockingWithMultipleHAGroups() throws Exception {
assertFalse(haGroupStoreManager.isMutationBlocked(haGroupName1));
assertFalse(haGroupStoreManager.isMutationBlocked(haGroupName2));

// Update only second group to ACTIVE_NOT_IN_SYNC_TO_STANDBY
// Move only the second group into a mutation-blocking drain state
// (ACTIVE_IN_SYNC_TO_STANDBY maps to ACTIVE_TO_STANDBY, which blocks mutations).
HAGroupStoreRecord transitionRecord2 = new HAGroupStoreRecord("1.0", haGroupName2,
HAGroupStoreRecord.HAGroupState.ACTIVE_NOT_IN_SYNC_TO_STANDBY, 0L,
HAGroupStoreRecord.HAGroupState.ACTIVE_IN_SYNC_TO_STANDBY, 0L,
HighAvailabilityPolicy.FAILOVER.toString(), this.peerZKUrl, CLUSTERS.getMasterAddress1(),
CLUSTERS.getMasterAddress2(), CLUSTERS.getHdfsUrl1(), CLUSTERS.getHdfsUrl2(), 0L);

haAdmin.updateHAGroupStoreRecordInZooKeeper(haGroupName2, transitionRecord2, 0);
Thread.sleep(ZK_CURATOR_EVENT_PROPAGATION_TIMEOUT_MS);

// Global mutations should be blocked due to second group
// Blocking is per-group: only the second group is blocked; the first stays writable.
assertFalse(haGroupStoreManager.isMutationBlocked(haGroupName1));
assertTrue(haGroupStoreManager.isMutationBlocked(haGroupName2));
}
Expand Down
14 changes: 13 additions & 1 deletion phoenix-core/src/test/java/org/apache/phoenix/jdbc/HAGroupStoreRecordTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -239,7 +239,7 @@ public void testHAGroupStateGetClusterRole() {
HAGroupStoreRecord.HAGroupState.ACTIVE_IN_SYNC.getClusterRole());
assertEquals(ClusterRoleRecord.ClusterRole.ACTIVE,
HAGroupStoreRecord.HAGroupState.ACTIVE_NOT_IN_SYNC.getClusterRole());
assertEquals(ClusterRoleRecord.ClusterRole.ACTIVE_TO_STANDBY,
assertEquals(ClusterRoleRecord.ClusterRole.ACTIVE,
HAGroupStoreRecord.HAGroupState.ACTIVE_NOT_IN_SYNC_TO_STANDBY.getClusterRole());
assertEquals(ClusterRoleRecord.ClusterRole.ACTIVE,
HAGroupStoreRecord.HAGroupState.ACTIVE_NOT_IN_SYNC_WITH_OFFLINE_PEER.getClusterRole());
Expand All @@ -257,6 +257,12 @@ public void testHAGroupStateGetClusterRole() {
HAGroupStoreRecord.HAGroupState.STANDBY_TO_ACTIVE.getClusterRole());
assertEquals(ClusterRoleRecord.ClusterRole.UNKNOWN,
HAGroupStoreRecord.HAGroupState.UNKNOWN.getClusterRole());

// ANISTS is non-blocking (maps to ACTIVE); only AISTS blocks mutations.
Comment thread
apurtell marked this conversation as resolved.
assertFalse(HAGroupStoreRecord.HAGroupState.ACTIVE_NOT_IN_SYNC_TO_STANDBY.getClusterRole()
.isMutationBlocked());
assertTrue(HAGroupStoreRecord.HAGroupState.ACTIVE_IN_SYNC_TO_STANDBY.getClusterRole()
.isMutationBlocked());
}

@Test
Expand Down Expand Up @@ -294,6 +300,12 @@ public void testHAGroupStateValidTransitions() {
.isTransitionAllowed(HAGroupStoreRecord.HAGroupState.ABORT_TO_STANDBY));
assertTrue(HAGroupStoreRecord.HAGroupState.STANDBY_TO_ACTIVE
.isTransitionAllowed(HAGroupStoreRecord.HAGroupState.ACTIVE_IN_SYNC));

// A degraded standby can recover to STANDBY or promote directly on failover.
assertTrue(HAGroupStoreRecord.HAGroupState.DEGRADED_STANDBY
.isTransitionAllowed(HAGroupStoreRecord.HAGroupState.STANDBY));
assertTrue(HAGroupStoreRecord.HAGroupState.DEGRADED_STANDBY
.isTransitionAllowed(HAGroupStoreRecord.HAGroupState.STANDBY_TO_ACTIVE));
}

@Test
Expand Down