HDDS-15715. Add helpful CLI flags to finalization CLIs by dombizita · Pull Request #10678 · apache/ozone

dombizita · 2026-07-06T19:27:28Z

What changes were proposed in this pull request?

Added new CLI flags to the upgrade finalization CLI:

finalize --wait: the client will periodically poll the cluster for status from OM until the whole cluster is finalized. The finalize command is idempotent so waiting can be resumed at any time
status --json: json version of the status output for scripting
status --verbose: includes internal information like OM and SCM apparent versions, the min and max DN apparent versions in the cluster

Used Claude Opus 4.8 for planning and coding, reviewed by me.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15715

How was this patch tested?

Added unit tests, green run on my fork: https://github.com/dombizita/ozone/actions/runs/28660447709

errose28

Thanks for adding this @dombizita.

errose28 · 2026-07-14T14:51:43Z

Thanks for the updates. Can you check on the CI failures? This should re-run against the base branch as well since we just merged from master.

errose28

Thanks for the updates. Left some more comments.

Also here is an AI generated test we can add to cover the case of RPC failure during the wait loop. This assumes the desired behavior in this case is to exit the loop though, which I'm not sure is the case. It might be better to adjust the wait loop to print to stderr when this happens and retry automatically on the next interval, and update this test accordingly.

diff --git a/hadoop-ozone/cli-admin/src/test/java/org/apache/hadoop/ozone/admin/upgrade/TestFinalizeSubCommand.java b/hadoop-ozone/cli-admin/src/test/java/org/apache/hadoop/ozone/admin/upgrade/TestFinalizeSubCommand.java
index e6e3a5ed48..70bf533b11 100644
--- a/hadoop-ozone/cli-admin/src/test/java/org/apache/hadoop/ozone/admin/upgrade/TestFinalizeSubCommand.java
+++ b/hadoop-ozone/cli-admin/src/test/java/org/apache/hadoop/ozone/admin/upgrade/TestFinalizeSubCommand.java
@@ -169,6 +169,18 @@ public void testWaitFlagPollsUntilFinalized() throws Exception {
     verify(omClient, times(3)).queryUpgradeStatus();
   }
 
+  @Test
+  public void testWaitFlagRpcFailurePrintsMessageAndReturnsNonZero() throws Exception {
+    when(omClient.queryUpgradeStatus()).thenThrow(new IOException("RPC timeout"));
+
+    new CommandLine(cmd).parseArgs("--wait");
+    assertEquals(1, cmd.call());
+
+    String errOutput = errContent.toString(DEFAULT_ENCODING);
+    assertTrue(errOutput.contains("Failed to query upgrade status"));
+    assertTrue(errOutput.contains("RPC timeout"));
+  }
+
   @Test
   public void testWaitFlagInterruptIsHandledCleanly() throws Exception {
     // Make the poll interval long enough that the interrupt lands during the sleep (after the first poll).

errose28 · 2026-07-17T20:20:34Z

+      try {
+        result.set(cmd.call());
+      } catch (Exception ignored) {
+        // The command handles interruption internally and should not throw.


If no exception is expected we should fail the test if one is encountered.

errose28 · 2026-07-17T20:37:26Z

      out().println("Cluster finalization has been started. Monitor progress with `ozone admin upgrade status`");
+
+      if (wait) {


We should probably alter this message if --wait is provided to indicate that we are waiting for finalization to complete.

errose28 · 2026-07-17T20:39:08Z

 */
 @CommandLine.Command(
    name = "finalize",
    description = "Initiates the the process to finalize a cluster upgrade.",


Existing issue, but we can fix it here.

Suggested change

description = "Initiates the process to finalize a cluster upgrade. This command is idempotent.",

errose28 · 2026-07-17T20:41:01Z

+        out().printf("Waiting for finalization: OM=%s, SCM=%s, datanodes=%d/%d%n",
+            status.getOmFinalized(), hdds.getScmFinalized(),
+            hdds.getNumDatanodesFinalized(), hdds.getNumDatanodesTotal());


Shouldn't we use the regular status output here instead of making a new one?

errose28 · 2026-07-17T20:48:47Z

+  static boolean isClusterFinalized(QueryUpgradeStatusResponse status) {
+    HddsProtos.UpgradeStatus hdds = status.getHddsStatus();
+    return status.getOmFinalized()
+        && hdds.getScmFinalized()
+        && hdds.getNumDatanodesFinalized() == hdds.getNumDatanodesTotal();
+  }


Looking at this again, I think we should add a new field to the proto to represent this, with this computation happening on the server side, i.e. OM returns clusterFinalized = hdds.isFinalized() && omFinalized and the client just consumes clusterFinalized to make its decision. It can still print the counts, but this allows us to change the definition of finalized on the server side without impacting old clients.

This is similar to what we did with the shouldFinalize field that OM gets from SCM instead of checking the SCM and datanode counts directly.

errose28 · 2026-07-17T21:01:20Z

+  }
+
+  @Test
+  public void testJsonFlagWithVerboseIncludesVersions() throws Exception {


The names of this and the previous test make it seem like the verbose flag affects json output, which is doesn't. If this is intentional, then there should be one test for all json output and it should also assert that content is identical when verbose is passed.

errose28 · 2026-07-17T21:06:39Z

+        throw new SCMException("Cannot query upgrade status while SCM is in safe mode. Wait until SCM exits "
+            + "safe mode and try again.", ResultCodes.SAFE_MODE_EXCEPTION);


We should update OM in the PR as well to handle the SCM exception with this result code and return a propagate a coherent OMException back to the user, probably with result code UNSUPPORTED_OPERATION like #10777 does when the SCM versions don't match.

dombizita added 3 commits July 3, 2026 12:09

HDDS-15715. Add helpful CLI flags to finalization CLIs

1e897ff

Fix checkstyle

aa874a8

Fix pmd

a90ae08

github-actions Bot added the zdu Pull requests for Zero Downtime Upgrade (ZDU) https://issues.apache.org/jira/browse/HDDS-14496 label Jul 6, 2026

errose28 self-requested a review July 7, 2026 18:42

errose28 reviewed Jul 8, 2026

View reviewed changes

dombizita added 2 commits July 14, 2026 14:05

Address review comments

2468e4a

Fix checkstyle

a681033

dombizita added 3 commits July 15, 2026 15:53

Adjust upgrade status test when SCM is in safemode

2207f8d

Adjust test with already finalized status

01d7035

Merge remote-tracking branch 'apache/HDDS-14496-zdu' into HDDS-15715

cee0669

errose28 reviewed Jul 17, 2026

View reviewed changes

		out().println("Cluster finalization has been started. Monitor progress with `ozone admin upgrade status`");

		if (wait) {


	description = "Initiates the process to finalize a cluster upgrade. This command is idempotent.",

		throw new SCMException("Cannot query upgrade status while SCM is in safe mode. Wait until SCM exits "
		+ "safe mode and try again.", ResultCodes.SAFE_MODE_EXCEPTION);

Uh oh!

Conversation

dombizita commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

errose28 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

errose28 commented Jul 14, 2026

Uh oh!

errose28 left a comment

Choose a reason for hiding this comment

Uh oh!

errose28 Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

errose28 Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

errose28 Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

errose28 Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

errose28 Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

errose28 Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

errose28 Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dombizita commented Jul 6, 2026 •

edited

Loading