HDDS-3128. Add ozone debug commands for Kerberos by navinko · Pull Request #9868 · apache/ozone

navinko · 2026-03-05T13:20:34Z

What changes were proposed in this pull request?

HDDS-3128. Add support for kdiag and kerbname commands to ozone script

This PR introduces two new subcommands under ozone debug kerberos to assist with Kerberos related troubleshooting in secure Ozone clusters.
Added changes for verifying how Kerberos principals map to local Unix users and collecting kerberos diagnostic which is useful when debugging and troubleshooting in secure clusters.
Implemented two subcommand 'daignose' and 'translate-principal' under ozone debug kerberos and multiple probe ,each one doing special job of getting configs and doing validation.
Once the 'daignose' subcommand executed successfully it will present diagnostic summary with 'PASS', 'WARN' & 'FAIL'
translate-principal will show the local user mapped with principal for auth_to_local rules.
Injected fault and validated o/ps
- FAULT-1: Invalid KRB5_CONFIG
- FAULT-2: Revoked execute permission
- FAULT-3: krb5.conf with empty contents
- FAULT-4: Updated /etc with permission

1: ozone debug kerberos translate-principal .. one or many .
This command translates Kerberos principals to local user names using the configured hadoop.security.auth_to_local rules.
Features:

Displays the active auth_to_local rule
Supports multiple principals in a single invocation
Provides clear error messages for malformed principals
Helps diagnose identity mapping issues

2: ozone debug kerberos diagnose
This command provides Kerberos diagnostics for Ozone clusters similar to hadoop kdiag.
Features:

It validates and shows below diagnostics.
Host Information
Environment Variables
JVM Kerberos Properties
Kerberos Configuration
Kerberos kinit Command
Kerberos Ticket presence
Principal Mapping
Keytab Validation
Security Configuration
Authorization Configuration
HTTP Kerberos Authentication
== Diagnostic Summary ==
PASS : pass count
WARN : warn count
FAIL : failure count

Fixed spotbugs + PMD issues and removed leftover configs.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-3128

How was this patch tested?

Added test class (TestOzoneDebugKerberos)- for ozone debug kerberos translate-principal and ozone debug kerberos diagnose , validated all probe execution with success and failure cases.
Tested both the functionality locally in (secure +insecure) cluster
Attached diagnostic and o/p captured in- [ozone-debug-kerberos-features]
Latest CI build - https://github.com/navinko/ozone/actions/runs/24403928422

adoroszlai

Thanks @navinko for working on this. I think we should add these new commands under ozone debug, not at the top-level. Also, please implement as a picocli @Command. See VersionDebug as an example. KDiag would need to be copied to Ozone and adapted.

navinko · 2026-03-05T16:26:27Z

I think we should add these new commands under ozone debug, not at the top-level. Also, please implement as a picocli @Command. See VersionDebug as an example. KDiag would need to be copied to Ozone and adapted.

Hi @adoroszlai Thank you for reviewing . I have one doubt

KDiag would need to be copied to Ozone and adapted.
Does it mean keeping the wrapper and reuse Hadoop KDiag and append Ozone diagnostics or copy the KDiag implementation into Ozone and extend it further ?

adoroszlai · 2026-03-05T19:52:47Z

KDiag would need to be copied to Ozone and adapted.

Does it mean keeping the wrapper and reuse Hadoop KDiag and append Ozone diagnostics or copy the KDiag implementation into Ozone and extend it further ?

The latter (copy to Ozone, change to use picocli instead of its own argument parsing, replace Hadoop-specific variables/properties with Ozone-specific ones, etc.).

navinko · 2026-03-06T14:57:26Z

KDiag would need to be copied to Ozone and adapted.

Does it mean keeping the wrapper and reuse Hadoop KDiag and append Ozone diagnostics or copy the KDiag implementation into Ozone and extend it further ?

The latter (copy to Ozone, change to use picocli instead of its own argument parsing, replace Hadoop-specific variables/properties with Ozone-specific ones, etc.).

Thank you , I will update the PR shortly.

navinko · 2026-03-14T17:09:59Z

Hi @adoroszlai
The review suggestions incorporated. Kindly review the PR once you get some time.

adoroszlai

Thanks @navinko for updating the patch. I'll need some time for the review.

adoroszlai

Thanks @navinko for updating the patch, and sorry for the delay in review.

In addition to inline comments, I have the following suggestions:

create a grouping subcommand ozone debug kerberos (class: KerberosSubcommands)
move the two new subcommands under ozone debug kerberos as
- kerbname => ozone debug kerberos translate-principal
- kdiag => ozone debug kerberos diagnose
move all classes to a single package org.apache.hadoop.ozone.debug.kerberos (no need for separate authtolocal and kdiag packages)

adoroszlai · 2026-03-24T17:25:18Z

+    print(conf, "ozone.acl.enabled");
+    print(conf, "ozone.acl.authorizer.class");
+    print(conf, "hadoop.security.authorization");
+    print(conf, "ozone.om.security.client.protocol.acl");
+
+    print(conf, "hdds.security.client.datanode.container.protocol.acl");
+    print(conf, "hdds.security.client.scm.container.protocol.acl");
+    print(conf, "hdds.security.client.scm.block.protocol.acl");
+    print(conf, "hdds.security.client.scm.certificate.protocol.acl");


Please use constants from HddsConfigKeys, OzoneConfigKeys etc. where available.

there were few fields not available and i tried avoiding cyclic dependency , used as it is string .

adoroszlai · 2026-03-24T17:31:10Z

+    String value = conf.get(key);
+    System.out.println(key + " = "
+        + (value == null ? "(unset)" : value));


Please reuse code across various probes. The same logic also appears in HttpAuthProbe, OzonePrincipalProbe and SecurityConfigProbe. Maybe a parent class ConfigProbe?

Thanks @adoroszlai tried incorporation suggestions.

adoroszlai · 2026-03-24T17:32:03Z

+
+  String name();
+
+  boolean run() throws Exception;


I think test() would be a better name.

adoroszlai · 2026-03-24T17:34:10Z

+      String realm = KerberosUtil.getDefaultRealm();
+      System.out.println("Default realm = " + realm);
+    } catch (Exception e) {
+      System.out.println("WARNING: Unable to determine default realm");


I think warnings and errors should be printed to System.err.

used error stream , was using out only ealier .

adoroszlai · 2026-03-24T17:40:53Z

+  @Override
+  public Void call() throws Exception {
+    System.out.println("-- Kerberos Principal Translation --");
+    OzoneConfiguration conf = new OzoneConfiguration();


Let KerbNameDebug extends AbstractSubcommand, then configuration can be obtained via getOzoneConf(). It allows setting specific properties, as well as alternative config files.

The same applies to OzoneKDiag.

Thanks @adoroszlai tried incorporation suggestions.

adoroszlai · 2026-03-24T17:52:44Z

+
+  @Override
+  public boolean run() {
+    System.out.println("-- Authorization Configuration --");


All DiagnosticProbe implementations have name() that could be automatically printed instead of duplicating the title in run().

renamed to test.

adoroszlai · 2026-03-24T18:51:53Z

+    System.out.println("PASS : " + pass);
+    System.out.println("WARN : " + warn);
+    System.out.println("FAIL : " + fail);
+    return null;


We may want to set non-zero exit code if there are any failures. I think it can be done by using Callable<Integer> instead of Callable<Void>.

Thanks @adoroszlai tried incorporation suggestions.

adoroszlai · 2026-03-24T19:01:00Z

+          warn++;
+          System.out.println("[WARN] " + probe.name());


I tried introducing problems, e.g. by making kinit non-executable, removing permissions from /etc/krb5.conf, etc. The probes returned false, which resulted only in warnings. But the system is non-functional with this setup. Shouldn't it give errors instead of warnings?

-- Kerberos kinit Command -- Executable kinit must be available on PATH PATH = /opt/hadoop/libexec:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/jdk-21.0.2/bin:/opt/hadoop/bin kinit not found on PATH [WARN] Kerberos kinit Command -- Kerberos Ticket -- ERROR checking kerberos credentials: Can't get Kerberos realm [WARN] Kerberos Ticket

adoroszlai

Thanks @navinko for updating the patch.

adoroszlai · 2026-04-06T18:40:23Z

+@MetaInfServices(DebugSubcommand.class)
+public class TranslatePrincipalSubcommand extends AbstractSubcommand
+    implements Callable<Void>, DebugSubcommand {


Please remove leftover references to DebugSubcommand (now translate-principal is under KerberosSubcommand).

I am really sorry ,my bad:)
I am still working on the pr din't realised it is not draft anymore.

Will update the PR shortly.

navinko · 2026-04-07T15:00:40Z

Thanks @navinko for updating the patch, and sorry for the delay in review.

In addition to inline comments, I have the following suggestions:

create a grouping subcommand ozone debug kerberos (class: KerberosSubcommands)

move the two new subcommands under ozone debug kerberos as

kerbname => ozone debug kerberos translate-principal

kdiag => ozone debug kerberos diagnose

move all classes to a single package org.apache.hadoop.ozone.debug.kerberos (no need for separate authtolocal and kdiag packages)

Thanks @adoroszlai tried incorporation suggestions.

navinko · 2026-04-07T15:08:08Z

Hi @adoroszlai thank you for reviewing and suggestions .
The changes has been updated and tried incorporating . Please do review once you get some time.

adoroszlai

Thanks a lot @navinko for updating the patch, LGTM. Let's give others some time to take a look.

patch updated

adoroszlai · 2026-04-07T15:15:22Z

@Gargi-jais11 would you like to take a look?

Gargi-jais11 · 2026-04-07T17:30:28Z

@Gargi-jais11 would you like to take a look?

Sure @adoroszlai will take a look into this.

Gargi-jais11

Thanks @navinko for working on the patch. This will really be helpful change.
Please find inline comments.

Gargi-jais11 · 2026-04-08T04:23:47Z

+      BufferedWriter writer = new BufferedWriter(
+          new OutputStreamWriter(buffer, StandardCharsets.UTF_8));


A BufferedWriter is created that wraps the ByteArrayOutputStream buffer. writer is created and closed in finally — that's it. The stream that actually captures probe output is PrintStream ps. Only ps is ever used.
BufferWriter is a dead code and can be remove Line 67-68 and Line 89.

My bad , din't realise while fixing spotbugs issues. Fixed now.

Gargi-jais11 · 2026-04-08T04:28:20Z

+      } catch (Exception e) {
+        System.err.println("ERROR: Failed to translate principal "
+            + principal + " : " + e.getMessage());
+      }
+    }
+    return null;


This subcommand impliments Callable<Void> and always returns null (exit code 0), even when a principal fails t o translates.
It should be Callable<Integer> and return a non-zero code if any principal translation fails — consistent with how DiagnoseSubcommand which returns 1 on failures.

Thanks @Gargi-jais11 I just fixed it. I initially thought to keep it simpler since we are not doing many things here but keeping it consistent with diagnose design overall will be great .

Gargi-jais11 · 2026-04-08T04:40:59Z

+      if (output.contains("ERROR:")) {
+        fail++;
+        out().println("[FAIL] " + probe.name());
+      } else if (!valid) {
+        warn++;
+        out().println("[WARN] " + probe.name());
+      } else {
+        pass++;
+        out().println("[PASS] " + probe.name());
+      }
+      out().println();
+    }


Right now FAIL is infferred with output.contains("ERROR:") while WARN/PASS use probe.test()’s boolean return. Those two signals can disagree (e.g. a probe prints ERROR: but still returns success, or the opposite).
It is better to have one source of truth and drop the string scan.

Right Approach would be:
step1: create them as an enum

public enum ProbeResult { PASS, // probe ran cleanly, everything looks healthy WARN, // probe ran, something is misconfigured but not broken FAIL // probe detected a critical problem }

Step2: Update the DiagnosticProbe interface to:

ProbeResult test(OzoneConfiguration conf) throws Exception; // was: boolean

step3: then you can Update DiagnoseSubcommand — remove the string scan entirely and use the Enum created
step4: Update probes to return ProbeResult explicitly
Here are examples of all three cases:

Purely informational probe (was always return true) — EnvironmentProbe
WARN probe (suboptimal but not broken) — AuthorizationProbe
FAIL probe (critical, system broken) — KerberosConfigProbe
A probe with both WARN and FAIL paths — KerberosTicketProbe

Used enum and updated the code.

Gargi-jais11 · 2026-04-08T04:44:27Z

+
+    try {
+      UserGroupInformation.setConfiguration(conf);
+      String authType = conf.getTrimmed("hadoop.security.authentication");


Please use constant here: HADOOP_SECURITY_AUTHENTICATION

Gargi-jais11 · 2026-04-08T05:14:33Z

+ *
+ * The probe does NOT attempt to perform a login (kinit), Instead it reports
+ * the current state so operators can diagnose security issues.
+ * Checks the current Kerberos authentication state of the process.


Nit: Duplicate this can be removed.

Gargi-jais11 · 2026-04-08T05:53:29Z

+    // Force Kerberos failure
+    System.setProperty("java.security.krb5.conf", "/invalid/path");


I think you need to update the comment as java.security.krb5.conf is for jvm kerberos probe to fail and accordingly assertion should check for -- JVM Kerberos Properties -- not Kerberos configuration.
DiagnoseSubcommand prints a banner for every probe:
So "-- Kerberos Configuration --" is always printed when that probe runs, whether it PASS, WARN, or FAIL.
Asserting that string only proves “this section appeared,” not that KerberosConfigProbe was the thing that failed.
So: the test works for “something Kerberos-related failed,” but the intent and assertions don’t match what really broke.

Updated the comment , although the failure case i was testing for KerberosConfigProbe ,thats is why i was checking "-- Kerberos Configuration --
Now updated it to -- JVM Kerberos Properties --

That's fine you could have also done it either way.

Gargi-jais11 · 2026-04-08T06:21:17Z

+    boolean valid = true;
+
+    // Print relevant configs
+    print(conf, OzoneConfigKeys.OZONE_ACL_ENABLED);


We should also add "ozone.authorization.enabled" in the authorisation probe,since this acts as a master switch to enable/disable authorization checks in Ozone(admin privilege checks and ACL checks).

Suggested change

print(conf, OzoneConfigKeys.OZONE_ACL_ENABLED);

print(conf, OzoneConfigKeys.OZONE_AUTHORIZATION_ENABLED);

print(conf, OzoneConfigKeys.OZONE_ACL_ENABLED);

thanks it make sense , added.

navinko · 2026-04-11T18:12:41Z

Thanks @Gargi-jais11 for the review, tried addressing review suggestions.
Once you get sometime please review .

Gargi-jais11

Thanks @navinko for updating the patch. Just few more minor comments.

Gargi-jais11 · 2026-04-13T04:08:14Z

+      warn("Ozone security is disabled (ozone.security.enabled=false). "
+          + "Authorization checks are not enforced in non-secure mode.");


Instead of hardcoding please use the constants for key and value:

"Ozone security is disabled (" + OZONE_SECURITY_ENABLED_KEY + "=" + OZONE_SECURITY_ENABLED_KEY_DEFAULT + ". Authorization checks are not enforced in non-secure mode."

Gargi-jais11 · 2026-04-13T04:09:08Z

+        OzoneConfigKeys.OZONE_AUTHORIZATION_ENABLED, true);
+
+    if (!ozoneAuthEnabled) {
+      warn("Ozone authorization is disabled (ozone.authorization.enabled=false). "


Similarly use the constants for ozone.authorization.enabled and its default value.

Thanks @Gargi-jais11
Updated the changes .

Gargi-jais11

Thanks @navinko for updating the patch. LGTM!
Please document this tool as well. Is there any tracking jira for documenting this new tool added?

navinko · 2026-04-14T04:48:50Z

Thanks @navinko for updating the patch. LGTM! Please document this tool as well. Is there any tracking jira for documenting this new tool added?

Thanks @Gargi-jais11
I have created https://issues.apache.org/jira/browse/HDDS-14991

adoroszlai · 2026-04-14T13:05:04Z

+
+    // Read rule
+    String rules = conf.getTrimmed(
+        "hadoop.security.auth_to_local", "DEFAULT");


Can use CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTH_TO_LOCAL.

adoroszlai · 2026-04-14T13:05:11Z

+  public static final String HDDS_SCM_HTTP_AUTH_TYPE = "hdds.scm.http.auth.type";
+


Just noticed that this is a duplicate of:

ozone/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMHTTPServerConfig.java

Lines 79 to 80 in 3daabc8

public static final String HDDS_SCM_HTTP_AUTH_TYPE =

HDDS_SCM_HTTP_AUTH_CONFIG_PREFIX + "type";

My bad , removed and reused exiting.

adoroszlai · 2026-04-14T13:06:53Z

+
+    // Validate Ozone ACLs enforcement
+    boolean ozoneAclEnabled = conf.getBoolean(
+        OzoneConfigKeys.OZONE_ACL_ENABLED, false);


Suggested change

OzoneConfigKeys.OZONE_ACL_ENABLED, false);

OzoneConfigKeys.OZONE_ACL_ENABLED, OzoneConfigKeys.OZONE_ACL_ENABLED_DEFAULT);

Thanks @adoroszlai ,
I have just updated the changes.

adoroszlai

Thanks @navinko for the reminder that the patch is updated.

adoroszlai · 2026-04-16T16:08:52Z

Thanks @navinko for the patch, @Gargi-jais11 for the review.

navinko · 2026-04-16T16:29:33Z

Thanks @navinko for the patch, @Gargi-jais11 for the review.

Thank you so much @adoroszlai , @Gargi-jais11 for your continued support through the review process and for getting this PR merged.

navinko added 2 commits March 5, 2026 03:07

HDDS-3128. Add support for kdiag and kerbname commands to ozone script

3860ba5

HDDS-3128. Add support for kdiag and kerbname commands to ozone script

4772c11

adoroszlai previously requested changes Mar 5, 2026

View reviewed changes

navinko marked this pull request as draft March 6, 2026 14:57

navinko added 8 commits March 6, 2026 22:06

Merge branch 'apache:master' into HDDS-3128

2fdccdd

HDDS-3128. Add support for kdiag and kerbname commands to ozone script

584b3ae

HDDS-3128. Removed unused dependencies

3f6c128

HDDS-3128. Removed unused dependencies

fbb15b8

HDDS-3128. Fixed test case and checkstyle issues

1be65a5

HDDS-3128. Formatted command output.

73a8dac

HDDS-3128. Formatted command output.

ffbc98c

Merge branch 'master' into HDDS-3128

084bbf4

navinko marked this pull request as ready for review March 12, 2026 19:05

navinko requested a review from adoroszlai March 12, 2026 19:07

adoroszlai reviewed Mar 15, 2026

View reviewed changes

adoroszlai reviewed Mar 24, 2026

View reviewed changes

HDDS-3128. Add support for ozone debug kerberos subcommands

7ee1efa

adoroszlai reviewed Apr 6, 2026

View reviewed changes

navinko marked this pull request as draft April 6, 2026 19:17

navinko added 3 commits April 7, 2026 02:06

HDDS-3128. Fixed spotbugs and pmd issues

86b69ba

HDDS-3128. Removed leftover references

1603ebb

Merge branch 'apache:master' into HDDS-3128

58c6b2e

navinko marked this pull request as ready for review April 7, 2026 15:08

adoroszlai reviewed Apr 7, 2026

View reviewed changes

adoroszlai requested a review from ChenSammi April 7, 2026 15:14

Gargi-jais11 reviewed Apr 8, 2026

View reviewed changes

navinko added 3 commits April 10, 2026 00:29

Merge branch 'apache:master' into HDDS-3128

cffaa00

HDDS-3128. Improvements and addressed review comments

a4224d6

HDDS-3128. Checkstyle fix

220cae6

navinko requested a review from Gargi-jais11 April 11, 2026 18:12

Gargi-jais11 reviewed Apr 13, 2026

View reviewed changes

HDDS-3128. Addressed review comments

1f12919

navinko requested a review from Gargi-jais11 April 13, 2026 14:14

adoroszlai changed the title ~~HDDS-3128. Add support for kdiag and kerbname commands to ozone script~~ HDDS-3128. Add ozone debug commands for Kerberos Apr 13, 2026

Gargi-jais11 approved these changes Apr 14, 2026

View reviewed changes

adoroszlai reviewed Apr 14, 2026

View reviewed changes

navinko added 2 commits April 14, 2026 19:12

HDDS-3128. Addressed review comments

2db4eeb

HDDS-3128. Checkstyle issue fix

3d49dc1

navinko requested a review from adoroszlai April 16, 2026 14:10

adoroszlai approved these changes Apr 16, 2026

View reviewed changes

adoroszlai merged commit 191c01c into apache:master Apr 16, 2026
46 checks passed

		BufferedWriter writer = new BufferedWriter(
		new OutputStreamWriter(buffer, StandardCharsets.UTF_8));

		// Force Kerberos failure
		System.setProperty("java.security.krb5.conf", "/invalid/path");

	print(conf, OzoneConfigKeys.OZONE_ACL_ENABLED);
	print(conf, OzoneConfigKeys.OZONE_AUTHORIZATION_ENABLED);
	print(conf, OzoneConfigKeys.OZONE_ACL_ENABLED);

		warn("Ozone security is disabled (ozone.security.enabled=false). "
		+ "Authorization checks are not enforced in non-secure mode.");

		public static final String HDDS_SCM_HTTP_AUTH_TYPE = "hdds.scm.http.auth.type";

	public static final String HDDS_SCM_HTTP_AUTH_TYPE =
	HDDS_SCM_HTTP_AUTH_CONFIG_PREFIX + "type";

	OzoneConfigKeys.OZONE_ACL_ENABLED, false);
	OzoneConfigKeys.OZONE_ACL_ENABLED, OzoneConfigKeys.OZONE_ACL_ENABLED_DEFAULT);

Conversation

navinko commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

navinko commented Mar 5, 2026

Uh oh!

adoroszlai commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

navinko commented Mar 6, 2026

Uh oh!

navinko commented Mar 14, 2026

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

navinko commented Apr 7, 2026

Uh oh!

navinko commented Apr 7, 2026

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

adoroszlai commented Apr 7, 2026

Uh oh!

Gargi-jais11 commented Apr 7, 2026

Uh oh!

Gargi-jais11 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

navinko commented Mar 5, 2026 •

edited

Loading

adoroszlai commented Mar 5, 2026 •

edited

Loading

navinko Apr 10, 2026 •

edited

Loading

Gargi-jais11 Apr 8, 2026 •

edited

Loading