feat: kerberized Kraft controllers by razvan · Pull Request #1024 · stackabletech/kafka-operator

razvan · 2026-09-17T12:49:39Z

Description

Note

For the reviewer:

Each commit message has explanations of what it is trying to achieve.
The commit fix: dial controller pod FQDNs, and give the quorum-manager krb5.conf reverts a design decision from feat: add support for dynamic KRaft quorum scaling #1010. See commit comment for details.
The commit fix controller liveness probe is a fix that is independent of Kerberos support. I needed it here and didn't want the churn of a separate PR.
The commit fix: cleanup LLM verbosity removes the Claude implementation plan, unnecessary unit tests and updates docs and comments to be more human friendly.

Tests

✅ Local nightly suite:

--- PASS: kuttl (1035.18s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/cluster-operation_kafka-latest-3.9.2_zookeeper-latest-3.9.5_openshift-false (78.35s)
        --- PASS: kuttl/harness/logging_kafka-3.9.2_zookeeper-latest-3.9.5_openshift-false (82.27s)
        --- PASS: kuttl/harness/opa_kafka-latest-3.9.2_zookeeper-latest-3.9.5_opa-latest-1.16.2_use-opa-tls-false_openshift-false_krb5-1.21.1 (116.82s)
        --- PASS: kuttl/harness/tls_kafka-3.9.2_zookeeper-latest-3.9.5_use-client-tls-false_use-client-auth-tls-false_openshift-false (36.75s)
        --- PASS: kuttl/harness/upgrade_upgrade_old-3.9.2_upgrade_new-4.2.1_use-client-tls-false_use-client-auth-tls-false_openshift-false (49.66s)
        --- PASS: kuttl/harness/smoke-kraft_kafka-kraft-4.2.1_openshift-false (272.86s)
        --- PASS: kuttl/harness/smoke_kafka-3.9.2_zookeeper-3.9.5_use-client-tls-false_openshift-false (81.18s)
        --- PASS: kuttl/harness/kerberos_kafka-3.9.2_zookeeper-latest-3.9.5_openshift-false_krb5-1.21.1_kerberos-realm-PROD.MYCORP_kerberos-backend-mit_broker-listener-class-cluster-internal_bootstrap-listener-class-external-unstable (96.65s)
        --- PASS: kuttl/harness/configuration_kafka-latest-3.9.2_openshift-false (11.55s)
        --- PASS: kuttl/harness/operations-kraft_kafka-kraft-4.2.1_openshift-false (604.57s)
        --- PASS: kuttl/harness/delete-rolegroup_kafka-3.9.2_zookeeper-latest-3.9.5_openshift-false (30.33s)
        --- PASS: kuttl/harness/kraft-kerberos_kafka-kraft-4.2.1_openshift-false_krb5-1.21.1_kerberos-realm-PROD.MYCORP_kerberos-backend-mit_broker-listener-class-cluster-internal (494.58s)
PASS

Definition of Done Checklist

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
Please make sure all these things are done and tick the boxes

Author

Changes are OpenShift compatible
CRD changes approved
CRD documentation for all fields, following the style guide.
Helm chart can be installed and deployed operator works
Integration tests passed (for non trivial changes)
Changes need to be "offline" compatible
Links to generated (nightly) docs added
Release note snippet added

Reviewer

Code contains useful comments
Code contains useful logging statements
(Integration-)Test cases added
Documentation added or updated. Follows the style guide.
Changelog updated
Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

Feature Tracker has been updated
Proper release label has been added
Links to generated (nightly) docs added
Release note snippet added
Add type/deprecation label & add to the deprecation schedule
Add type/experimental label & add to the experimental features tracker

Covers the rebase of #999 onto main, making the quorum-manager admin client Kerberos-aware so dynamic quorum scaling (#1010) keeps working with Kerberos enabled, and cleaning up the discovery ConfigMap client properties. Part of stackabletech/issues#815. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Eight tasks reconstructing #999 against current main, adding the Kerberos-aware admin client so dynamic quorum scaling keeps working, and fixing the discovery ConfigMap client properties. Part of stackabletech/issues#815. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Controllers have no listener-operator Listener volume: they are only reachable through their own StatefulSet pod DNS name, so their keytab must be pod-scoped. Brokers keep listener-volume scoping. `add_kerberos_pod_config` was previously only called from the broker StatefulSet builder, so controller pods had no keytab at all. Part of stackabletech/issues#815. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Brokers reuse their broker-listener principal for the CONTROLLER section; controllers use their own pod FQDN, matching the pod-scoped keytab. The section deliberately leaves `isInitiator` at its default because controllers connect to each other for Raft. The controller startup command now exports $KERBEROS_REALM and templates jaas.properties, mirroring `broker_start_command`. Part of stackabletech/issues#815. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Also sets sasl.mechanism.controller.protocol=GSSAPI on both broker and controller properties, and corrects the now-false CONTROLLER listener doc comment claiming SASL is unsupported. The controller admin client still assumes plain SSL and is repaired in the following commit; these two must land together. Part of stackabletech/issues#815. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…eros `controller_admin_client_properties` ignored its security argument and hardcoded SSL, so the quorum-manager sidecar and the preStop hook could not reach the now-SASL_SSL CONTROLLER listener. It authenticates as the controller's own pod principal, from the pod-scoped keytab — the correct identity for a voter registering itself. The principal uses config-utils placeholders, so the rendered file must be templated before use (wired up next). Part of stackabletech/issues#815. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The quorum-manager sidecar and the controller preStop hook were both skipped whenever Kerberos was on, because admin-client.properties only covered the TLS/SSL case. Now that it carries GSSAPI settings, remove both gates. admin-client.properties is rendered through `config-utils template` before use in the sidecar and in the kafka container (for preStop), since its principal carries ${env:...} placeholders. The sidecar mounts the keytab and derives $KERBEROS_REALM itself; it deliberately does not get KAFKA_OPTS, which points at a jaas file only the kafka container renders. test_support gains `validated_cluster_with_auth` so fixtures can reference an AuthenticationClass. Part of stackabletech/issues#815. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ent properties The discovery ConfigMap's client.properties is consumed by clients running outside Kafka pods. It carried a sasl.jaas.config whose principal was the literal placeholder `kafka/todo@$KERBEROS_REALM`, which such a client could never use: it has neither the keytab at /stackable/kerberos/keytab nor a per-pod principal. Drop the entry rather than invent a principal, and document that clients bring their own login configuration. Also drops the broker-side sasl.mechanism.inter.broker.protocol and replaces sasl.enabled.mechanisms (broker-side) with the client-side sasl.mechanism. Part of stackabletech/issues#815. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Ports the suite from the draft PR (MIT KDC, 3-controller quorum, produce/consume) and extends it with controller scale-up and scale-down steps. Those steps are the regression test for un-gating the quorum manager under Kerberos, which the original suite predates. The scale asserts check the Raft voter set, not just the StatefulSet replica count, and point the admin client at the rendered /tmp/admin-client.properties. Part of stackabletech/issues#815. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Adds a Kerberos section to the KRaft controller guide (SASL_SSL on the CONTROLLER listener, pod-scoped controller keytabs, dynamic quorum scaling supported) and removes the now-false "Kerberos is currently not supported for KRaft" known issue. Documents in the security guide that clients using the discovery ConfigMap must supply their own principal and keytab. Part of stackabletech/issues#815. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Two defects found running the kraft-kerberos suite; both kept every peer from authenticating to the CONTROLLER listener. 1. `controller.quorum.bootstrap.servers` used the role group headless Service name (#1010), but a GSSAPI client derives its service principal from the hostname it dials, and the CONTROLLER acceptor can offer only one principal - the pod's own, which is also what the Raft voter endpoints advertise. Every broker and joining controller therefore asked for `kafka/<service>` and was rejected with "invalid credentials". Verified on a live cluster: the same admin client against the same pod succeeds via the pod FQDN and fails via the Service name. The property now lists individual pod FQDNs. This gives up #1010's property that the list is stable across replica-count changes, so scaling a controller role group rolls the controller pods. How to handle that churn is deferred. 2. The quorum-manager sidecar had KRB5_CONFIG but not `-Djava.security.krb5.conf`. The JVM reads the system property; the env var only reaches native MIT tools, so the admin client failed with "Unable to locate KDC for realm". It gets KAFKA_OPTS with only that property - still not the kafka container's `java.security.auth.login.config`, which points at a jaas file only that container renders. Part of stackabletech/issues#815. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

This is a bugfix independent of Kerberos support. The controller's liveness probe tried to connect using `localhost`, but the controller doesn't bind to this address. This lead to the probe failing after every 10 minutes and causing the controller pods to be restarted.

Update supported version text to be inline with the version bump for SDP 26.11 release.

razvan and others added 12 commits September 16, 2026 16:19

razvan self-assigned this Sep 17, 2026

razvan added this to Stackable Engineering Sep 17, 2026

razvan moved this to Development: In Progress in Stackable Engineering Sep 17, 2026

razvan added 2 commits September 17, 2026 15:00

Merge branch 'main' into feat/kerberized-kraft-controllers

1865144

fix: cleanup LLM verbosity

9f4a803

Update supported version text to be inline with the version bump for SDP 26.11 release.

razvan moved this from Development: In Progress to Development: Waiting for Review in Stackable Engineering Sep 17, 2026

Navigation Menu

Uh oh!

feat: kerberized Kraft controllers - #1024

feat: kerberized Kraft controllers#1024
razvan wants to merge 14 commits into
mainfrom
feat/kerberized-kraft-controllers

razvan commented Sep 17, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

razvan commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Tests

Definition of Done Checklist

Author

Reviewer

Acceptance

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

razvan commented Sep 17, 2026 •

edited

Loading