[SPARK-58203][K8S] Support configuring spark-managed UI service to work with spark.ui.port=0

[zhengchenyu]

What changes were proposed in this pull request?

Add config spark.kubernetes.driver.ui.service.enabled (default false). When enabled:

• A new DriverUIServiceFeatureStep creates a dedicated ClusterIP Service exposing the driver's Web UI port.
• After SparkContext initializes and the Web UI binds,K8sDriverUIServicePatcher (invoked from KubernetesClusterSchedulerBackend.start()) patches the Service's targetPort to the actual bound port.

When spark.ui.port=0, the Service is initially built with the default UI port as placeholder to satisfy Kubernetes port validation; the runtime patch then rewrites targetPort.

Two other configs: spark.kubernetes.driver.ui.service.type (ClusterIP / NodePort / LoadBalancer, default ClusterIP) and spark.kubernetes.driver.ui.service.name (optional explicit name).

Why are the changes needed?

In hostNetwork mode, spark.ui.port=0 is needed to avoid UI port collisions between drivers on the same host. Today the driver UI Service is created by spark-operator, not by Spark itself, and spark-operator has no way to know the actual UI port the driver ends up binding to.

Letting Spark create and manage the UI Service resolves this because Spark knows the actual bound port and patches the Service accordingly.

Does this PR introduce any user-facing change?

Yes. Three new opt-in configs, default off.

How was this patch tested?

• New DriverUIServiceFeatureStepSuite:
• Test on kubernetes cluster.

Was this patch authored or co-authored using generative AI tooling?

[sunchao]

Summary

This PR adds an opt-in, Spark-managed Kubernetes Service for the driver Web UI and patches its targetPort after Jetty binds. The direction addresses the host-network/random-port problem, but the current head does not yet make the stated spark.ui.port=0 path valid and misses one supported backend path.

Prior state and problem

The existing headless driver Service is built from the configured spark.ui.port. In host-network mode, multiple drivers can collide on a fixed UI port, while an external operator cannot know the random port selected by spark.ui.port=0.

Design approach

The patch adds enable/type/name configurations, creates a dedicated UI Service with a stable Service port and placeholder target port, passes the Service name to the driver, and patches targetPort from SparkUI.boundPort during Kubernetes scheduler-backend startup.

Correctness / compatibility analysis

Default-off behavior remains backward compatible, and keeping a stable Service port while patching only targetPort is appropriate for ClusterIP, NodePort, and LoadBalancer. The enabled path is currently blocked because the existing driver Pod and headless Service still contain port zero. In addition, driver-only local[*] applications never call the patcher, and the documented RBAC grant omits the GET verb used by the implementation.

Key design decisions

• Keep the externally visible Service port stable while patching only targetPort.
• Ensure no generated Kubernetes manifest contains a numeric port of zero.
• Run patching from a driver lifecycle shared by Kubernetes and local driver-only backends, or explicitly scope the feature away from driver-only mode.
• Treat get and patch as required Service permissions if retaining the read-before-write implementation.

Implementation sketch

1. When the dedicated UI Service is enabled and spark.ui.port=0, omit the UI container-port declaration and omit or substitute the UI entry on the existing headless Service.
2. Move the patch hook to a shared driver initialization point after UI binding, with Kubernetes-client access, or reject this configuration for driver-only mode.
3. Update the RBAC/configuration documentation and add full-builder plus patcher/backend tests.

Behavioral changes worth calling out

Enabling the feature creates an additional Kubernetes Service and currently makes patch failure fatal during SparkContext initialization. LoadBalancer can expose the UI externally. The behavior for supported spark.kubernetes.driver.master=local[*] applications therefore needs to be defined explicitly.

Suggested improvements

Add a full KubernetesDriverBuilder test for spark.ui.port=0, focused patcher tests for matching/missing/updated targets, integration coverage for driver-only mode, and the three new settings plus their RBAC requirements to docs/running-on-kubernetes.md.

[sunchao]

[P1] Handle port zero in the existing driver resources too

This placeholder only fixes the new dedicated Service. With spark.ui.port=0, BasicDriverFeatureStep still emits containerPort=0, and DriverServiceFeatureStep still emits port=0 / targetPort=0 for the mandatory headless Service. Kubernetes requires these numeric ports to be in 1..65535, so the driver Pod/resources are rejected before this runtime patch can run. Please omit or substitute both existing UI port declarations and cover the complete KubernetesDriverBuilder output in a test.

[sunchao]

[P2] Patch the Service in driver-only mode too

spark.kubernetes.driver.master=local[*] is a supported driver-only mode, but KubernetesClusterManager returns LocalSchedulerBackend for it, so this method never runs even though DriverUIServiceFeatureStep still creates the Service. After the zero-port manifest issue is fixed, the UI binds randomly while targetPort remains at the placeholder; a nonzero bind collision produces the same stale target. Please move this hook to a driver lifecycle shared by both backends, or explicitly reject/disable the feature in driver-only mode.

[sunchao]

[P2] Include GET in the required RBAC permissions

The patcher calls get() before patch(), and Kubernetes RBAC verbs are independent. A least-privilege role granting only the documented patch verb receives a 403 on the GET; the exception is rethrown from backend startup and aborts SparkContext initialization. Please document both get and patch, or issue a targeted patch that does not require the preliminary read.