[Kubernetes] In a pod, context=None means this cluster, not the kubeconfig's current-context

[tigist-far]

Summary

A managed job pinned to nodes (kubernetes.allowed_nodes with names or ips) failed to provision on the jobs controller — every pod, every retry — with unknown command "oidc-login" for "kubectl", while the same task unpinned provisioned fine.

Cause. The controller provisions with context=None: get_context_from_config maps the in-cluster context to None so pods are created with in-cluster auth. Pinning is the only path that lists nodes per pod (inject_allowed_nodes_affinity → get_kubernetes_nodes), and get_kubernetes_nodes(context=None) replaced None with the kubeconfig's current-context name before building its client. The adaptor authenticates an explicit name through the kubeconfig, and the kubeconfig on the controller is the client's, rebuilt by the API server from the client's KUBECONFIG at every launch: its current-context was the developer's other cluster, authenticated with an exec plugin the pod does not have.

flowchart LR
    L["client launch<br/>allowed_nodes set"] --> C["jobs controller pod<br/>context=None (in-cluster auth)"]
    C --> P["inject_allowed_nodes_affinity"]
    P --> N["get_kubernetes_nodes(context=None)"]
    N -- before --> K["current-context NAME from the<br/>client's kubeconfig → exec plugin<br/>oidc-login: not in the pod"]
    N -- after --> I["resolve_current_context()<br/>→ in-cluster context, in-cluster auth"]

Loading

Fix. resolve_current_context() returns the in-cluster context name when in-cluster auth is available and the kubeconfig's current-context otherwise. The five places in sky/provision/kubernetes/utils.py that substituted the current-context name for None use it:

Site	Effect of the change
get_kubernetes_nodes	node listing from a pod authenticates in-cluster; the allowed-nodes filter sees the in-cluster name
get_allocated_resources_by_node	same
get_kubernetes_node_info (two substitutions)	same; per-context config lookups key on the cluster the pod is in
get_skypilot_pods	same

Outside a pod nothing changes: the kubeconfig's current-context is used as before, and an explicit context is used as given.

Areas changed

Area	Files	What changed
sky/provision/kubernetes/utils.py	1 (+29/−6)	resolve_current_context(); the five None → current-context substitutions above go through it
tests/unit_tests/kubernetes/test_kubernetes_utils.py	1 (+66)	In a pod, context=None reaches the client as the in-cluster context and the allowed-nodes filter sees the same name; outside a pod it is still the kubeconfig's current-context; an explicit context is used as given

Testing

tests/unit_tests/kubernetes/test_kubernetes_utils.py: 212 passed (the three new tests included). The fork has no GitHub Actions on pull requests.

Upstream

The same code is on upstream master (get_kubernetes_nodes and get_context_from_config are identical), so this is an upstream bug. The commit is written upstream-first; a copy cherry-picked onto upstream master is on tigist/upstream-in-cluster-current-context for an upstream PR (this repository is not a GitHub fork of skypilot-org/skypilot, so that PR must come from one that is).

The workaround in nemotron/nemo-rl (forwarding a kubeconfig whose current-context is the pinned cluster) stays; it is harmless and covers today's pinned build.

[tigist-far]

Checks run for this PR (the repository runs no CI on PRs): tests/unit_tests/kubernetes/test_kubernetes_utils.py passes in full (212 on this branch; 234 on the upstream-master copy tigist/upstream-in-cluster-current-context, 83bb480). Formatting verified with the pinned yapf==0.32.0 and the repo's [tool.yapf] config: zero hunks on both changed files. isort==5.12.0 --check-only reports the same pre-existing import-order differences on the unmodified base versions of both files, so nothing here changes that; the one added import (io) sits in alphabetical order.