fix(chart): Helm-managed GPU device plugin (client#564)

[shujaatTracebloc]

Problem

The GPU device plugin was applied with an imperative kubectl apply of an upstream manifest fetched at install time, outside any Helm release. It wasn't release-tracked, so it lingered through helm uninstall and was re-created on every installer re-run, and it added a raw.githubusercontent.com network dependency to the install path. (scripts/install-k8s.ps1:2490 in the issue is stale; the live Linux apply was scripts/lib/gpu-plugins.sh → deploy_gpu_device_plugin.)

Fix

Render the plugin as a Helm-managed DaemonSet — client/templates/gpu-device-plugin.yaml, gated on gpu.devicePlugin.enabled + vendor, in kube-system, with the manifest baked into the chart (no install-time download). It's reconciled on upgrade and removed on helm uninstall. No helm.sh/resource-policy: keep — sharing the release lifecycle is the point.

• lib/install-client-helm.sh sets gpu.devicePlugin.{enabled,vendor} from GPU_VENDOR, in lockstep with the GPU request it already writes for training jobs.
• install-k8s.sh no longer applies the plugin imperatively before Helm; verify_gpu moves to after the Helm install (the plugin now rolls out with the release).
• A bounded, best-effort adoption shim (_adopt_orphaned_gpu_device_plugin) labels/annotates a pre-existing, Helm-unowned DaemonSet so a re-run after helm uninstall adopts it in place instead of failing with "exists and cannot be imported" — exactly the issue's re-run scenario.
• Chart.yaml version + appVersion bumped 1.9.39 → 1.9.40 (chart-version-guard).

Scoping: Linux vs Windows

Linux requests a GPU purely on GPU_VENDOR detection, so a chart-managed plugin works cleanly. Windows can't use it here: the non-WSL2 fallback must verify the node advertises a GPU before it writes the GPU request, and a Helm-managed resource only rolls out during Helm install — so that path keeps its imperative apply for now. The WSL2/CDI path deliberately never runs the NVML plugin. Default gpu.devicePlugin.enabled: false means Windows renders no chart DaemonSet (no double-deploy).

Tests

• New helm-unittest suite client/tests/gpu_device_plugin_test.yaml (default renders nothing; nvidia/amd render; namespace+image override; Helm-ownership label; fail-on-missing-vendor).
• scripts/tests/gpu-nvidia.bats retargeted from the removed imperative-apply assertions to the chart-managed invariants; install-client-helm.bats +3 for the adoption shim.
• Local: helm lint --strict (all 4 platforms), helm template renders, bats suites all green. (4 unrelated helm-unittest suites fail only under local helm v4 / unittest 1.1.0 vs CI-pinned v3.15.4 / 0.5.2 — same on develop.)

Follow-ups (not in this PR)

1. Migrate the Windows non-WSL2 fallback once Confirm-GpuNode can be reordered (verify-before-request). The chart template already supports it.
2. Pin the AMD image (rocm/k8s-device-plugin:latest) to a digest for air-gapped/reproducible installs.

Closes #564

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes cluster-level DaemonSets in kube-system and install ordering for GPU paths; adoption/delete shims and bounded rollout waits reduce but do not eliminate upgrade friction on clusters with pre-existing plugins.

Overview
Moves the GPU device plugin from imperative kubectl apply + GitHub manifest fetch into the Helm chart, so it is reconciled on upgrade and removed on helm uninstall (client#564). Chart 1.9.41 adds gpu.devicePlugin (NVIDIA/AMD DaemonSets in kube-system by default), nil-safe gating, global.imageRegistry / digest pinning, and optional imagePullSecrets with the pull secret mirrored into the plugin namespace.

The Linux installer stops deploying the plugin before cluster/Helm setup; install-client-helm.sh turns the plugin on from GPU_VENDOR and runs _adopt_orphaned_gpu_device_plugin so leftover imperative DaemonSets are adopted (or deleted) instead of blocking helm upgrade --install. verify_gpu runs after Helm and waits on DaemonSet rollout before node capacity polling. Imperative deploy logic is removed from gpu-plugins.sh; helm-unittest and bats cover rendering, adoption, and ordering.

^{Reviewed by Cursor Bugbot for commit dd78549. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 2438ef0. Configure here.}

[LukasWodka]

Reviewed the full diff and the Bugbot iteration end-to-end. This correctly moves the GPU device plugin off the imperative kubectl apply of a fetched raw.githubusercontent.com manifest and into a Helm-tracked DaemonSet (client/templates/gpu-device-plugin.yaml), gated on gpu.devicePlugin.enabled (default false, so CPU-only and the Windows/WSL2 path render nothing), with the manifest baked into the chart and deliberately no helm.sh/resource-policy: keep — so it now reconciles on upgrade and is removed on helm uninstall, which is exactly what #564 asked for.

I confirmed the old Linux imperative path is fully gone, not just superseded: deploy_gpu_device_plugin / _apply_remote_manifest / _deploy_{nvidia,amd}_plugin are deleted from scripts/lib/gpu-plugins.sh, install-k8s.sh no longer calls them (verify_gpu now runs after the Helm install), and a bats guard asserts no kubectl apply/curl/old-helpers survive — so there's no orphaned DaemonSet double-management. The NVIDIA/AMD DaemonSets faithfully mirror upstream (kube-system, nvidia.com/gpu toleration + system-node-critical, FAIL_ON_INIT_ERROR=false so a non-GPU node no-ops instead of crash-looping, arch nodeSelector + CriticalAddonsOnly for AMD, hostPath device-plugins socket), the installer flips the flag in lockstep with GPU_VENDOR, and _adopt_orphaned_gpu_device_plugin cleanly handles the re-run-after-uninstall 'exists and cannot be imported' collision. The reuse-values nil-guard (default dict chain) and global.imageRegistry + pull-secret mirroring close the air-gap/reuse-values gaps Bugbot raised.

Chart bumped 1.9.40 -> 1.9.41 (version guard green), manifest.sha256 regenerated for the 3 touched scripts (Static-analysis/R8 green), all 9 Bugbot findings resolved, and every required check is green. Solid, well-tested change (helm-unittest + bats both assert real invariants rather than restating the template).

Minor, non-blocking: the adoption-fallback kubectl delete in install-client-helm.sh still carries only --timeout=30s while its sibling get/label/annotate calls all gained --request-timeout — a fully wedged apiserver could leave that one HTTP call unbounded. Bugbot flagged and it was resolved, so just noting for awareness. Nice work.