Add machine-readable deployment validation tools

[dholt]

Summary

• Add scripts/validation/ with three dependency-free Python tools that answer "did the deployment work?" with a stable machine-readable contract for automation and AI agents:
  • deepops_doctor.py — provisioning-environment preflight: Ansible present, Galaxy dependencies installed, Kubespray submodule initialized, config directory present, inventory parses with hosts; --remote adds host reachability, NVIDIA PCI device detection, and sshd override reporting.
  • validate_slurm.py — Slurm deployment validation: controller reachability, node availability, gres/gpu configuration, and a single-GPU srun nvidia-smi job.
  • validate_k8s.py — Kubernetes GPU deployment validation: node readiness, allocatable nvidia.com/gpu, GPU Operator pod health, and an optional CUDA smoke pod (--cuda-smoke).
• All tools print one line per check by default or one flat JSON object with --json, exit 0/1/2, and fail loudly with an explicit failures list.
• Document the Slurm login GPU-hiding behavior (direct nvidia-smi over SSH reports no devices on cluster nodes) that otherwise reads as a driver failure.
• Add docs/deepops/validation.md and standard-library unit tests.

Validation

• python3 -m unittest discover -s scripts/validation/tests: 10 tests, all pass
• py_compile and --help smoke for all three tools
• deepops_doctor.py run against an unprepared checkout correctly fails with actionable fix hints for each missing prerequisite
• Live single-node GPU validation: results will be posted as a follow-up comment before review
• git diff --check, public sanitizer on all added files, OS compatibility and component audits unchanged

Notes

• These tools are read-only except for the opt-in CUDA smoke pod, which creates and deletes a dedicated deepops-validate namespace (kept on failure for debugging).
• scripts/k8s/verify_gpu.sh remains the exhaustive every-GPU job test; these tools are the fast structured health contract.

[dholt]

Live validation summary

All three tools were exercised end-to-end on a disposable single-GPU server (Ubuntu 22.04) through full Slurm and Kubernetes deployments:

• deepops_doctor.py --remote --json — full pass on a prepared provisioning environment: Ansible, Galaxy dependencies, Kubespray submodule, config, inventory parse (1 host, all expected groups), host reachable, NVIDIA PCI device detected. Run again after the Slurm deployment, it correctly reports the host restricting GPU device access for SSH sessions.
• validate_slurm.py --json on the deployed node — "ok": true, exit 0: controller reachable, 1/1 nodes available, gres/gpu configured, single-GPU srun job passed (slurm_version: 26.05.1).
• validate_k8s.py --json --cuda-smoke on the deployed cluster — a first run during GPU Operator startup honestly reported "ok": false with 1 of 5 GPU stack pods are not ready while the CUDA smoke still passed; a run on the settled stack reports "ok": true, exit 0, with gpus_allocatable: 1, all 5 GPU stack pods ready, and cuda_smoke_ok: true (namespace created and cleaned up).

The live run also caught and fixed a real detection bug: Slurm login GPU hiding is applied as a runtime DeviceAllow property (drop-in under /etc/systemd/system.control/), so the doctor now queries the property instead of listing unit-override directories.

All CodeQL/code-quality findings on the initial head were fixed and each bot thread answered.