feat: TEM-110 pytorch 2.12.0/2.12.1/2.13.0

[Chmokachka]

Summary

Adds PyTorch 2.12.0, 2.12.1 and 2.13.0 to the runpod/pytorch and runpod/pytorch:*-cluster matrices, replaces the single-package Pillow patch with a requirements.txt of CVE pins, upgrades Grafana and Prometheus in the cluster layer, and widens the GPU pool the PyTorch smoke test may draw from.

Matrix	Before	After
runpod/pytorch images	25	36
runpod/pytorch:*-cluster images	4	15

PyTorch 2.12 / 2.13

New TORCH_META entries with torchvision 0.27.0 / 0.27.1 / 0.28.0 and torchaudio pinned to 2.11.0 for all three: 2.11.0 is torchaudio's last release — it was dropped from the PyTorch release process in 2.12, decode/encode moved to TorchCodec. Upstream states it is compatible with later torch versions, and its wheel declares no dependencies, so it cannot downgrade torch. That pairing is the main risk in this PR and it is measured below, on torch 2.12.1 and 2.13.0.

Wheel sources differ from the CUDA base on purpose:

Base	2.12.0	2.12.1	2.13.0
cuda 12.8.1	—	cu129	cu129
cuda 12.9.0	—	cu129	cu129
cuda 13.0.0	cu130	cu130	cu130

• No cu128 wheels from 2.12 on. CUDA 12.8 was deprecated in 2.12 and removed from the build matrix in 2.13, so the 12.x bases take cu129 — it stays in both release matrices and carries the same SM list as cu130 on x86_64. The other 12.x option, cu126, stops at sm_90 and ships no PTX, so it cannot run on Blackwell at all.
• 2.12.0 is CUDA 13 only. cu129 landed in the 2.12 matrix after .0, so 2.12.0+cu129 wheels do not exist; 2.12.1 covers the 12.x bases instead.

The cluster matrix stays a curated subset: 2.12.1 and 2.13.0 on every base, plus 2.12.0 on cuda 13.0.0.

torchcodec

From torchaudio 2.9 on, torchaudio.load and save are thin wrappers around the separate torchcodec package and raise ImportError: TorchCodec is required without it. That was never installed, so audio I/O is already broken in the released 2.9.0 and 2.9.1 images — this PR fixes it rather than introducing it.

torchcodec also replaces torchaudio.io (StreamReader / StreamWriter), the old NVDEC video path, which was deprecated in 2.8 and removed in 2.9 — so shipping the CUDA build keeps the 2.9+ images at parity with 2.6–2.8 on hardware video decoding.

torch	torchcodec	Wheel source
2.9.0, 2.9.1	0.9.1	cu128, or cu130 on the cuda 13.0.0 base
2.12.0, 2.12.1, 2.13.0	0.16.0	same as torch (cu129 / cu130)
2.6.0, 2.7.1, 2.8.0	—	not needed; their torchaudio still ships its own ffmpeg backend and torchaudio.io

torchcodec gets its own install step and its own wheel source, because its CUDA builds do not track torch's index cadence: the cu129 index has no torchcodec for torch 2.9 at all — it jumps from 0.7.0 to 0.10.0 — so the two 2.9.x rows on the cuda 12.9.0 base carry codec_src = "128". Same CUDA major, so the runtime sonames match. Everything else defaults to torch's own whl_src. Installed with --no-deps, so it can never pull a different torch; the arg is empty for 2.6–2.8.

Measured on pods, not inferred:

Image	torchcodec	Audio load / save	Video CPU	Video GPU
torch 2.12.1+cu129, before this PR	none	ImportError	—	—
torch 2.9.0+cu129	0.9.1+cu128	ok, 440 Hz round-trip	(3, 240, 320)	cuda:0
torch 2.9.1+cu129	0.9.1+cu128	ok, 440 Hz round-trip	(3, 240, 320)	cuda:0
torch 2.12.1+cu129	0.16.0+cu129	ok, 440 Hz round-trip	(3, 240, 320)	cuda:0
torch 2.13.0+cu130	0.16.0+cu130	ok, 440 Hz round-trip	(3, 240, 320)	cuda:0

Rows two and three are the codec_src = "128" override — the pairings where torchcodec's CUDA build does not match torch's. GPU decoding lands on cuda:0 in all three, and the base image's ffmpeg carries all ten *_cuvid decoders, so the NVDEC claim in the README is measured rather than assumed. A PyPI CPU wheel of 0.9.1 was also measured working next to torch 2.9.0+cu129, and stays the fallback if a codec_src pairing ever fails to load.

Both torchcodec pins and both wheel sources (cu128/cu129 and cu130) are covered.

Also fixed: torchvision missing on 2.9.1

TORCH_META had "2.9.1" = {} — no torchvision at all, though torchvision 0.24.1 exists on cu128, cu129 and cu130. So the released torch291 images ship without torchvision, and without numpy, which nothing else in the repo installs: import torch warns Failed to initialize NumPy and tensor.numpy() does not work. Pinned to 0.24.1, which brings numpy with it. Pre-existing in 1.1.0, found by running the audio checks above; verified on a rebuilt pod, where the same image reports torchvision 0.24.1+cu129 and numpy 2.5.2 and the warning is gone.

The 2.12.1 and 2.13.0 runs also confirm the thing torchaudio 2.11.0 with torch 2.12/2.13 actually risks: its compiled libtorchaudio links against libtorch, and the wheel declares no dependencies, so pip would install it regardless of ABI. import torchaudio loads the library, checks its compiled CUDA version against torch.version.cuda and calls an op through torch's registry — all three passed, and Spectrogram/MelSpectrogram/Resample produce correct values on the GPU.

Vulnerability pins

The PILLOW_VERSION build arg is replaced by official-templates/pytorch/requirements.txt, mounted through a named build context, so new pins no longer need a bake variable each:

setuptools==81.0.0
jaraco-context==6.1.0
wheel==0.46.2
pillow==12.3.0

torch 2.12.x caps setuptools at <82, and 81.0.0 is the highest release under that cap whose vendored setuptools/_vendor/ copies of jaraco.context and wheel clear CVE-2026-23949 / CVE-2026-24049. Grype catalogs those vendored dist-info dirs, so the top-level pins alone do not clear the findings.

COPY --from=requirements references a build context rather than a FROM alias, which hadolint flags as DL3022 — added to the ignore list in both hadolint workflows.

Monitoring stack: Grafana 13.1.5, Prometheus 3.14.0

Both were flagged by Grype in the cluster image.

• Grafana 13.1.1 → 13.1.5 closes CVE-2026-13438, CVE-2026-17183 (High), CVE-2026-12704, CVE-2026-14199 (High) and CVE-2026-19475, plus — because the tarball also carries the bundled datasource plugins — a Critical in kin-openapi and findings in etcd, cel-go, otel, klauspost/compress, grpc, x/crypto, x/net and x/text.
• Prometheus 3.13.1 → 3.14.0 picks up x/text 0.40.0 (CVE-2026-56852, High), grpc 1.82.1 (GHSA-hrxh-6v49-42gf, High) and a newer Go toolchain.

Neither is a breaking change for this image. Grafana is a patch bump inside 13.1.x; its three behaviour-affecting security fixes touch alert-rule authorization, SQL-datasource macro parsing and the auth-proxy cache key, and this image provisions no alert rules and only a Prometheus datasource. Prometheus 3.14.0 lists five CHANGE entries — the deprecated stats query parameter (warning only), empty separator/replacement in /api/v1/status/config, a dropped Hetzner label, duration expressions on by default, and first_over_time promoted to stable. The generated scrape config uses only scrape_interval, evaluation_interval and static_configs; the dashboards use no range selector other than [5m]; the supervisor program passes no feature flags. TSDB data is pod-local, so storage compatibility does not apply.

CI cache key

The cluster sticky-disk key gains a -v2 suffix. The previous snapshot held a partial blob from an interrupted base-image pull, and every following run failed on it with failed commit on ref "layer-sha256:…": unexpected commit digest … failed precondition while computing a cache key — so a re-run could not clear it. Only the /pr lineage was affected; the release lineage on main was clean.

Compatibility, measured

53 pods on RunPod, 2026-09-01, across sm_80 … sm_120 in both cloud tiers, on 1.2.0-rc.162. Every row is a real pod: created, booted, SSH-verified, then torch.cuda.is_available() plus a matmul on the GPU.

Image (runpod/pytorch:1.2.0-rc.162-…)	Compatible CUDA	Incompatible GPU
cu1281-torch2121-ubuntu2204	≥ 12.8	none found
cu1281-torch2121-ubuntu2404	≥ 12.8	none found
cu1281-torch2130-ubuntu2204	≥ 12.8	none found
cu1281-torch2130-ubuntu2404	≥ 12.8	none found
cu1290-torch2121-ubuntu2204	≥ 12.9	none found
cu1290-torch2121-ubuntu2404	≥ 12.9	none found
cu1290-torch2130-ubuntu2204	≥ 12.9	none found
cu1290-torch2130-ubuntu2404	≥ 12.9	none found
cu1300-torch2120-ubuntu2404	≥ 13.0	RTX 5080
cu1300-torch2121-ubuntu2404	≥ 13.0	RTX 5080
cu1300-torch2130-ubuntu2404	≥ 13.0	RTX 5080

• "Compatible CUDA" is the host driver floor and equals the CUDA version of the image's base layer. Below it the image falls back to CUDA forward compatibility, which NVIDIA supports on datacenter GPUs only — a cu1290 image pinned to 12.8 passed on A100, A40 and RTX PRO 6000 and failed on RTX 4090, RTX 4080 SUPER and RTX 5090. Set the floor from the tag.
• Ubuntu 22.04 vs 24.04 made no difference in any pairing measured on both.

Follow-ups

• Bump dcgm-exporter to 4.6.0-4.8.3 and update the dashboards for the Hostname → hostname label rename.

[kodxana]

Reviewed the latest changes. I don’t see a blocking code issue, and the current PyTorch and cluster build/test jobs passed across all three CUDA groups. Looks good from my side.

One thing to coordinate with #161: both PRs change the GPU exclusion list in base.yml and conflict there. Please resolve that with the older PyTorch versions still in the matrix in mind, then recheck the rebased PR.

[satish-runpod]

Since this PR is adding torch 2.12.0/2.12.1/2.13.0, would it make sense to also add a CUDA 13.2.1 entry here? download.pytorch.org/whl/cu132 actually has native wheels for exactly these torch versions (2.12.0, 2.12.1, 2.13.0, 2.14.0) — none of the currently-supported CUDA versions (12.8.1/12.9.0/13.0.0) can use a native cu132 wheel, so right now 2.12+ can only be paired via a borrowed whl_src (e.g. 130).

Concretely:

• Add { version = "13.2.1", ubuntu = ["24.04"] } to CUDA_VERSIONS in official-templates/shared/versions.hcl (mirrors the 13.0.0 scoping — 24.04-only).
• Add to CUDA_TORCH_COMBINATIONS:

  { cuda_version = "13.2.1", torch = "2.12.0", whl_src = "132" },
  { cuda_version = "13.2.1", torch = "2.12.1", whl_src = "132" },
  { cuda_version = "13.2.1", torch = "2.13.0", whl_src = "132" },
  

• base.yml's CI matrix (cuda: [cu1281, cu1290, cu1300], 4 spots) would need cu1321 added.

Happy to help wire this up if useful — just flagging since it lines up closely with what's already in flight here.