fix: move /tmp off the node root filesystem onto a bounded ephemeral volume

[ppat]

Problem

A long-lived workspace pod's /tmp — part of the container's writable overlay layer, no explicit mount — had grown to ~7 GB, ~6.3 GB of it Claude Code's own scratch directory ($TMPDIR/claude-<uid>/..., used heavily by agent sessions for downloads and experiments). The overlay lives on the node's single root partition (~125 GB, ext4), which was at 81.7% usage — close to the kubelet's disk-pressure eviction threshold. That's a risk to every pod on the node, not just the workspace that caused it.

/tmp is wiped for free on every container restart today, because a new container instance gets a fresh overlay upperdir. That property needed to survive whatever replaced the overlay.

Change

/tmp now mounts a Kubernetes generic ephemeral volume on the sc-longhorn-local-non-replicated-ephemeral storage class, capped at 20Gi (deployment.tf). That class is node-local NVMe on a separate, much larger Longhorn-backed partition on the same node — not the constrained root partition — and non-replicated, since scratch data costs nothing to lose. script-agent-startup.sh now wipes /tmp's contents explicitly on every agent start (set -eo pipefail already in that script means a failed wipe aborts the blocking startup script and shows up as a failed agent startup in the Coder UI, not a silent leak).

CLAUDE.md and DESIGN.md get a short note each explaining the volume choice and the lifecycle gap it creates (Pod-scoped, like the existing system volume — a container-only restart within a live Pod does not get a fresh volume, which is exactly why the explicit wipe exists).

Rejected alternatives

• Redirect only Claude Code's scratchpad (CLAUDE_CODE_TMPDIR, verified as the actual mechanism Claude Code honors for this — checked before os.tmpdir()/TMPDIR, independent of the generic sockets/IPC paths that still use raw os.tmpdir()) to the NFS-backed home PVC. Solves ~90% of the observed growth but leaves the remaining non-Claude scratch (build caches, coder-src checkouts, node_modules) on the constrained partition, and NFS is a bad fit for exactly what's left (write-heavy, latency-sensitive build caches). Moving all of /tmp to fast node-local storage instead subsumes this fix with no Claude-specific configuration at all.
• A bounded empty_dir. Ruled out once the node's actual partition layout was checked: empty_dir lives on the node's root filesystem, the exact partition this change exists to stay off of. The template's existing system volume uses this pattern for /usr, /etc, /var, but that precedent doesn't transfer here.
• A dedicated PVC whose lifecycle is gated on workspace stop/start (count = data.coder_workspace.me.start_count, matching how this template's other resources are gated). Checked against the live pod: it's 18 days old with 3 container restarts, most recently 8 days ago — meaning the Pod itself has not been recreated in that window, only the container within it. A start-gated resource would not have reset across any of those 3 restarts, which is strictly worse than the free per-restart wipe the overlay already provided. This is why the wipe is an explicit script step rather than something left to any volume's lifecycle.

What this does not do

• Does not touch the current running workspace's existing /tmp contents — the volume change only takes effect on that workspace's next rebuild (its Pod's RestartPolicy-driven in-place restarts don't recreate the Deployment's pod template; a Coder-initiated start/rebuild does). No cleanup of the existing accumulation is included.
• 20Gi is a judgment call, not measured against real peak legitimate usage (tarball downloads, build caches) — generous headroom over the observed ~7 GB/8-day accumulation, not a calibrated number. Easy to revisit once real usage is observed against the new bound.
• Does not prove the new storage class actually binds and mounts under this template — see Verification below. That can only be confirmed by an actual workspace build, which is outside what this PR's own checks perform.

How a failed wipe surfaces

script-agent-startup.sh runs under set -eo pipefail and startup_script_behavior = "blocking"; a failed find /tmp -mindepth 1 -delete aborts the script, which blocks the agent from reporting ready and shows as a failed startup script in the Coder UI — not a gradual, unnoticed leak.

What to measure

• Node root partition usage trend over the following weeks (should stop climbing from this workspace's /tmp).
• Whether the new PVC binds and attaches without adding materially to pod start/restart latency.
• Whether build-cache-heavy workflows (Go, npm, docker buildx) see any latency regression versus the old overlay-backed /tmp — expected to be negligible (still node-local NVMe) but not yet measured.
• Whether 20Gi is ever approached in practice, as the signal for whether that number needs revisiting.

Does Longhorn's CSI driver support this?

Yes, and no driver capability is involved — the question turns on which of two
different mechanisms is used.

• CSI inline ephemeral volumes (spec.volumes[].csi:) require the driver to declare
  volumeLifecycleModes: [Ephemeral] on its CSIDriver object. Longhorn does not.
• Generic ephemeral volumes (spec.volumes[].ephemeral.volumeClaimTemplate, what this
  PR uses) are a Kubernetes-level feature. The controller creates an ordinary PVC named
  <pod>-tmp, owned by the Pod, satisfied through normal dynamic provisioning. Any driver
  that can dynamically provision works; Longhorn does.

sc-longhorn-local-non-replicated-ephemeral was confirmed present in the cluster
(provisioner driver.longhorn.io, WaitForFirstConsumer, Delete reclaim, Flux-managed).
It pre-dates this change and was not created for it.

Verification

• pre-commit run --all-files, terraform fmt -check, terraform validate, tflint --config=../../../.tflint.hcl all pass. Pre-existing hadolint findings (SC3037, DL3066) on the Dockerfile are untouched by this change.
• The dry-run release pipeline passed: lint fully green, and release's publish-template job confirms coder template push accepts this config against the real Coder deployment and Kubernetes API — the plan resolves sc-longhorn-local-non-replicated-ephemeral and produces Plan: 7 to add, 0 to change, 0 to destroy with no errors.
• That job is a Terraform plan only (coder template push registers a template version; it does not build a workspace), so it does not create a real Pod or PVC. Whether the ephemeral volume actually binds, attaches, and mounts under WaitForFirstConsumer scheduling is not yet verified — that needs an actual workspace build from this template version, which is outside what I can do here (building one would itself be an apply, and this task is scoped to plan/lint-level checks only). Worth doing before or shortly after merge.

Overlap with #863

No file overlap — that PR touches script-memory-watchdog.sh and script-memory-watchdog-test.sh, this one touches deployment.tf and script-agent-startup.sh. Both branches add content to CLAUDE.md and DESIGN.md, so whichever merges second will need a routine rebase (text-only, no logic conflict expected).

[homelab-workflows-bot]

🎉 This PR is included in version 2.26.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀