fix: move /tmp off the node root filesystem onto a bounded ephemeral volume

CLAUDE.md

@@ -92,6 +92,7 @@ Things that look arbitrary in the code but are load-bearing (full reasoning in [
 - **The watchdog decides what is a VS Code process by executable path — `argv[0]` under `~/.vscode-server/` — never by whether something "is node".** A provisioned workspace has two unrelated node installations: VS Code's bundled one under `~/.vscode-server/cli/servers/Stable-<commit>/server/`, and mise's on `PATH`, which is what repo tooling and the operator's agent sessions run on. (There is no `/usr/bin/node`, and nothing named `node` on `PATH` at all without dotfiles.) Matching on `comm`, on a basename, or on a loose cmdline substring would classify an agent session spawned by an extension — a child of the extension host, and *not* under ptyHost — as a sheddable editor helper. `comm` in particular is `MainThread` for every node process in a real tree, never `node`, because V8 renames its main thread; nothing may key off it. `script-memory-watchdog-test.sh` asserts this three ways, each paired with the mutation that flips it.
 - The watchdog never signals anything in the `--type=ptyHost` subtree. Tree membership alone is *not* a safe kill criterion: tmux sessions and agent runs started from a VS Code integrated terminal are descendants of the server tree through ptyHost, so a tree-wide kill would take the operator's work with it. The exclusion is asserted, together with the mutation that must flip it, in `script-memory-watchdog-test.sh`.
 - Adding a package/tool has three possible homes, and picking the wrong one is a real mistake, not a style choice — route by the rule in [DESIGN.md](DESIGN.md#where-the-workspace-environment-comes-from): universal + stable → image (`Dockerfile`); occasionally-needed + apt-only + too heavy to bake in → the template's `system_packages` parameter; personal, fast-moving, or not an apt package → the operator's dotfiles (a *different* repo — see below), never this one.
+- `deployment.tf` mounts `/tmp` on its own ephemeral Longhorn volume, not the node's root filesystem and not the NFS-backed home PVC - see [DESIGN.md](DESIGN.md#design-tensions-and-decisions) for why both of those are wrong for it. Its lifecycle is per-Pod, the same as the `system` volume, so it is *not* wiped by a container-only restart within a live Pod - `script-agent-startup.sh` wipes it explicitly on every agent start instead. Anything relying on `/tmp` persisting across an agent restart was already wrong before this (the same was true for free when it was the container's writable overlay).
 - `deployment.tf`'s Deployment `metadata.name` (`local.workload_name` in `main.tf`) is not cosmetic: the cluster's Prometheus resolves pod → ReplicaSet → Deployment via an existing `kube_pod_owner` recording rule and exposes the result as a `workload` label with no other join needed, so whatever this Deployment is named *is* the identity CPU/memory/PSI/OOM metrics get attributed to. Don't revert it to an opaque identifier (e.g. the workspace UUID) without re-breaking that attribution — see [DESIGN.md](DESIGN.md#design-tensions-and-decisions).
 
 ## Neighbouring repos

DESIGN.md

@@ -61,6 +61,8 @@ The rule that ties the layers together: a package or tool belongs in the *lowest
 
 **No staging environment, so the release pipeline carries its own rehearsal path.** There's exactly one live template and one live cluster — no separate staging Coder deployment to try changes against first. Rather than accept "every merge to main is a live-fire test," the release pipeline itself can run in a mode that exercises a real build and a real (but disposable, clearly-named) template push without touching the production template or its persistent state. That path is what makes it safe to iterate on template/image changes at the same pace as everything else in the repo. See [TESTING.md](TESTING.md) for how to use it.
 
+**`/tmp` is node-local scratch space, deliberately not the shared home volume.** Every workspace's `/tmp` used to be whatever the container's writable overlay layer gave it for free - fast, but unbounded, and on the node's root filesystem. On the operator's own long-lived workspace that grew to several GiB (dominated by Claude Code's own scratch directory, `$TMPDIR/claude-<uid>/...`, which agent sessions use for downloads and experiments) and pushed the node toward the kubelet's disk-pressure eviction threshold - a risk to every other pod on that node, not just the workspace that caused it. The home PVC has ample free space, but is NFS-backed, which is a bad fit for what actually lives in `/tmp`: build caches and compiler intermediates are exactly the write-heavy, latency-sensitive workload NFS handles worst. A plain `empty_dir` would keep `/tmp` fast but doesn't fix anything, because `empty_dir` lives on the same constrained node root filesystem the container overlay already did. The fix is a Kubernetes "generic ephemeral volume" on a Longhorn storage class that is both node-local (so still fast) and backed by a separate, much larger partition on the same node than the root filesystem is - see the comment on the `tmp` volume in `deployment.tf` for the specific class and why it's the non-replicated one (scratch data costs nothing to lose) rather than the default replicated class other PVCs in this cluster use. Its size is a fixed ceiling rather than left unbounded, so a runaway consumer now fails predictably inside its own volume instead of eventually pressuring the node. Because this volume's lifecycle is tied to the Pod rather than the container, `script-agent-startup.sh` also wipes it on every agent start, so a container restart within a live Pod doesn't just inherit whatever the previous container left behind.
+
 **Unprivileged by default.** The workspace itself runs as an unprivileged, non-root, fixed-identity container. Anything that genuinely needs elevated privilege (installing packages, preparing shared volume state) is scoped to a narrow, short-lived setup step that runs before the workspace shell exists, not to something the workspace user can reach into.
 
 **The Deployment name is a Prometheus identity, not just a Kubernetes identifier.** `deployment.tf` names the workspace Deployment `coder-workspace-<owner>-<workspace-name>` (`local.workload_name` in `main.tf`) rather than the workspace UUID it used before. cAdvisor's `container_*` series carry no Kubernetes labels — they come from the cgroup filesystem, with no API-server connection — so per-workspace CPU/memory/PSI/OOM can only be attributed to a human-readable identity through the cluster's existing `namespace_workload_pod:kube_pod_owner:relabel` recording rule, which resolves pod → ReplicaSet → Deployment into a `workload` label. That rule already runs for free; naming the Deployment meaningfully is the only lever this repo has to make its output meaningful, at zero added Prometheus series and no PromQL join. Three things shape the exact scheme:

templates/kubernetes/homelab-workspace/deployment.tf

@@ -156,6 +156,10 @@ resource "kubernetes_deployment_v1" "deployment" {
             name       = "system"
             sub_path   = "var"
           }
+          volume_mount {
+            mount_path = "/tmp"
+            name       = "tmp"
+          }
         }
         enable_service_links = false
         hostname             = local.sanitized_workspace_name
@@ -202,6 +206,40 @@ resource "kubernetes_deployment_v1" "deployment" {
             size_limit = "10Gi"
           }
         }
+        # /tmp is scratch space (agent/tool tempfiles, build caches, downloaded
+        # archives) and needs to be fast - it cannot be the NFS-backed "home"
+        # PVC, and it cannot be an empty_dir either, because empty_dir lives on
+        # the node's root filesystem, which is the exact partition this volume
+        # exists to stay off of (single ~125Gi ext4 partition shared by every
+        # pod on the node; container writable layers and empty_dirs all land
+        # there, and it is what the kubelet's disk-pressure eviction threshold
+        # watches). A Kubernetes "generic ephemeral volume" on
+        # sc-longhorn-local-non-replicated-ephemeral instead lands on the same
+        # node's much larger Longhorn-backed partition: still node-local NVMe
+        # (no NFS latency), not replicated (this is scratch data - losing it on
+        # node failure costs nothing, so paying to replicate it would be pure
+        # overhead), and bounded by the size below instead of growing until the
+        # node notices. Its lifecycle matches the Pod's (created fresh, deleted
+        # with it) - like the "system" volume above, that means a Pod restart
+        # gets a clean volume but a container-only restart within a live Pod
+        # does not, which is why script-agent-startup.sh also wipes /tmp's
+        # contents explicitly on every agent start instead of relying on this.
+        volume {
+          name = "tmp"
+          ephemeral {
+            volume_claim_template {
+              spec {
+                access_modes       = ["ReadWriteOnce"]
+                storage_class_name = "sc-longhorn-local-non-replicated-ephemeral"
+                resources {
+                  requests = {
+                    storage = "20Gi"
+                  }
+                }
+              }
+            }
+          }
+        }
       }
     }
   }

templates/kubernetes/homelab-workspace/script-agent-startup.sh

@@ -2,7 +2,22 @@
 set -eo pipefail
 
 
+wipe_tmp() {
+  # /tmp is a per-Pod volume (see deployment.tf), not per-container, so it
+  # survives a container restart within a live Pod even though it never used
+  # to: the previous /tmp was part of the container's writable overlay, which
+  # a fresh container instance always got a clean copy of for free. This
+  # restores that property explicitly. It runs before anything else so
+  # nothing has written into /tmp yet this boot, and pipefail/errexit above
+  # mean a failure here aborts this blocking startup script rather than
+  # leaving stale scratch space to accumulate silently - a failed wipe shows
+  # up as a failed agent startup script in the Coder UI, not as a slow leak.
+  echo "Wiping /tmp..."
+  find /tmp -mindepth 1 -delete
+}
+
 main() {
+  wipe_tmp
   if [[ ! -s ~/.bashrc ]]; then
     echo "Setting up starter bash rc scripts from /etc/skel..."
     cp /etc/skel/.bashrc ~/.bashrc