feat: add observe-only memory watchdog to the workspace template

.github/workflows/test.yaml

@@ -0,0 +1,43 @@
+---
+# yamllint disable rule:line-length
+name: test
+
+# Repo-local rather than a reusable workflow because ppat/github-workflows has
+# nothing for "execute a test script" - the shellcheck workflow only lints,
+# and lint.yaml's detect-changes/job-per-linter structure exists to share one
+# change-detection matrix across many linters, which a single test job has no
+# need of. This guards process-selection safety in the memory watchdog (which
+# processes count as the editor's, and which of those may be signalled) via a
+# fixture suite, not a linter, so it does not belong in lint.yaml.
+
+on:
+  pull_request:
+    paths:
+    - .github/workflows/test.yaml
+    - templates/kubernetes/homelab-workspace/script-memory-watchdog*.sh
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+jobs:
+  watchdog:
+    runs-on: ubuntu-24.04
+    timeout-minutes: 5
+    steps:
+    - name: Checkout
+      uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+      with:
+        fetch-depth: 1
+        persist-credentials: false
+        ref: ${{ github.head_ref || github.ref }}
+
+    - name: Run memory watchdog fixture tests
+      # Exits non-zero when any assertion fails, which is what fails the build.
+      # Needs nothing but bash and a writable TMPDIR: builds its fixtures under
+      # mktemp -d, touches no network, finishes in a couple of seconds.
+      run: ./templates/kubernetes/homelab-workspace/script-memory-watchdog-test.sh

CLAUDE.md

@@ -32,6 +32,12 @@ tflint --config=../../../.tflint.hcl
 
 CI (`.github/workflows/lint.yaml`) runs the same checks per file-type via reusable workflows in `ppat/github-workflows`, scoped to changed files on PRs, or everything on `workflow_dispatch`/schedule.
 
+A separate workflow, `.github/workflows/test.yaml`, runs the one thing here that's a test rather than a linter: its `watchdog` job runs `script-memory-watchdog-test.sh` and fails the build on a failed assertion. It's repo-local rather than a reusable workflow because `ppat/github-workflows` has nothing for "execute a test script", and the suite needs only bash and a writable `TMPDIR`:
+
+```bash
+./templates/kubernetes/homelab-workspace/script-memory-watchdog-test.sh
+```
+
 There is no local way to build/publish the image or push the Coder template — see [TESTING.md](TESTING.md) for how a change actually gets exercised (including the `test_mode` flow), and the **Release flow** section below for how it ships for real.
 
 ## Commit messages
@@ -65,8 +71,11 @@ Quick orientation map — for what each piece is *for* and the decisions behind
 | `coder-agent.tf` | `coder_agent` resource: startup script, `coder stat` metadata |
 | `deployment.tf` / `configmap.tf` | Kubernetes Pod spec, volumes, ConfigMap |
 | `env.tf` | `coder_env` resources exposed to the agent |
+| `scripts.tf` | `coder_script` resources — the memory watchdog daemon and the weekly `vscode-server` GC schedule |
 | `variables.tf` | `workspace_image`, `test_mode` — both supplied by the release workflow |
 | `script-agent-startup.sh` / `script-prepare-workspace.sh` | Scripts run on agent/workspace startup |
+| `script-memory-watchdog.sh` | Userspace memory watchdog — see [DESIGN.md](DESIGN.md#design-tensions-and-decisions). **Defaults to observe-only mode**: it measures and logs, and sets no limits and sends no signals unless the `memory_watchdog_mode` parameter is switched to `enforce` |
+| `script-memory-watchdog-test.sh` | Fixture tests for the watchdog's arithmetic and process selection. Run by hand (`./script-memory-watchdog-test.sh`) and by the `watchdog` job in `.github/workflows/test.yaml` |
 
 **Image** (`images/homelab-workspace/Dockerfile`): three build stages — `base` (minimal bootstrap deps) → `system-base` (`unminimize` + full interactive toolset) → final stage (env vars into `/etc/environment`, fixed-UID/GID `coder` user, `USER coder`). All `apt`-touching `RUN` steps use BuildKit cache mounts — match that pattern when adding packages.
 
@@ -78,7 +87,10 @@ Things that look arbitrary in the code but are load-bearing (full reasoning in [
 
 - `deployment.tf`'s `system` volume is an `empty_dir`, rebuilt from the image on every pod start — a fix to anything under `/usr`, `/etc`, `/var` must go in the image or the init script, not be treated as a one-time patch.
 - The Dockerfile writes shared env vars to `/etc/environment` rather than using `ENV`, because `PATH` needs to be extended by a script running after the image is built, not fixed at build time.
-- `parameters.tf`'s `local.validated_*` regex allowlist is the only thing stopping `system_packages`/`preferred_nodes` from injecting shell metacharacters into the init container — any new list-type parameter must go through the same decode-then-validate step.
+- `parameters.tf`'s `local.validated_*` allowlist is the only thing stopping `system_packages`/`preferred_nodes` from injecting shell metacharacters into the init container — any new parameter whose value reaches a shell must go through the same validate-then-use step. `memory_watchdog_mode` follows it too: Coder constrains the value server-side, but it is the single switch deciding whether the watchdog may signal processes, so an unrecognised value falls back to the inert `observe` rather than being passed through.
+- `script-memory-watchdog.sh` computes headroom as `memory.max − U`, where `U` sums only the *unreclaimable* fields of `memory.stat` (`anon`, `shmem`, `unevictable`, `slab_unreclaimable`, `kernel_stack`, `pagetables`, `sec_pagetables`, `percpu`, `sock`). Do not "simplify" it to `memory.current` or to `memory.stat`'s `kernel` roll-up: on the live pod those read 92% and 42% of the limit while true `U` is 23%, so either substitution makes the watchdog fire permanently on an idle container. Its thresholds are absolute bytes, not percentages, because the page cache a workload needs is a property of the workload rather than of the limit — which also means the 4 GiB memory parameter needs its own numbers, and the script logs a warning when it detects that mismatch.
+- **The watchdog decides what is a VS Code process by executable path — `argv[0]` under `~/.vscode-server/` — never by whether something "is node".** A provisioned workspace has two unrelated node installations: VS Code's bundled one under `~/.vscode-server/cli/servers/Stable-<commit>/server/`, and mise's on `PATH`, which is what repo tooling and the operator's agent sessions run on. (There is no `/usr/bin/node`, and nothing named `node` on `PATH` at all without dotfiles.) Matching on `comm`, on a basename, or on a loose cmdline substring would classify an agent session spawned by an extension — a child of the extension host, and *not* under ptyHost — as a sheddable editor helper. `comm` in particular is `MainThread` for every node process in a real tree, never `node`, because V8 renames its main thread; nothing may key off it. `script-memory-watchdog-test.sh` asserts this three ways, each paired with the mutation that flips it.
+- The watchdog never signals anything in the `--type=ptyHost` subtree. Tree membership alone is *not* a safe kill criterion: tmux sessions and agent runs started from a VS Code integrated terminal are descendants of the server tree through ptyHost, so a tree-wide kill would take the operator's work with it. The exclusion is asserted, together with the mutation that must flip it, in `script-memory-watchdog-test.sh`.
 - Adding a package/tool has three possible homes, and picking the wrong one is a real mistake, not a style choice — route by the rule in [DESIGN.md](DESIGN.md#where-the-workspace-environment-comes-from): universal + stable → image (`Dockerfile`); occasionally-needed + apt-only + too heavy to bake in → the template's `system_packages` parameter; personal, fast-moving, or not an apt package → the operator's dotfiles (a *different* repo — see below), never this one.
 - `deployment.tf`'s Deployment `metadata.name` (`local.workload_name` in `main.tf`) is not cosmetic: the cluster's Prometheus resolves pod → ReplicaSet → Deployment via an existing `kube_pod_owner` recording rule and exposes the result as a `workload` label with no other join needed, so whatever this Deployment is named *is* the identity CPU/memory/PSI/OOM metrics get attributed to. Don't revert it to an opaque identifier (e.g. the workspace UUID) without re-breaking that attribution — see [DESIGN.md](DESIGN.md#design-tensions-and-decisions).
 

DESIGN.md

@@ -69,6 +69,16 @@ The rule that ties the layers together: a package or tool belongs in the *lowest
 - *The prefix is `coder-workspace-`, not just `coder-`*, matching the `app.kubernetes.io/part-of` value already used in `main.tf`'s `common_labels`. A bare `coder-` prefix isn't enough to unambiguously mean "workspace": the same Kubernetes namespace also holds the `coder` control-plane Deployment itself and other `coder`-prefixed infra (e.g. a CloudNativePG cluster named `coder-db-<date>`) that a naive `workload=~"coder-.+"` match would also catch.
 - *Renaming a workspace already relocates its home directory* (the `home` volume's `sub_path` is `data.coder_workspace.me.name`), so coupling the Deployment name to the workspace name too doesn't introduce a new class of rename hazard — it's already priced in. A rename recreates the Deployment (the pod restarts anyway) and needs a fresh `coder-workspace-<owner>-<new-name>` home subdirectory, exactly as it needed a fresh `sub_path` before this change.
 
+**A userspace memory watchdog, because the kernel's own mechanisms are out of reach.** The workspace pod has a hard memory limit, and a memory-hungry editor server can walk it into a cgroup OOM. The kill itself would be tolerable; its blast radius is not. `memory.oom.group` is set to `1` by the kubelet, so a cgroup OOM kills *every process in the container as a group* — the IDE, every tmux session, and every long-running agent, together. That also rules out the usual mitigation: with `oom.group = 1`, nudging `oom_score_adj` cannot make one process die instead of all of them, because there is no victim selection left to influence.
+
+The two obvious fixes are both unreachable from inside this container. Throttling with `memory.high`, or confining the editor to a child cgroup, would need a writable `/sys/fs/cgroup` — but it is mounted read-only, `cgroup.subtree_control` is empty, the cgroup namespace is private, and the workspace user has no capabilities. Getting either would mean `privileged: true` or a read-write host mount of `/sys/fs/cgroup`, which is exactly what *Unprivileged by default* above exists to prevent. Raising the limit was also considered and rejected: it moves the wall rather than removing it, and the pod is already large for a single-operator homelab.
+
+What is left is to never reach the limit in the first place, which is what [`script-memory-watchdog.sh`](templates/kubernetes/homelab-workspace/script-memory-watchdog.sh) does. It samples how much genuinely unreclaimable memory the cgroup holds, and — as the editor's helper processes grow — lowers their *soft* `RLIMIT_DATA` so that one of them fails its own allocation and restarts, instead of the kernel taking down the whole container. Lowering another same-uid process's soft limit needs no privilege, and leaving the hard limit alone means any shell that inherits the ceiling can lift it again.
+
+Which processes it may touch is settled by executable path, not by name or role heuristics: only a process whose own binary lives under `~/.vscode-server` counts as the editor's. That boundary is doing more work than it appears to. A provisioned workspace carries two unrelated node installations — VS Code's bundled one, which arrives with the server download, and the operator's from mise, which is what repo tooling and long-running agent sessions run on — and a rule that asked "is this node" instead of "whose binary is this" would classify an agent session spawned by an extension as an editor helper and shed it. The watchdog exists to stop the operator's work being collateral damage, so a detection rule that makes it the target would be a self-defeating one. Terminal descendants are excluded on top of that, by excising the editor's pty host and everything beneath it.
+
+The trade is that this is a userspace daemon in a pod with no supervisor, doing something the kernel would do better if it were allowed to. It is therefore built to be deletable in one step if the constraint ever lifts, and it defaults to an observe-only mode — measuring and logging, changing nothing — so that the thresholds at which it acts get set from a week of this workload's own data rather than from a guess. That default is a workspace parameter rather than a constant, because the thresholds are absolute byte counts sized for the larger pod, and the same setting that suits it sits permanently near the first tier on a smaller one. Its measurement deliberately disagrees with every stock memory reading, including Coder's own: page cache and reclaimable slab make this pod look near death while it is idle, and a watchdog that believed them would fire constantly. That disagreement is the point of the thing, so the honest number is surfaced next to the misleading one in the workspace UI rather than replacing it.
+
 ## Outcomes targeted
 
 - One operator can keep dependencies current and ship template/image changes at low ongoing effort, without a fleet of environments to maintain.

templates/kubernetes/homelab-workspace/coder-agent.tf

@@ -49,4 +49,16 @@ resource "coder_agent" "main" {
     interval = 60
     timeout  = 1
   }
+  metadata {
+    display_name = "Memory Headroom"
+    key          = "6_memory_headroom"
+    # Published by the memory watchdog (see scripts.tf). This is the honest
+    # number: bytes left before something in the pod has to die. "Memory Usage"
+    # above reads ~63% on a pod whose true unreclaimable share is ~23%, because
+    # it counts page cache the kernel will hand straight back.
+    script   = "cat $${HOME}/.local/state/vscode-memory-watchdog/headroom 2>/dev/null || echo '-'"
+    interval = 60
+    timeout  = 1
+  }
+
 }

templates/kubernetes/homelab-workspace/configmap.tf

@@ -9,6 +9,7 @@ resource "kubernetes_config_map_v1" "workspace_scripts" {
 
   data = {
     agent_startup_script     = file("${path.cwd}/script-agent-startup.sh")
+    memory_watchdog_script   = file("${path.cwd}/script-memory-watchdog.sh")
     prepare_workspace_script = file("${path.cwd}/script-prepare-workspace.sh")
     workspace_init_script    = coder_agent.main.init_script
   }

templates/kubernetes/homelab-workspace/deployment.tf

@@ -131,6 +131,11 @@ resource "kubernetes_deployment_v1" "deployment" {
             name       = "coder-scripts"
             sub_path   = "agent_startup_script"
           }
+          volume_mount {
+            mount_path = "/memory-watchdog.sh"
+            name       = "coder-scripts"
+            sub_path   = "memory_watchdog_script"
+          }
           volume_mount {
             mount_path = "/workspace-init.sh"
             name       = "coder-scripts"

templates/kubernetes/homelab-workspace/env.tf

@@ -3,3 +3,20 @@ resource "coder_env" "welcome_message" {
   name     = "HOMEBREW_PREFIX"
   value    = local.homebrew_directory
 }
+
+# The switch that arms the memory watchdog. "observe" measures, publishes
+# headroom and logs what it would have done; "enforce" additionally sets
+# RLIMIT_DATA ceilings and sheds load.
+#
+# Set from a mutable workspace parameter rather than hardcoded here, because the
+# right value is a per-workspace judgement: the tier thresholds are absolute
+# bytes sized for an 8 GiB pod, so the same setting that is right there sits
+# permanently near L1 on a 4 GiB one. It defaults to "observe" and should stay
+# there until the ceilings and thresholds have been set from the calibration data
+# the watchdog collects - too low kills a healthy extension host mid-edit, too
+# high makes the mechanism inert.
+resource "coder_env" "memory_watchdog_mode" {
+  agent_id = coder_agent.main.id
+  name     = "WATCHDOG_MODE"
+  value    = local.validated_watchdog_mode
+}

templates/kubernetes/homelab-workspace/parameters.tf

@@ -61,8 +61,39 @@ data "coder_parameter" "system_packages" {
   type         = "list(string)"
 }
 
+data "coder_parameter" "memory_watchdog_mode" {
+  name = "memory_watchdog_mode"
+
+  default      = "observe"
+  display_name = "Memory Watchdog"
+  description  = "What the memory watchdog is allowed to do when the pod runs low on unreclaimable-memory headroom"
+  icon         = "/icon/memory.svg"
+  mutable      = true
+
+  option {
+    name        = "Observe only"
+    value       = "observe"
+    description = "Measure, publish headroom and log what it would have done. Sets no limits and sends no signals"
+  }
+  option {
+    name        = "Enforce"
+    value       = "enforce"
+    description = "Also cap helper processes with RLIMIT_DATA and shed load as headroom falls. Do not enable before the thresholds have been set from calibration data"
+  }
+}
+
 
 locals {
+  # Coder already constrains this to the two option values server-side, but it
+  # reaches the agent as an environment variable and from there a shell, and it
+  # is the single switch that decides whether the watchdog may signal processes.
+  # So it goes through the same validate-then-use step as the list parameters
+  # below, and anything unrecognised falls back to the inert mode rather than to
+  # whatever was supplied.
+  validated_watchdog_mode = contains(
+    ["observe", "enforce"], data.coder_parameter.memory_watchdog_mode.value
+  ) ? data.coder_parameter.memory_watchdog_mode.value : "observe"
+
   validated_system_packages = (data.coder_parameter.system_packages.value != "") ? [
     for str in jsondecode(data.coder_parameter.system_packages.value) :
     str if length(regexall("[^a-zA-Z0-9-]", str)) == 0