feat: add observe-only memory watchdog to the workspace template

[ppat]

What this adds

A userspace memory watchdog for the workspace pod, defaulting to observe-only: it measures, publishes headroom, and logs what it would have done. Switching it to enforce is a per-workspace parameter.

File	Role
script-memory-watchdog.sh	The watchdog. Pure bash + /proc + /sys/fs/cgroup, plus /usr/bin/sleep and (enforce mode only) /usr/bin/prlimit
script-memory-watchdog-test.sh	Fixture tests, transcribed from a live tree. Run by hand and by the watchdog job in .github/workflows/test.yaml
scripts.tf	coder_script for the watchdog daemon, and the weekly vscode-server GC schedule
parameters.tf / env.tf	memory_watchdog_mode parameter → validated → WATCHDOG_MODE
coder-agent.tf	Memory Headroom metadata block
configmap.tf / deployment.tf	Mount the script at /memory-watchdog.sh
.github/workflows/test.yaml	Repo-local watchdog job — deliberately not in lint.yaml: this guards process-selection safety, not style, and would be too easy to prune by anyone tidying linters

Why it can't be a cgroup limit

Verified in-pod: memory.oom.group = 1, so a cgroup OOM kills every process in the container as a group — the IDE, every tmux session and every long-running agent, together. oom_score_adj cannot pick a victim for the same reason.

memory.high or a child cgroup would fix this properly, but /sys/fs/cgroup is mounted ro, cgroup.subtree_control is empty, the cgroup namespace is private, and the container runs as an unprivileged uid with CapEff = 0. Getting write access means privileged: true or a read-write host mount, which DESIGN.md's Unprivileged by default rules out. Raising the pod limit moves the wall rather than removing it.

That leaves one strategy: never reach memory.max. Lowering another same-uid process's soft RLIMIT_DATA needs no capability, and since Linux 4.7 it bounds private writable anonymous mmap — so it caps V8 heap, ArrayBuffers and native malloc while leaving the PROT_NONE pointer-compression cage uncounted. The hard limit is never touched, so an inheriting shell restores itself with ulimit -d unlimited.

Three things that are load-bearing

The measurement disagrees with every stock reading, and that is the point. Headroom is memory.max − U, where U sums only the unreclaimable fields of memory.stat. On the 8 GiB pod at rest, memory.current reads 92%, the memory.stat kernel roll-up 42%, coder stat mem 63% — and true U is 23%, with memory.pressure full avg10 at 0.00 throughout. A memory.current > 85% trigger would fire permanently on an idle container. Thresholds are absolute bytes rather than percentages, because the page cache a workload needs is a property of the workload, not of the limit.

What counts as a VS Code process is decided by executable path, never by "is this node". A provisioned workspace has two unrelated node installations: VS Code's bundled one under ~/.vscode-server/cli/servers/Stable-<commit>/server/, and mise's on PATH, which is what repo tooling and the operator's agent sessions run on. There is no /usr/bin/node and nothing named node on PATH without dotfiles. A process is eligible only if its own argv[0] lives under ~/.vscode-server/. Matching on comm, on a basename, or on a cmdline substring would classify an agent session spawned by an extension — a child of the extension host, and not under ptyHost — as a sheddable editor helper, which is the exact outcome this design exists to prevent. comm is MainThread for every node process in a real tree, never node, because V8 renames its main thread; nothing keys off it.

Tree membership is not a safe kill criterion. tmux sessions and agent runs started from a VS Code integrated terminal are descendants of the server tree, via --type=ptyHost. The watchdog excises that fork and its entire subtree, in every server tree it finds — --reconnection-grace-time keeps a disconnected server alive for eight hours, so two concurrent trees is ordinary, and excision only happens inside a tree that was discovered. It independently refuses to signal coder agent, tmux, claude, chezmoi, pid 1 or itself; that name-based net is a second line of defence, not the primary one.

Why observe-only by default

The RLIMIT_DATA ceilings and the tier thresholds were derived from role and an 8 GiB budget, not from measurement. Too low silently kills a healthy extension host mid-edit; too high makes the mechanism inert. The watchdog writes a calibration CSV (~/.local/state/vscode-memory-watchdog/calibration.csv) with U and every component, PSI, and refault/pgscan rates. Set the numbers from a week of that, then switch the parameter.

The mode is per-workspace rather than per-template because the thresholds are absolute byte counts: a 4 GiB workspace sits permanently near L1 against numbers sized for 8 GiB. The watchdog logs a warning when it detects that mismatch rather than silently clamping.

Verification

Fixture tests — 94 assertions

Transcribed from a captured live tree, not hand-written. Each negative assertion is paired with the mutation that must flip it:

Mutation	Failures
Restore the previous comm-preferring root selection	16
Remove the argv[0]-under-.vscode-server rule	3
Key roles on the joined cmdline instead of argv[0]	1
Remove the ptyHost subtree excision	3

CI proof that the harness can fail the build: this has now been demonstrated twice, most recently against test.yaml's watchdog job specifically — a fixture assertion was deliberately broken, the job went red on real CI, and the branch was restored and went green again on the next push.

Live workspace — durable evidence on the test Coder workspace

The table below is the only claim this PR makes about live behavior. It is backed by artifacts left on disk on test, not by a summary — everything in it can be re-checked by reading the files listed, without re-running anything.

A real VS Code client connection to test wasn't available in the session that produced this evidence, so the server tree itself is fabricated: real OS processes with argv/comm shaped to match a real VS Code server tree exactly (including the comm = MainThread V8 rename the design calls load-bearing), driven by the real, unmodified script-memory-watchdog.sh. Everything the watchdog does — reading the real cgroup, discovering the real process tree, calling real kill()/prlimit(2) — is real; only the "VS Code" side of the tree is a stand-in. Full disclosure of exactly what's real vs. fabricated, and one methodology bug this exercise found and fixed along the way (a fabrication path that accidentally collided with the watchdog's own self-protection rule and silently invalidated the first attempt), is in the evidence itself.

Left in place on test, inspectable without re-running anything:

• ~/watchdog-live-evidence/README.txt — what's real vs. fabricated, and why
• ~/watchdog-live-evidence/SUMMARY.txt — per-run findings
• ~/watchdog-live-evidence/runs/{run1-observe-gradual,run2-enforce-gradual,run3-enforce-l4-jump}/ — before/after ps, before/after /proc/*/limits, spawn logs, and a snapshot of the watchdog's own state dir after each run
• ~/.local/state/vscode-memory-watchdog/{actions.log,headroom,calibration.csv} — the watchdog's own live, cumulative state

Check	Result	Basis
Root/tree discovery on a real process tree	Correct pid, comm = MainThread, tree = 8 procs	live
Observe mode	Ceilings and a tier-3 signal both logged with [observe]; /proc/*/limits and process liveness confirmed unchanged before and after	live
Enforce ceilings	Soft RLIMIT_DATA set per role via real prlimit(2), hard limit unlimited in every case, verified via /proc/*/limits after the run	live
Enforce L3	Real SIGTERM to the extension-host role; confirmed genuinely killed (zombie state, RSS 0), real memory freed (cgroup memory.current dropped accordingly)	live
Enforce L4	Real SIGTERM to both server roots plus the file watcher, heaviest RSS first, no role restriction, on the first sample (no debounce, by design)	live
ptyHost and everything beneath it — the memory hog included	Never signalled, never mentioned in the log, confirmed alive after every run including the L4 run	live
A non-.vscode-server process inside the tree but outside ptyHost (an extension-spawned agent stand-in)	Protected by path alone, confirmed alive after the L4 run	live
memory.events.oom_group_kill	0 before and after every run, including the run where real cgroup U reached 3.33 GiB of a 4 GiB limit	live
L2 (PSI/refault-corroborated shedding)	Real PSI and refault rate stayed 0 throughout; L2's corroboration gate was never satisfied	not exercised live — matches the known gap below
SIGKILL escalation (2 s grace after SIGTERM)	Not exercised — the fabricated processes have no signal handler, so SIGTERM alone always sufficed	not exercised live
A genuine VS Code client, real IPC-spawned extensionHost/ptyHost, real respawn-on-kill	Not available in-session	not tested

Also found by running it for real

• Negative headroom projections printed as -27.-79 GiB.
• Observe mode re-logged every proposed ceiling on every cycle, burying the tier transitions.
• Two comments asserted python3, flock and awk were unavailable in the image. All three are present; the real constraint is only that PATH is brew-shadowed, so absolute /usr/bin/... paths are required. The bash-only implementation is kept — it forks nothing per scan — but the stated reason was wrong.

Other

terraform fmt -check, terraform validate, tflint, pre-commit run --all-files, and the dry-run release pipeline all clean.

Known gaps

• The thresholds and ceilings are still uncalibrated. That is why the default is observe. The live runs above used values scaled for a 4 GiB pod to make the ladder reachable, which is not evidence that the shipped numbers are right.
• L2 is hard to reach. It never fired under sustained low headroom without page-cache thrashing, and a fast allocator vaults past it to L3 via the projection term. That may be correct — the pod was not suffering — but it means the tier is rarer than the design implies.
• Language-server role matching is a prefix/suffix guess. There is no common launch convention. A wrong guess is contained to one restartable helper, since roles are only consulted inside a server tree, outside the ptyHost subtree, and only for binaries under ~/.vscode-server/.
• Native extension helpers get no ceiling. RLIMIT_DATA on a Go or Rust runtime is an abrupt abort rather than the recoverable V8 heap OOM the ceilings are premised on, and there is no measured number to pick. They are shed at L2 instead.
• vscode-server-gc does not exist yet. The cron resource is a no-op until the script lands in the dotfiles repo.

[homelab-workflows-bot]

🎉 This PR is included in version 2.25.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀