Container-recycle recovery: timeout misclassification, unretried recovery clone, and an ADR that still says the container FS is durable

[debuggingfuture]

Three defects in the container-recycle recovery path that the #127–#134 series left behind, plus a fourth item on what the ADR should say instead. Items 1–3 were surfaced by #126, whose own mechanism (execInWorkspace) is superseded by ensureWorkspace — these are the parts that are still live on main.

1. A repo whose name matches /timed?\s*out|timeout/i turns a container recycle into a non-retryable kill

In packages/runtime-cf/src/sandbox-cf.ts, the working-directory-missing case throws inside the try with a message that interpolates cwd:

throw new Error(
  `working directory '${cwd}' was missing at exec time — the checkout did not survive to this step (container recycled). stderr: ${stderr.slice(0, 200)}`,
);

The catch directly below classifies by regex over that message, with no workspace-missing check ahead of it:

if (/timed?\s*out|timeout/i.test(message)) {
  return new ExecTimeout({ timeoutSec: timeoutSec ?? 0, command: cmd });
}

cwd is /workspace/${repo.split("/").pop() ?? "repo"} — the consumer's own repo name. This is BYOC, so any org's repo reaches it. A repo named request-timeout or timeout-utils makes the regex fire on the path, not on a timeout.

The result is not a cosmetic mislabel. ExecTimeout is absent from RETRY_ON in runs/offload-test.ts, runs/check.ts and runs/oxlint.ts, so rethrowForRetryPolicy (packages/runtime-cf/src/step-runner-cf.ts) wraps it as NonRetryableError and the step dies with no retry — and the retry is the only thing that gives ensureWorkspace's probe a chance to run.

cwd reaches the message twice — once literally, and again inside the appended stderr tail (Failed to change directory to '<cwd>').

No test pins the ordering. packages/runtime-cf/src/sandbox-cf.test.ts:405-430 does assert ExecFailed for a missing working directory, but with cwd: "/workspace/repo" — no timeout substring, so it stays green with the bug present. :432-448 exercises only the isWorkingDirFailure predicate, never the catch.

Fix: classify the workspace-missing case before the timeout regex, on a typed field rather than a message match — #126 adds an optional workspaceMissing to ExecFailed for exactly this, which is the right shape.

2. A recovery clone that fails transiently kills the run

ensureWorkspace re-clones inside the caller's retryable step (packages/core/src/primitives/workspace.ts; call sites in runs/offload-test.ts). A failing clone surfaces as CheckoutFailed, which is not in RETRY_ON — so the same rethrowForRetryPolicy path makes it non-retryable.

That fires in precisely the conditions that caused the recycle. #134 set RETRY_ON to ["ExecFailed", "StepFailed"] without considering CheckoutFailed, and no test covers it.

Unconditional — the StepFailed entry never rescues it. errorTagOf (packages/runtime-cf/src/step-runner-cf.ts:139-149) returns "StepFailed" only when cause === undefined. runEffect attaches the live Effect Cause to the thrown Error, and a CheckoutFailed always carries one — so errorTagOf reads "CheckoutFailed", which is not in RETRY_ON, and the run dies with 0 of its 3 PLATFORM_RETRIES spent.

The asymmetry sharpens it: the initial step("checkout", acquireWorkspace) (runs/offload-test.ts:632) passes no opts, so retryOn === undefined and CF's default retry applies. The first clone is retried; the recovery clone — the one that runs when the container is already unstable — is not.

Fix: RETRY_ON = ["ExecFailed", "StepFailed", "CheckoutFailed"] as const in the three runs, with a test.

3. The ADR still documents the belief that caused this class of bug

specs/adr/0001-cloudflare-workflows-scope.md still states that "the container filesystem is shared state across durable steps". That is the claim #127–#134 were spent disproving, and it remains the written record. The sleepAfter comment in apps/dispatcher/src/sandbox.ts repeats it and credits 10m with buying "durability across normal inter-step gaps" — it narrows the idle window and does nothing when the container dies mid-exec.

REWRITE.md is already correct ("step-scoped non-durable state"), so the correction is confined to the ADR and that comment.

Fix: #126's ADR rewrite is reusable as-is, including its rule 3 and rule 3's scoping — a re-clone restores the tree the spec describes, which is right for a suite, lint or build and wrong for a step reading a tree an earlier step mutated. Applied blindly to self-heal-pr's verify step it returns a clean checkout and passes on unmodified code, converting an infra failure into a wrong green. Those steps need captured bytes (the FileRef work), and the rule should say so.

The ADR must also stop implying re-clone is the only option. It is not, and the alternative is first-party.

4. Name @cloudflare/computer in the ADR as the durable-workspace path

@cloudflare/computer (0.2.0, MIT, published 2026-08-12) is a persistent SQLite-backed virtual filesystem for Durable Objects — workspace.fs looks like node:fs/promises and is durable across DO restarts, backed by the DO's own SQLite storage. Its container backend runs computerd and syncs the DO-side and container-side stores over a capnweb WebSocket. It also ships workspace.git (isomorphic-git operating directly on the VFS, so a clone lands in the durable store with no shell) and R2-backed read-only mounts.

That is a direct answer to the category this whole series has been working around: workspace state that survives a step boundary, without capturing and restoring bytes by hand. It is worth naming next to the FileRef chokepoint in REWRITE.md, which is currently the only route the docs offer for a tree that must survive.

Where it fits, and where it does not. The split is sharp, and it happens to fall along the same line rule 3 already draws:

• The mutated-tree runs are exactly its target. self-heal-pr and refresh-fixtures need an earlier step's edits to survive into a verify step. They are agent-scale working directories — precisely what the package is built for — and today they have no correct answer at all, since a re-clone silently discards the edits.
• offload-test is exactly what it warns against. The README caps a workspace at ~10 GB (shared with the DO), states the container-side filesystem is held in memory — "aim for agent-scale workspaces, not full monorepos" — and routes container access through FUSE. The measured cost: ~2× slower than ext4 and ~3.6× slower than tmpfs overall; an npm install of 854 packages takes 124.7 s vs 63.9 s on ext4; large sequential I/O is far worse (64 MiB copy 852.9 ms, ~39× tmpfs). This repo's own sandbox-cf.ts comments cite a 14 GB target/ on an 18 GB container disk, and stages run up to 35 minutes. It does not fit, and ensureWorkspace's re-clone remains correct there.

It is also explicitly preview software — "APIs are unstable… NOT suitable for production use at this time." So this is a documented option and a tracking item, not an adoption proposal.

Fix: rule 3 currently reads as "re-clone, except for mutated trees, which need FileRef." Make it three-way — re-clone for a spec-described tree, FileRef or @cloudflare/computer for a tree that must carry mutations forward — and record the 10 GB / in-memory / FUSE limits as the reason offload-test stays on re-clone. Otherwise the next person to hit a mutated-tree step reimplements a durable VFS by hand.

Suggested shape

One PR, three commits, no new primitive — execInWorkspace / hydrateWorkspace / workspace().spec are all superseded by ensureWorkspace and should not come along:

1. fix(sandbox): classify a lost workspace before the timeout regex
2. fix(runs): retry a clone that fails mid-recovery
3. docs(adr): the container filesystem is step-scoped, not durable — carrying item 4's three-way rule 3

Item 4 is docs-only in this PR. Actually adopting @cloudflare/computer for the mutated-tree runs is separate work and should not start while it is preview.