fix(runtime): sandbox budget = script CPU-time, not wall clock (ADR-0102 D1, #3295)

.changeset/sandbox-cpu-budget.md

@@ -0,0 +1,39 @@
+---
+'@objectstack/runtime': minor
+'@objectstack/types': minor
+---
+
+feat(runtime): sandbox budget is script CPU-time, not wall clock (ADR-0102 D1, #3295)
+
+The QuickJS sandbox now meters each hook/action invocation against how much
+**VM-active (CPU) time** the body burns, not wall clock. Idle host-await time and
+a nested hook's own execution (which runs host-side while the caller's VM is
+parked) are no longer charged to the caller — so a slow/loaded host or a deep
+nested-write chain can't trip the budget while a script is merely waiting (the
+root cause of the #3259 CI flake). A separate, generous **wall-clock ceiling**
+(default 30s, `max(ceiling, cpuBudget)`) remains as the backstop for a body stuck
+on a host call that never settles.
+
+What changes for consumers (behaviour, not API signatures):
+
+- **Meaning of the timeout knobs.** `body.timeoutMs`, the `hookTimeoutMs` /
+  `actionTimeoutMs` runner options, and `OS_SANDBOX_HOOK_TIMEOUT_MS` /
+  `OS_SANDBOX_ACTION_TIMEOUT_MS` keep their **names, defaults (250ms / 5000ms),
+  and precedence** — but now bound CPU-time instead of wall-clock. In practice
+  this only *loosens* legitimate slow/nested work; a runaway synchronous script
+  is still cut at the same budget.
+- **Error messages.** `exceeded timeout of Nms` → either `exceeded CPU budget of
+  Nms` (script burned its CPU budget) or `exceeded wall-clock ceiling of Nms
+  while awaiting host calls` (stuck on a never-settling host call). Update any
+  code/tests matching the old string.
+
+New knobs (additive):
+
+- `QuickJSScriptRunner` option `wallCeilingMs` and env `OS_SANDBOX_WALL_CEILING_MS`
+  — tune the wall ceiling (explicit option › env › 30s).
+- `resolveSandboxTimeoutMs` (`@objectstack/types`) gains a `'wallCeiling'` kind.
+
+Also fixes a latent init bug in the new accounting where the interrupt handler
+could fire during `installCtx` and corrupt ctx marshalling. The nested-write
+integration suites now run at the stock 250ms budget (previously forced to 10s),
+which is itself the regression guard for the nested-charging fix.

content/docs/automation/hook-bodies.mdx

@@ -129,11 +129,11 @@ Hooks mutate `ctx.input`/`ctx.result`; actions return their output value explici
 
 The sandbox engine is **`quickjs-emscripten`** — pure-WASM, runs on every JS host. We considered `isolated-vm` but its native dependency disqualifies edge targets. The choice is hidden behind the `ScriptRunner` interface in `packages/runtime/src/sandbox/`, so a node-only deployment can swap in a faster engine later without touching call sites.
 
-Per-invocation timeouts default to **250ms** for hooks and **5000ms** for actions; per-invocation memory caps at **32 MB**. Both are overridable per body.
+Per-invocation budgets default to **250ms** (hooks) / **5000ms** (actions) of **script CPU time** — VM-active time, *not* wall clock (ADR-0102): time spent awaiting host calls, or running a nested hook, is not charged. A separate **30s wall-clock ceiling** backstops a body stuck on a host call that never settles. Per-invocation memory caps at **32 MB**. All are overridable per body and deployment-wide via `OS_SANDBOX_HOOK_TIMEOUT_MS` / `OS_SANDBOX_ACTION_TIMEOUT_MS` / `OS_SANDBOX_WALL_CEILING_MS`.
 
 ### Nested cross-object writes
 
-A body may write *other* objects — e.g. `await ctx.api.object('parent').update({ ... })` from a child's `afterInsert`/`afterUpdate` (requires `api.write`). The target's own hooks fire too: the nested write runs in a **fresh sandbox VM** while the calling body is suspended, and this composes to any depth. This is the natural "when a child changes, roll the total up to the parent" automation — it does **not** need a denormalized, hand-maintained mirror field. The 250ms default suits a single quick write; give a deeper or slower rollup chain headroom by declaring a larger `timeoutMs` on the outer body (up to 30_000ms), so the whole chain settles within budget.
+A body may write *other* objects — e.g. `await ctx.api.object('parent').update({ ... })` from a child's `afterInsert`/`afterUpdate` (requires `api.write`). The target's own hooks fire too: the nested write runs in a **fresh sandbox VM** while the calling body is suspended, and this composes to any depth. This is the natural "when a child changes, roll the total up to the parent" automation — it does **not** need a denormalized, hand-maintained mirror field. Because each body's budget is **CPU time** (ADR-0102), the caller is **not** charged for the nested write's own run — so the stock 250ms default comfortably covers deep rollup chains, and you rarely need to raise `timeoutMs` (the spec still permits up to 30_000ms for a genuinely CPU-heavy body).
 
 ## L3 — Compiled modules (intentionally disabled)
 

content/docs/deployment/environment-variables.mdx

@@ -331,8 +331,9 @@ the hosted ObjectOS Cloud control plane.
 | `OS_ENV_CACHE_TTL_MS` | number | `300000` | Env-record cache TTL in ms. |
 | `OS_ARTIFACT_CACHE_TTL_MS` | number | `300000` | Artifact-record cache TTL in ms. |
 | `OS_ARTIFACT_FETCH_TIMEOUT_MS` | number | `60000` | Timeout for remote artifact fetches. Only a positive value is honored — an unset, non-numeric, or `0` value falls back to the 60s default (pass `fetchTimeoutMs: 0` in `artifactSource` config to actually disable the timeout). |
-| `OS_SANDBOX_HOOK_TIMEOUT_MS` | number | `250` | Default wall-clock budget for a sandboxed **hook** body (QuickJS). Each invocation compiles a fresh WASM module and a nested hook compiles another inside the parent's budget, so a loaded/slow host (e.g. an oversubscribed CI runner) may need a larger floor. Only a positive integer is honored; unset / non-numeric / non-positive keeps the 250ms default. A hook body's own declared `timeoutMs` still wins over this. |
-| `OS_SANDBOX_ACTION_TIMEOUT_MS` | number | `5000` | Default wall-clock budget for a sandboxed **action** body (QuickJS). Same resolution rules as the hook variant above (positive integer only; an action body's own `timeoutMs` still wins). |
+| `OS_SANDBOX_HOOK_TIMEOUT_MS` | number | `250` | Default **CPU-time** budget for a sandboxed **hook** body (QuickJS, ADR-0102): how much *VM-active* time a body may burn — idle host-await time and a nested hook's own run are NOT charged. A loaded/slow host rarely needs to raise this now (it is not wall-clock), but the knob remains. Only a positive integer is honored; unset / non-numeric / non-positive keeps the 250ms default. A hook body's own declared `timeoutMs` still wins over this. |
+| `OS_SANDBOX_ACTION_TIMEOUT_MS` | number | `5000` | Default **CPU-time** budget for a sandboxed **action** body (QuickJS). Same resolution rules as the hook variant above (positive integer only; an action body's own `timeoutMs` still wins). |
+| `OS_SANDBOX_WALL_CEILING_MS` | number | `30000` | Wall-clock ceiling (ADR-0102) — the backstop that cuts a hook/action body stuck on a host call that never settles (which burns no CPU, so the CPU budget alone would never fire). The effective ceiling is `max(this, cpuBudget)`, so it can never cut a body still inside its CPU budget. Positive integer only; unset keeps 30s. |
 | `OS_INLINE_SEED_BUDGET_MS` | number | `8000` | Time budget for synchronous seed execution at boot before deferring to a worker. |
 | `OS_TENANT_AUDIT` | flag | `1` | Set to `0` to silence the tenant-isolation audit warnings emitted by the SQL driver. |
 

docs/adr/0102-sandbox-cpu-budget-and-engine-variant.md

@@ -1,6 +1,6 @@
 # ADR-0102: Sandbox Execution Budget & Engine Variant — CPU-time budget with a wall ceiling; per-invocation sync WASM modules, precompiled
 
-**Status**: Proposed (2026-07-19) — spec'd and API-verified in framework#3275; lands as Phase 1 #3295 (D1), Phase 2 #3296 (D2), Phase 3 #3297 (D3). Flips to **Accepted** when Phase 1 merges; D3 is explicitly deferrable without weakening D1/D2.
+**Status**: Accepted (2026-07-19) — **D1 landed** with Phase 1 (#3295): CPU-time budget + wall ceiling in `QuickJSScriptRunner`, nested-write integration tests now green at the stock 250ms budget. D2 (#3296, drop asyncify) and D3 (#3297, precompile) remain **Proposed**; D3 is explicitly deferrable without weakening D1/D2.
 **Deciders**: ObjectStack Protocol Architects
 **Builds on**: the #1867 sandbox redesign (deferred-promise host calls + pump loop) — the mechanism that both retired asyncify's only remaining justification (D2) and created the discrete VM-entry points D1 meters. No earlier ADR covers the sandbox runner; prior art is code history (#3232 honor body `timeoutMs`, #3264 test de-flake, #3270 env-overridable defaults).
 **Tracking**: framework#3275 (implementation-ready spec) · #3295 / #3296 / #3297 (phases) · #3259 (motivating CI flake, closed by the #3270 stopgap)
@@ -41,7 +41,7 @@ The value resolved by `resolveTimeout()` (body `timeoutMs` › enclosing `opts.t
 
 Consequence for the knobs: names, defaults (250ms hooks / 5000ms actions), and precedence are unchanged; only the *dimension* changes. Raising the CPU budget no longer trades away host-latency tolerance, and vice versa. This restores meaning to the 250ms default on any hardware — which is what lets CI drop its 10s floor (a stopgap that today also loosens the runaway bound for every test).
 
-Open point to settle at Phase 1 review: whether the ceiling also gets `OS_SANDBOX_WALL_CEILING_MS` (recommended — symmetric with the CPU knobs, wanted by slow-IO deployments) or stays constructor-only.
+Settled (Phase 1): the ceiling is both a constructor option (`wallCeilingMs`, needed for fast tests) AND env-tunable via `OS_SANDBOX_WALL_CEILING_MS` — symmetric with the CPU knobs and wanted by slow-IO deployments. Precedence: explicit option › env › built-in 30s.
 
 ### D2 — Drop asyncify: per-invocation *sync* modules (isolation unchanged)
 

packages/runtime/src/sandbox/nested-write-real-sqlite.integration.test.ts

@@ -86,9 +86,9 @@ describe('#1867 nested cross-object write — REAL SqlDriver (better-sqlite3, on
     engine.registerDriver(driver, true);
     await engine.init();
     for (const o of [EXPENSE_REPORT, EXPENSE_LINE]) engine.registry.registerObject(o as any);
-    // Generous hook budget — the subject is the nested-write path on a real
-    // driver, not the 250ms default (see nested-write.integration.test.ts).
-    engine.setDefaultBodyRunner(hookBodyRunnerFactory(new QuickJSScriptRunner({ hookTimeoutMs: 10_000 }), { ql: engine, appId: 'expense' }));
+    // Runs at the STOCK 250ms CPU-time budget (ADR-0102 D1): idle host-await time
+    // and the nested rollup's own work are not charged (see nested-write.integration.test.ts).
+    engine.setDefaultBodyRunner(hookBodyRunnerFactory(new QuickJSScriptRunner(), { ql: engine, appId: 'expense' }));
     bindHooksToEngine(engine, [ROLLUP_HOOK as any], { packageId: 'expense' });
     return engine;
   }

packages/runtime/src/sandbox/nested-write.integration.test.ts

@@ -90,13 +90,13 @@ describe('#1867 nested cross-object write from a hook (real engine + sandbox)',
     engine.registerDriver(driver, true);
     await engine.init();
     for (const o of [parent, child]) engine.registry.registerObject(o as any);
-    // Generous hook budget: the subject here is nested-write correctness, not
-    // the 250ms default. Each hook invocation compiles a fresh WASM module and
-    // the nested parent hook compiles another inside the child's budget — on a
-    // loaded CI machine that overhead alone blew 250ms ("hook
-    // 'rollup_parent_total' exceeded timeout of 250ms").
+    // Runs at the STOCK 250ms budget on purpose (ADR-0102 D1): the budget is now
+    // CPU-time, so the child hook parking on its nested `parent.update(...)` — and
+    // the parent hook that fires host-side inside it — is not charged to the
+    // child. If this suite ever needs a generous budget again, that is a
+    // regression in CPU accounting, not an inherent cost of nested writes.
     engine.setDefaultBodyRunner(
-      hookBodyRunnerFactory(new QuickJSScriptRunner({ hookTimeoutMs: 10_000 }), { ql: engine, appId: 'test' }),
+      hookBodyRunnerFactory(new QuickJSScriptRunner(), { ql: engine, appId: 'test' }),
     );
   });