fix(runtime): per-request kernel lives on the request, not on the dispatcher (#5155)

One HttpDispatcher serves a whole host, but the kernel a request resolves to
is per request. That answer was stored on the instance field `this.kernel`,
written once per request by `resolveRequestScope()` and read by every service
lookup afterwards — each behind at least one `await`. Two interleaved requests
on a multi-tenant host therefore swapped data sources under each other: A
resolved env-1, yielded, B resolved env-2, and A resumed reading env-2.

`HttpProtocolContext` now carries `kernel`, written by `resolveRequestScope()`
next to the `environmentId` / `dataDriver` / `executionContext` it already
writes there. `this.kernel` is gone. Every kernel-reading member of
`DomainHandlerDeps` / `ActionExecutionDeps` takes the request as its first
parameter, so the dependency is visible at the call site and the compiler asks
for it — chosen over AsyncLocalStorage, which would have reintroduced implicit
mutable ambient context, the same defect in a new costume.

Three host-level readers (`/ready`, its driver-health probe, the memoized
`default-project` lookup) now name `defaultKernel` explicitly instead of
reading whichever tenant resolved most recently.

Covered by a deterministic interleaving regression test: request A parks inside
its own identity resolution, request B runs to completion, A resumes. On the
old code A is served env-2's i18n bundle.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VkPSGsX9o17MsGv3Lbxu2w

Author: claude

.changeset/dispatcher-per-request-kernel.md

@@ -0,0 +1,73 @@
+---
+"@objectstack/runtime": minor
+---
+
+fix(runtime): the HTTP dispatcher serves each request from its OWN resolved kernel — two tenants can no longer swap data sources under each other (#5155)
+
+A host constructs exactly **one** `HttpDispatcher` (`dispatcher-plugin.ts`
+`start()`), and every route it serves shares that instance. The kernel a request
+resolves to, however, is per request: on a multi-tenant host the injected
+`kernelResolver` (ADR-0006) picks a different one per environment.
+
+That per-request answer was being stored on a dispatcher **instance field**,
+`this.kernel`, written once per request by `resolveRequestScope()` and then read
+by `resolveService()` / `getService()` / `getObjectQL()` /
+`getRequestKernelService()` / `announceKernelEvent()` / `getRegisteredAiRoutes()`
+— every one of them behind at least one `await`. Node's single thread is no
+protection here: what it protects is code that does **not** hold mutable shared
+state across an `await`, and this held it across several.
+
+So two interleaved requests on two environments produced this:
+
+1. request A resolves, `this.kernel` = env-1's kernel;
+2. A yields at an `await` (session lookup, driver query);
+3. request B resolves, `this.kernel` = env-2's kernel;
+4. A resumes and resolves `objectql` / `metadata` / `automation` off **env-2**.
+
+One tenant's request reading another tenant's data source — a correctness and
+isolation defect, not a performance one. Single-environment deployments were
+never affected (`this.kernel === defaultKernel` always, so the write was
+idempotent), which is exactly why no local run or CI job ever showed it. It is
+now covered by a deterministic interleaving regression test
+(`http-dispatcher.multi-tenant-concurrency.test.ts`), which fails on the old
+code with request A being served env-2's data.
+
+**The fix: the resolved kernel travels on the request, and every facility that
+reads a kernel takes the request explicitly.** `HttpProtocolContext` gains a
+`kernel` field, written by `resolveRequestScope()` alongside the
+`environmentId` / `dataDriver` / `executionContext` it already writes there.
+There is no longer any `this.kernel` to rewrite. An `AsyncLocalStorage` carrier
+was deliberately **not** used: it would have reintroduced implicit mutable
+ambient context, which is the shape of this bug in a new costume.
+
+Three host-level readers moved to the host kernel explicitly, where they had
+been reading whichever tenant resolved most recently: `/ready` (readiness is a
+property of the replica), its driver-health probe, and the memoized
+single-environment `default-project` lookup.
+
+**Migration — `DomainHandlerDeps` and `ActionExecutionDeps`.** Every
+kernel-reading member now takes the request as its **first** parameter. If you
+implement or call either contract (both are exported from
+`@objectstack/runtime`; nothing in this monorepo or the sibling distributions
+did):
+
+- `deps.resolveService(name, envId)` becomes `deps.resolveService(context, name, envId)`
+- `deps.getService(name)` becomes `deps.getService(context, name)`
+- `deps.getObjectQL(envId)` becomes `deps.getObjectQL(context, envId)`
+- `deps.getRequestKernelService(name)` becomes `deps.getRequestKernelService(context, name)`
+- `deps.announceKernelEvent(event, payload)` becomes `deps.announceKernelEvent(context, event, payload)`
+- `deps.getRegisteredAiRoutes()` becomes `deps.getRegisteredAiRoutes(context)`
+
+`context` is the `HttpProtocolContext` the domain handler already receives. The
+same rule applies to the `action-execution` helpers, which take it right after
+`deps`: `callData`, `resolveAutomationService`, `dispatchFlowAction`,
+`invokeBusinessAction`, `resolveRouteActionDeclaration`.
+
+`HttpDispatcher.getDiscoveryInfo(prefix)` gains an **optional** second argument,
+the request context. Callers that serve `/discovery` straight off the host (the
+adapters, the dispatcher plugin) need no change and now describe the host kernel
+deterministically instead of whichever tenant asked last.
+
+`resolveProjectKernelObjectQL(context)` keeps its direct-caller kernel swap;
+the swap is now written onto that context, so it stays visible to the rest of
+that request and to nothing else.

packages/runtime/src/action-body-identity.test.ts

@@ -246,7 +246,7 @@ describe('#3914 — MCP run_action dispatch binds ctx.api and ctx.engine', () =>
         };
         const ec = { userId: 'user_42', tenantId: 'org_acme', positions: [], permissions: [] };
 
-        await invokeBusinessAction(mcpDeps, 'close_case', { recordId: 'case_1' }, {
+        await invokeBusinessAction(mcpDeps, { request: {} } as any, 'close_case', { recordId: 'case_1' }, {
             driver: undefined,
             envId: 'platform',
             ec,

packages/runtime/src/action-execution-calldata-query.test.ts

@@ -21,8 +21,16 @@
 
 import { describe, it, expect, beforeEach } from 'vitest';
 import { callData, type ActionExecutionDeps } from './action-execution.js';
+import type { HttpProtocolContext } from './http-dispatcher.js';
 
 const EC = { userId: 'u1', isSystem: false, positions: [], permissions: [] } as any;
+/**
+ * The request `callData` is serving. [#5155] Every service lookup resolves off
+ * `context.kernel`, so the request has to be named at the call — a fake that
+ * ignored it would be modelling the shared-field shape this suite's subject no
+ * longer has.
+ */
+const REQ = { request: {} } as HttpProtocolContext;
 
 function makeHarness(opts: { withProtocol?: boolean } = {}) {
     const finds: any[] = [];
@@ -39,7 +47,7 @@ function makeHarness(opts: { withProtocol?: boolean } = {}) {
         ...(protocol ? { protocol } : {}),
     };
     const deps: ActionExecutionDeps = {
-        resolveService: (async (name: string) => services[name]) as any,
+        resolveService: (async (_ctx: HttpProtocolContext, name: string) => services[name]) as any,
         getObjectQL: async () => ql,
     };
     return { deps, finds, findData };
@@ -58,52 +66,52 @@ describe("callData('query') fallback serves the query it was given (#4386)", ()
             offset: 10,
             fields: ['id', 'title'],
         };
-        const out = await callData(h.deps, 'query', { object: 'task', query }, undefined, undefined, EC);
+        const out = await callData(h.deps, REQ, 'query', { object: 'task', query }, undefined, undefined, EC);
         expect(h.finds).toHaveLength(1);
         expect(h.finds[0]).toMatchObject({ ...query, context: EC });
         expect(out.records).toHaveLength(2);
     });
 
     it('extracts query fields from bare params when params.query is absent — same source as the protocol path', async () => {
-        await callData(h.deps, 'query', { object: 'task', where: { status: 'open' }, limit: 3 }, undefined, undefined, EC);
+        await callData(h.deps, REQ, 'query', { object: 'task', where: { status: 'open' }, limit: 3 }, undefined, undefined, EC);
         expect(h.finds[0]).toMatchObject({ where: { status: 'open' }, limit: 3 });
     });
 
     it('a caller-supplied context is dropped, never honoured — server-derived only, matching findData', async () => {
-        await callData(h.deps, 'query', { object: 'task', query: { where: { a: 1 }, context: { isSystem: true } } }, undefined, undefined, EC);
+        await callData(h.deps, REQ, 'query', { object: 'task', query: { where: { a: 1 }, context: { isSystem: true } } }, undefined, undefined, EC);
         expect(h.finds[0].context).toBe(EC);
     });
 
     it.each(['sort', 'select', 'skip', 'populate', 'search', 'expand', '$filter'])(
         'refuses %s with 501 instead of part-serving — nothing reaches ql.find',
         async (key) => {
             await expect(
-                callData(h.deps, 'query', { object: 'task', query: { where: { a: 1 }, [key]: 'x' } }, undefined, undefined, EC),
+                callData(h.deps, REQ, 'query', { object: 'task', query: { where: { a: 1 }, [key]: 'x' } }, undefined, undefined, EC),
             ).rejects.toMatchObject({ statusCode: 501 });
             expect(h.finds).toHaveLength(0);
         },
     );
 
     it('names the unservable keys and the served set in the refusal', async () => {
         await expect(
-            callData(h.deps, 'query', { object: 'task', query: { sort: '-x', select: 'id' } }, undefined, undefined, EC),
+            callData(h.deps, REQ, 'query', { object: 'task', query: { sort: '-x', select: 'id' } }, undefined, undefined, EC),
         ).rejects.toMatchObject({ message: expect.stringMatching(/'sort', 'select'.*where, fields, orderBy, limit, offset/s) });
     });
 
     it('an empty query still lists (the protocol path lists too) — no refusal, no predicate', async () => {
-        const out = await callData(h.deps, 'query', { object: 'task' }, undefined, undefined, EC);
+        const out = await callData(h.deps, REQ, 'query', { object: 'task' }, undefined, undefined, EC);
         expect(h.finds[0]).toMatchObject({ context: EC });
         expect(out.total).toBe(2);
     });
 
     it('null-valued keys are withdrawals, not unservable', async () => {
-        await callData(h.deps, 'query', { object: 'task', query: { sort: null, where: { a: 1 } } }, undefined, undefined, EC);
+        await callData(h.deps, REQ, 'query', { object: 'task', query: { sort: null, where: { a: 1 } } }, undefined, undefined, EC);
         expect(h.finds[0]).toMatchObject({ where: { a: 1 } });
     });
 
     it('with the protocol service present the fallback never runs — findData gets the query verbatim, wire spellings included', async () => {
         const withP = makeHarness({ withProtocol: true });
-        await callData(withP.deps, 'query', { object: 'task', query: { sort: '-title', top: 5 } }, undefined, undefined, EC);
+        await callData(withP.deps, REQ, 'query', { object: 'task', query: { sort: '-title', top: 5 } }, undefined, undefined, EC);
         expect(withP.findData).toHaveLength(1);
         expect(withP.findData[0].query).toEqual({ sort: '-title', top: 5 });
         expect(withP.finds).toHaveLength(0);

packages/runtime/src/action-execution.ts

@@ -19,6 +19,7 @@ import { validateActionParams, type ResolvedActionParam } from '@objectstack/spe
 import type { ExecutionContext } from '@objectstack/spec/kernel';
 import type { IObjectQLEngine, ServiceSlotContract, ServiceSlotContracts } from '@objectstack/spec/contracts';
 import { checkApiExposure } from './api-exposure.js';
+import type { HttpProtocolContext } from './http-dispatcher.js';
 import {
     GLOBAL_ACTION_OBJECT_KEY,
     actionHandlerObjectKeys,
@@ -78,21 +79,31 @@ function warnActionParamsOnce(key: string, message: string): void {
  * the pattern, alongside `ResolveOptions` in security/resolve-execution-context.
  * A lookup facade has to be typed everywhere it is re-declared, or the copy
  * that still says `any` becomes the way around all the others.
+ *
+ * [#5155] Both lookups take the REQUEST as their first parameter, for the
+ * reason spelled out on `DomainHandlerDeps` (of which this is the narrow view
+ * `HttpDispatcher.actionExecutionDeps` hands out): the object is shared by
+ * every request the host serves, so the kernel to resolve against is the
+ * request's, never the facade's.
  */
 export interface ActionExecutionDeps {
-    resolveService<K extends keyof ServiceSlotContracts>(name: K, environmentId?: string): Promise<ServiceSlotContract<K> | undefined>;
-    resolveService(name: string, environmentId?: string): any;
-    getObjectQL(environmentId?: string): Promise<IObjectQLEngine | null>;
+    resolveService<K extends keyof ServiceSlotContracts>(context: HttpProtocolContext, name: K, environmentId?: string): Promise<ServiceSlotContract<K> | undefined>;
+    resolveService(context: HttpProtocolContext, name: string, environmentId?: string): any;
+    getObjectQL(context: HttpProtocolContext, environmentId?: string): Promise<IObjectQLEngine | null>;
 }
 
 /**
  * Direct data service dispatch — replaces broker.call('data.*').
  * Tries protocol service first (supports expand/populate), falls back to ObjectQL.
  *
+ * @param requestContext - The request being served (#5155). Carries the kernel
+ *   every service lookup below resolves against; see
+ *   {@link HttpProtocolContext.kernel}.
  * @param dataDriver - Optional environment-scoped driver to use instead of kernel default
  * @param scopeId - Optional project ID for scoped service resolution (SharedProjectPlugin mode)
  */
-export async function callData(deps: ActionExecutionDeps, 
+export async function callData(deps: ActionExecutionDeps,
+    requestContext: HttpProtocolContext,
     action: string,
     params: any,
     dataDriver?: any,
@@ -106,7 +117,7 @@ export async function callData(deps: ActionExecutionDeps,
     if (!executionContext?.isSystem && params?.object) {
         let def: any;
         try {
-            const meta = await deps.resolveService('metadata', scopeId);
+            const meta = await deps.resolveService(requestContext, 'metadata', scopeId);
             def = await (meta as any)?.getObject?.(params.object);
         } catch {
             def = undefined; // fall open to schema defaults (apiEnabled=true)
@@ -117,9 +128,9 @@ export async function callData(deps: ActionExecutionDeps,
         }
     }
 
-    const protocol = await deps.resolveService('protocol', scopeId);
-    const qlService = dataDriver ?? await deps.getObjectQL(scopeId);
-    const ql = qlService ?? await deps.resolveService('objectql', scopeId);
+    const protocol = await deps.resolveService(requestContext, 'protocol', scopeId);
+    const qlService = dataDriver ?? await deps.getObjectQL(requestContext, scopeId);
+    const ql = qlService ?? await deps.resolveService(requestContext, 'objectql', scopeId);
     const qlOpts = executionContext ? { context: executionContext } : undefined;
     const findOpts = (extra?: any) => {
         const base = qlOpts ? { ...qlOpts } : {};
@@ -252,8 +263,8 @@ export async function callData(deps: ActionExecutionDeps,
         if (!Array.isArray(params.aggregations) || params.aggregations.length === 0) {
             throw { statusCode: 400, message: 'aggregate requires at least one aggregation' };
         }
-        const engine = (await deps.getObjectQL(scopeId))
-            ?? await deps.resolveService('objectql', scopeId).catch(() => null);
+        const engine = (await deps.getObjectQL(requestContext, scopeId))
+            ?? await deps.resolveService(requestContext, 'objectql', scopeId).catch(() => null);
         if (engine && typeof engine.aggregate === 'function') {
             const rows = await engine.aggregate(
                 params.object,
@@ -383,13 +394,13 @@ export function headlessActionTypeError(_deps: ActionExecutionDeps, action: any,
  * the single availability probe behind `type: 'flow'` dispatch (both the
  * headless-invokability filter and the two invoke paths ask through it).
  */
-export async function resolveAutomationService(deps: ActionExecutionDeps, envId?: string): Promise<any | null> {
+export async function resolveAutomationService(deps: ActionExecutionDeps, requestContext: HttpProtocolContext, envId?: string): Promise<any | null> {
     try {
         // [#4127 batch 4] Was `: any`, which voided the gate here. `execute` is
         // declared on IAutomationService, so this needed no contract work — only
         // for someone to notice, and three grep sweeps over `domains/*.ts` never
         // reached this file. The lint rule did.
-        const svc = await deps.resolveService('automation', envId);
+        const svc = await deps.resolveService(requestContext, 'automation', envId);
         return svc && typeof svc.execute === 'function' ? svc : null;
     } catch {
         return null; // no automation service on this kernel
@@ -483,6 +494,7 @@ export function seedFlowActionParams(_deps: ActionExecutionDeps,
  * doesn't keep.
  */
 export async function dispatchFlowAction(deps: ActionExecutionDeps,
+    requestContext: HttpProtocolContext,
     action: any,
     wiring: {
         objectName: string;
@@ -494,7 +506,7 @@ export async function dispatchFlowAction(deps: ActionExecutionDeps,
     },
 ): Promise<any> {
     const { objectName, record, params, recordId, ec, envId } = wiring;
-    const automation = await resolveAutomationService(deps, envId);
+    const automation = await resolveAutomationService(deps, requestContext, envId);
     if (!automation) {
         throw new Error(flowActionUnavailableError(action));
     }
@@ -790,7 +802,8 @@ export function buildActionEngineFacade(_deps: ActionExecutionDeps, ql: any, ec?
  * attributable and org-scoped. Flow actions differ: the flow engine receives
  * the caller's identity below and honours `runAs` (ADR-0049).
  */
-export async function invokeBusinessAction(deps: ActionExecutionDeps, 
+export async function invokeBusinessAction(deps: ActionExecutionDeps,
+    requestContext: HttpProtocolContext,
     name: string,
     input: { objectName?: string; recordId?: string; params?: Record<string, unknown> },
     wiring: {
@@ -821,7 +834,7 @@ export async function invokeBusinessAction(deps: ActionExecutionDeps,
     if (isSystemObjectName(objectName)) {
         throw new Error(`Action '${name}' is on a system object and is not exposed via MCP`);
     }
-    const hasAutomation = Boolean(await resolveAutomationService(deps, envId));
+    const hasAutomation = Boolean(await resolveAutomationService(deps, requestContext, envId));
     if (!isHeadlessInvokableAction(deps, action, hasAutomation)) {
         throw new Error(
             `Action '${name}' (type='${action?.type ?? 'script'}') cannot be invoked via MCP`,
@@ -873,15 +886,15 @@ export async function invokeBusinessAction(deps: ActionExecutionDeps,
 
     // ── flow dispatch ── (shared with the REST /actions route, #3915)
     if (action.type === 'flow') {
-        const result = await dispatchFlowAction(deps, action, { objectName, record, params, recordId, ec, envId });
+        const result = await dispatchFlowAction(deps, requestContext, action, { objectName, record, params, recordId, ec, envId });
         return { ok: true, action: action.name, objectName, ...(recordId ? { recordId } : {}), result };
     }
 
     // ── script/body dispatch via the engine's executeAction ──
     // [#4127] `executeAction` is
     // ObjectQL's own surface, outside IDataEngine; `getObjectQL` exists to reach
     // exactly that. Closing this needs ObjectQL's contract written, not a cast.
-    const ql: any = await deps.getObjectQL(envId);
+    const ql: any = await deps.getObjectQL(requestContext, envId);
     if (!ql || typeof ql.executeAction !== 'function') {
         throw new Error('Data engine not available for action dispatch');
     }
@@ -1093,6 +1106,7 @@ export async function executeRegisteredAction(_deps: ActionExecutionDeps,
  * lookup.
  */
 export async function resolveRouteActionDeclaration(deps: ActionExecutionDeps,
+    requestContext: HttpProtocolContext,
     args: { ql: any; objectName: string; actionName: string; envId?: string },
 ): Promise<{ action: any; obj: any; degraded?: boolean; reason?: string }> {
     const { ql, objectName, actionName, envId } = args;
@@ -1140,7 +1154,7 @@ export async function resolveRouteActionDeclaration(deps: ActionExecutionDeps,
         // and belongs in the batch that adds the four undeclared auth members.
         // [#4127 batch 4] `loadDiagnosed` is on IMetadataService now, so this
         // reads the contract instead of guessing at it.
-        const meta = await deps.resolveService('metadata', envId);
+        const meta = await deps.resolveService(requestContext, 'metadata', envId);
         if (meta && typeof meta.loadDiagnosed === 'function') {
             const diag: any = await meta.loadDiagnosed('action', actionName);
             if (diag?.data && ownsRoute(diag.data)) return { action: diag.data, obj };