diff --git a/.changeset/ready-probe-driver-health.md b/.changeset/ready-probe-driver-health.md
new file mode 100644
index 0000000000..c5add76c24
--- /dev/null
+++ b/.changeset/ready-probe-driver-health.md
@@ -0,0 +1,48 @@
+---
+"@objectstack/objectql": minor
+"@objectstack/runtime": minor
+---
+
+feat(runtime): `/ready` reports 503 when a data driver stops answering (#3756)
+
+`/health` returned `{status: 'ok'}` unconditionally and `/ready` only checked
+whether the kernel state was `running` — a flag set once when bootstrap finishes
+and never revisited. Neither probe touched the data layer. So a database that
+went away *after* boot (restart, failover, network policy change, pool exhausted,
+credentials rotated) left both probes green: the load balancer kept routing to a
+replica that failed 100% of its requests, and the orchestrator saw nothing wrong.
+The driver's `checkHealth()` already existed and was cheap (`SELECT 1` /
+`db.command({ping:1})`) but was only consumed by `datasource-admin`'s
+`testConnection` — no probe path called it, and `ObjectQL` exposed no way to ask
+(`drivers` is private with no accessor).
+
+This is the runtime-side half of #3741, which fixed only the boot-time version
+of the same defect.
+
+- New `ObjectQL.checkDriversHealth({ timeoutMs })` pings every registered driver
+ and returns a `DriverHealth[]` verdict. Each probe is settled independently and
+ bounded (default 2s) — `checkHealth()` swallows its own errors, but on a dead
+ knex pool it does not return at all, waiting out `acquireConnectionTimeout`
+ (60s by default), and a probe that hangs is as useless as one that lies. A
+ driver implementing no `checkHealth()` is reported healthy: absence of a probe
+ is not evidence of failure.
+- `GET /ready` now returns 503 with the failing driver names when the kernel is
+ running but a driver is down, on top of the existing booting/shutting-down
+ cases. The result is memoized for ~1s so Kubernetes' few-second polling does
+ not become one database round-trip per probe per replica.
+- `GET /health` deliberately still checks nothing, and now says why in the code.
+ A failing *liveness* probe restarts the pod, which cannot fix an unreachable
+ database but would put every replica into a restart storm for the length of the
+ outage. Readiness — leave the rotation — is the failure mode that helps.
+
+The readiness check **fails open**: a kernel with no data engine (lite kernels,
+edge, metadata-only hosts), an engine predating `checkDriversHealth`, or a probe
+that itself throws all read as ready, exactly as before. Readiness gates whether
+a replica receives any traffic at all, so an inconclusive answer must not
+black-hole a working deployment. Only a driver that positively reports itself
+unhealthy takes the replica out.
+
+**Migration.** None. Deployments already wiring `/api/v1/ready` as their
+readiness probe get the stricter check automatically; deployments that pointed a
+*liveness* probe at `/ready` should move it to `/health`, which is the endpoint
+that never fails on a dependency.
diff --git a/content/docs/deployment/self-hosting.mdx b/content/docs/deployment/self-hosting.mdx
index 7331466bc7..61a508ed28 100644
--- a/content/docs/deployment/self-hosting.mdx
+++ b/content/docs/deployment/self-hosting.mdx
@@ -231,7 +231,21 @@ Every runtime exposes two probe endpoints — wire them into Docker
| Endpoint | Meaning | Use as |
|:---|:---|:---|
| `GET /api/v1/health` | Process is up and serving HTTP | Liveness probe |
-| `GET /api/v1/ready` | Kernel booted, ready for traffic | Readiness probe |
+| `GET /api/v1/ready` | Kernel booted **and the data drivers answer** | Readiness probe |
+
+Wire each to the probe it is named for — they answer deliberately different
+questions:
+
+- **`/health` checks nothing but the process.** It never touches the database,
+ on purpose: a failing liveness probe makes the orchestrator *restart the pod*,
+ which cannot fix an unreachable database but would put every replica into a
+ restart storm for the length of the outage.
+- **`/ready` pings the data drivers** (bounded, and cached ~1s so frequent
+ polling costs no extra round-trips). A replica whose driver is down fails 100%
+ of its requests, so it returns 503 with the failing driver names and leaves
+ the load-balancer rotation until the database comes back. If the check is
+ inconclusive — no data engine at all, or the probe itself errors — the replica
+ stays ready rather than black-holing a working deployment.
### Kubernetes
@@ -278,9 +292,20 @@ spec:
targetPort: 8080
```
-`/api/v1/ready` returns 503 while the kernel is booting *and* during graceful
-shutdown, so rolling restarts drain cleanly. Before setting `replicas > 1`,
-read the multi-node note below.
+`/api/v1/ready` returns 503 while the kernel is booting, during graceful
+shutdown, and whenever a data driver stops answering — so rolling restarts drain
+cleanly and a replica that lost its database stops receiving traffic instead of
+serving 500s. Before setting `replicas > 1`, read the multi-node note below.
+
+
+Drivers do **not** reconnect on their own
+([#3759](https://github.com/objectstack-ai/objectstack/issues/3759)). Once a
+connection is lost the process stays broken until it restarts, so on Kubernetes
+the readiness probe above is what removes the replica — pair it with your
+normal restart policy. Without an orchestrator (bare `os serve`, systemd, a
+single container with no health check), plan for a supervisor that restarts on
+a failing `/api/v1/ready`.
+
## Reverse proxy & TLS
diff --git a/content/docs/protocol/kernel/lifecycle.mdx b/content/docs/protocol/kernel/lifecycle.mdx
index 09f26f08ba..46bda516dd 100644
--- a/content/docs/protocol/kernel/lifecycle.mdx
+++ b/content/docs/protocol/kernel/lifecycle.mdx
@@ -573,18 +573,31 @@ ObjectStack includes **built-in health monitoring** to validate system state.
```
GET /health
→ 200 with { status, version, uptime } if the process is alive (liveness)
+ → never fails on a dependency — see below
GET /ready
- → 200 if the kernel is running and ready to accept requests (readiness)
+ → 200 if the kernel is running AND every data driver answers (readiness)
→ 503 while still booting or shutting down
+ → 503 with { state, drivers: [...] } when a data driver stops answering
```
+The two probes answer deliberately different questions (#3756). `/health`
+checks nothing but the process, because a failing *liveness* probe makes the
+orchestrator restart the pod — which cannot fix an unreachable database, but
+would put every replica into a restart storm for the length of the outage.
+`/ready` pings the data drivers (bounded, memoized ~1s) because its failure mode
+— leave the load-balancer rotation — is the one that helps a replica that would
+otherwise fail 100% of its requests. The readiness check fails **open**: a
+kernel with no data engine, or a probe that itself errors, still reads as ready
+rather than black-holing a working deployment.
+
`GET /health` returns a compact liveness body (`status`, `version`, `uptime`) and
`GET /ready` returns readiness. The richer per-subsystem report and the
plugin-declared custom checks below describe the internal health-monitor model
-(`PluginHealthMonitor`) — they are **not yet** exposed as a dedicated HTTP
-endpoint or as a declarative plugin field.
+(`PluginHealthMonitor`), which covers **plugins, not driver connections** — they
+are **not yet** exposed as a dedicated HTTP endpoint or as a declarative plugin
+field.
### Health Status Response
diff --git a/packages/objectql/src/core.ts b/packages/objectql/src/core.ts
index d154ad8b69..c170652f29 100644
--- a/packages/objectql/src/core.ts
+++ b/packages/objectql/src/core.ts
@@ -43,7 +43,7 @@ export type { ObjectQLHostContext, HookHandler, HookEntry, OperationContext, Eng
// fails (framework#3741). Embedders that boot the engine themselves can catch
// it to render their own "database unreachable" message.
export { DriverConnectError } from './driver-connect-errors.js';
-export type { DriverConnectFailure } from './driver-connect-errors.js';
+export type { DriverConnectFailure, DriverHealth } from './driver-connect-errors.js';
// In-memory aggregation fallback
export { applyInMemoryAggregation, bucketDateValue } from './in-memory-aggregation.js';
diff --git a/packages/objectql/src/driver-connect-errors.ts b/packages/objectql/src/driver-connect-errors.ts
index 52d2e1dd05..3d2fa83287 100644
--- a/packages/objectql/src/driver-connect-errors.ts
+++ b/packages/objectql/src/driver-connect-errors.ts
@@ -6,6 +6,19 @@ export interface DriverConnectFailure {
error: unknown;
}
+/**
+ * One driver's verdict from `ObjectQL.checkDriversHealth()` (framework#3756) —
+ * whether it can serve a query right now, not whether it connected at boot.
+ */
+export interface DriverHealth {
+ driverName: string;
+ healthy: boolean;
+ /** Why it is unhealthy: `checkHealth()` returned false, threw, or timed out. */
+ error?: string;
+ /** True when the driver implements no `checkHealth()` and was assumed healthy. */
+ skipped?: boolean;
+}
+
/** `error.message` when it is an Error, its string form otherwise. */
function failureMessage(error: unknown): string {
if (error instanceof Error) return error.message || error.name;
diff --git a/packages/objectql/src/engine-driver-health.test.ts b/packages/objectql/src/engine-driver-health.test.ts
new file mode 100644
index 0000000000..df7b40f99c
--- /dev/null
+++ b/packages/objectql/src/engine-driver-health.test.ts
@@ -0,0 +1,115 @@
+// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license.
+//
+// framework#3756: `init()` answers "could the drivers connect at boot";
+// `checkDriversHealth()` answers "can they serve a query right now" — the
+// question a readiness probe needs the moment the database restarts or fails
+// over. The bound on each probe is load-bearing: `checkHealth()` swallows its
+// own errors, but on a dead pool it does not return at all.
+
+import { describe, it, expect } from 'vitest';
+import { ObjectQL } from './engine.js';
+
+function makeDriver(name: string, checkHealth?: () => Promise) {
+ const driver: any = {
+ name,
+ version: '0.0.0',
+ supports: {},
+ async connect() {}, async disconnect() {},
+ async execute() { return null; },
+ async find() { return []; },
+ findStream() { throw new Error('ni'); },
+ async findOne() { return null; },
+ async create(_o: string, data: Record) { return { id: 'r_1', ...data }; },
+ async update(_o: string, id: string, data: Record) { return { ...data, id }; },
+ async delete() { return true; },
+ async count() { return 0; },
+ async bulkCreate() { return []; },
+ async bulkUpdate() { return []; },
+ async bulkDelete() {},
+ async beginTransaction() { return { __trx: true, commit: async () => {}, rollback: async () => {} }; },
+ async commit() {}, async rollback() {},
+ };
+ if (checkHealth) driver.checkHealth = checkHealth;
+ return driver;
+}
+
+const healthy = (name: string) => makeDriver(name, async () => true);
+const unhealthy = (name: string) => makeDriver(name, async () => false);
+const throwing = (name: string, msg: string) =>
+ makeDriver(name, async () => { throw new Error(msg); });
+/** Models a dead knex pool: `SELECT 1` neither resolves nor rejects. */
+const hanging = (name: string) => makeDriver(name, () => new Promise(() => {}));
+
+function engineWith(...drivers: any[]) {
+ const engine = new ObjectQL({
+ logger: { debug() {}, info() {}, warn() {}, error() {} },
+ } as any);
+ drivers.forEach((d, i) => engine.registerDriver(d, i === 0));
+ return engine;
+}
+
+describe('ObjectQL.checkDriversHealth() (framework#3756)', () => {
+ it('reports every registered driver healthy when all pings succeed', async () => {
+ const engine = engineWith(healthy('sql'), healthy('mongodb'));
+
+ const results = await engine.checkDriversHealth();
+
+ expect(results).toEqual([
+ { driverName: 'sql', healthy: true },
+ { driverName: 'mongodb', healthy: true },
+ ]);
+ });
+
+ it('flags a driver whose checkHealth() returns false, naming it', async () => {
+ const engine = engineWith(healthy('sql'), unhealthy('mongodb'));
+
+ const results = await engine.checkDriversHealth();
+
+ expect(results.filter(r => !r.healthy).map(r => r.driverName)).toEqual(['mongodb']);
+ expect(results.find(r => r.driverName === 'mongodb')?.error).toContain('returned false');
+ });
+
+ it('flags a driver whose checkHealth() throws, carrying the message', async () => {
+ const engine = engineWith(throwing('sql', 'ECONNRESET'));
+
+ const [result] = await engine.checkDriversHealth();
+
+ expect(result).toMatchObject({ driverName: 'sql', healthy: false });
+ expect(result.error).toContain('ECONNRESET');
+ });
+
+ it('times out a hanging probe instead of awaiting it — a dead pool never settles', async () => {
+ const engine = engineWith(hanging('sql'));
+
+ const started = Date.now();
+ const [result] = await engine.checkDriversHealth({ timeoutMs: 50 });
+ const elapsed = Date.now() - started;
+
+ expect(result).toMatchObject({ driverName: 'sql', healthy: false });
+ expect(result.error).toContain('did not settle');
+ expect(elapsed).toBeLessThan(1_000); // would be knex's 60s acquire timeout
+ });
+
+ it('does not let one hanging driver hide a healthy one — probes settle independently', async () => {
+ const engine = engineWith(hanging('sql'), healthy('memory'));
+
+ const results = await engine.checkDriversHealth({ timeoutMs: 50 });
+
+ expect(results.find(r => r.driverName === 'memory')?.healthy).toBe(true);
+ expect(results.find(r => r.driverName === 'sql')?.healthy).toBe(false);
+ });
+
+ it('assumes a driver without checkHealth() is healthy rather than taking a replica out', async () => {
+ const engine = engineWith(makeDriver('legacy'));
+
+ const [result] = await engine.checkDriversHealth();
+
+ expect(result).toEqual({ driverName: 'legacy', healthy: true, skipped: true });
+ });
+
+ it('returns an empty verdict when no driver is registered', async () => {
+ const engine = engineWith();
+
+ await expect(engine.checkDriversHealth()).resolves.toEqual([]);
+ });
+});
diff --git a/packages/objectql/src/engine.ts b/packages/objectql/src/engine.ts
index cc7ae26815..2d4eb2abdd 100644
--- a/packages/objectql/src/engine.ts
+++ b/packages/objectql/src/engine.ts
@@ -21,7 +21,7 @@ import {
import { ExecutionContext, ExecutionContextInput, ExecutionContextSchema } from '@objectstack/spec/kernel';
import { IDataDriver, IDataEngine, Logger, createLogger, withTransientRetry, type RetryOptions } from '@objectstack/core';
import { SummaryRecomputeError, type SummaryRecomputeFailure } from './summary-errors.js';
-import { DriverConnectError, emitDegradedBootBanner, type DriverConnectFailure } from './driver-connect-errors.js';
+import { DriverConnectError, emitDegradedBootBanner, type DriverConnectFailure, type DriverHealth } from './driver-connect-errors.js';
import { resolveAllowDriverConnectFailure } from '@objectstack/types';
/**
@@ -1947,6 +1947,65 @@ export class ObjectQL implements IDataEngine {
this.logger.info('ObjectQL engine initialization complete');
}
+ /**
+ * Ping every registered driver and report which ones are usable RIGHT NOW
+ * (framework#3756).
+ *
+ * `init()` answers "could the drivers connect at boot"; this answers "can
+ * they serve a query at this instant", which is a different question the
+ * moment the database restarts, fails over, or drops the pool. Readiness
+ * probes are the intended caller — a replica whose driver is down fails 100%
+ * of its requests and must leave the load-balancer rotation.
+ *
+ * Every check is bounded by `timeoutMs` (default 2s) and settled
+ * independently. The bound is not optional: `IDataDriver.checkHealth()`
+ * swallows its own errors and returns `false`, but on a dead pool it does
+ * not return at all — knex's `SELECT 1` waits out `acquireConnectionTimeout`
+ * (60s by default). A probe that hangs is as useless as one that lies, so a
+ * timed-out driver is reported unhealthy rather than awaited.
+ *
+ * A driver that implements no `checkHealth()` is reported healthy: absence of
+ * a probe is not evidence of failure, and reporting "unhealthy" would take a
+ * working deployment out of rotation over a driver that simply never had the
+ * optional method.
+ */
+ async checkDriversHealth(opts?: { timeoutMs?: number }): Promise {
+ const timeoutMs = opts?.timeoutMs ?? 2_000;
+
+ return Promise.all(
+ Array.from(this.drivers, async ([driverName, driver]): Promise => {
+ if (typeof driver.checkHealth !== 'function') {
+ return { driverName, healthy: true, skipped: true };
+ }
+ let timer: ReturnType | undefined;
+ try {
+ const healthy = await Promise.race([
+ Promise.resolve(driver.checkHealth()),
+ new Promise((_, reject) => {
+ timer = setTimeout(
+ () => reject(new Error(`checkHealth did not settle within ${timeoutMs}ms`)),
+ timeoutMs,
+ );
+ // Never hold the event loop open for a probe.
+ (timer as { unref?: () => void }).unref?.();
+ }),
+ ]);
+ return healthy
+ ? { driverName, healthy: true }
+ : { driverName, healthy: false, error: 'checkHealth() returned false' };
+ } catch (e) {
+ return {
+ driverName,
+ healthy: false,
+ error: e instanceof Error ? e.message : String(e),
+ };
+ } finally {
+ if (timer) clearTimeout(timer);
+ }
+ }),
+ );
+ }
+
/**
* Does this object declare a media field at all? Cached per schema object —
* the registry hands back the same instance per object, so this scans a
diff --git a/packages/objectql/src/index.ts b/packages/objectql/src/index.ts
index ce81f6446c..2d7351c383 100644
--- a/packages/objectql/src/index.ts
+++ b/packages/objectql/src/index.ts
@@ -39,7 +39,7 @@ export type { SummaryRecomputeFailure } from './summary-errors.js';
// fails (framework#3741). Hosts that boot the engine themselves can catch it to
// render their own "database unreachable" message.
export { DriverConnectError } from './driver-connect-errors.js';
-export type { DriverConnectFailure } from './driver-connect-errors.js';
+export type { DriverConnectFailure, DriverHealth } from './driver-connect-errors.js';
export type { InsertManyRowOutcome } from './engine.js';
// Export in-memory aggregation fallback (used by engine.aggregate when the
diff --git a/packages/runtime/src/http-dispatcher.ready.test.ts b/packages/runtime/src/http-dispatcher.ready.test.ts
index 6e40a48b33..1829fdf21b 100644
--- a/packages/runtime/src/http-dispatcher.ready.test.ts
+++ b/packages/runtime/src/http-dispatcher.ready.test.ts
@@ -1,15 +1,20 @@
-import { describe, it, expect } from 'vitest';
+import { describe, it, expect, vi } from 'vitest';
import { HttpDispatcher } from './http-dispatcher.js';
-function kernel(state: string): any {
+function kernel(state: string, dataService?: unknown): any {
return {
getState: () => state,
- getService: () => undefined,
+ getService: (name: string) => (name === 'data' ? dataService : undefined),
getServiceAsync: async () => undefined,
};
}
const ctx: any = {};
+/** An engine whose `checkDriversHealth` reports the given verdicts. */
+function engine(results: Array<{ driverName: string; healthy: boolean }>) {
+ return { checkDriversHealth: vi.fn(async () => results) };
+}
+
describe('HttpDispatcher — GET /ready readiness probe', () => {
it('returns 200 when the kernel is running', async () => {
const res = await new HttpDispatcher(kernel('running')).dispatch('GET', '/ready', undefined, undefined, ctx);
@@ -24,4 +29,72 @@ describe('HttpDispatcher — GET /ready readiness probe', () => {
expect(res.response.status).toBe(503);
}
});
+
+ // framework#3756 — a running kernel whose driver is down fails 100% of its
+ // requests; readiness is what takes it out of the load-balancer rotation.
+ describe('data-driver gating', () => {
+ it('returns 200 when every driver is healthy', async () => {
+ const res = await new HttpDispatcher(
+ kernel('running', engine([{ driverName: 'sql', healthy: true }])),
+ ).dispatch('GET', '/ready', undefined, undefined, ctx);
+
+ expect(res.response.status).toBe(200);
+ });
+
+ it('returns 503 naming the driver when one is down, even though the kernel runs', async () => {
+ const res = await new HttpDispatcher(
+ kernel('running', engine([
+ { driverName: 'sql', healthy: false },
+ { driverName: 'memory', healthy: true },
+ ])),
+ ).dispatch('GET', '/ready', undefined, undefined, ctx);
+
+ expect(res.response.status).toBe(503);
+ expect(res.response.body.error.message).toBe('Data driver unavailable');
+ expect(res.response.body.error.details).toEqual({ state: 'running', drivers: ['sql'] });
+ });
+
+ it('does not re-probe within the memo TTL — k8s polls every few seconds', async () => {
+ const e = engine([{ driverName: 'sql', healthy: true }]);
+ const dispatcher = new HttpDispatcher(kernel('running', e));
+
+ await dispatcher.dispatch('GET', '/ready', undefined, undefined, ctx);
+ await dispatcher.dispatch('GET', '/ready', undefined, undefined, ctx);
+ await dispatcher.dispatch('GET', '/ready', undefined, undefined, ctx);
+
+ expect(e.checkDriversHealth).toHaveBeenCalledTimes(1);
+ });
+
+ it('stays ready when the probe itself throws — inconclusive is not unhealthy', async () => {
+ const res = await new HttpDispatcher(
+ kernel('running', {
+ checkDriversHealth: async () => { throw new Error('engine exploded'); },
+ }),
+ ).dispatch('GET', '/ready', undefined, undefined, ctx);
+
+ expect(res.response.status).toBe(200);
+ });
+
+ it('stays ready on an engine predating checkDriversHealth', async () => {
+ const res = await new HttpDispatcher(kernel('running', { find: async () => [] }))
+ .dispatch('GET', '/ready', undefined, undefined, ctx);
+
+ expect(res.response.status).toBe(200);
+ });
+ });
+});
+
+describe('HttpDispatcher — GET /health liveness probe', () => {
+ // framework#3756: liveness must NOT check the database. Its failure makes the
+ // orchestrator restart the pod, which cannot fix an unreachable database but
+ // would put every replica into a restart storm during the outage.
+ it('stays 200 while the data driver is down', async () => {
+ const e = engine([{ driverName: 'sql', healthy: false }]);
+ const res = await new HttpDispatcher(kernel('running', e))
+ .dispatch('GET', '/health', undefined, undefined, ctx);
+
+ expect(res.response.status).toBe(200);
+ expect(res.response.body.data.status).toBe('ok');
+ expect(e.checkDriversHealth).not.toHaveBeenCalled();
+ });
});
diff --git a/packages/runtime/src/http-dispatcher.ts b/packages/runtime/src/http-dispatcher.ts
index b8ba9c2e25..e0fcef5963 100644
--- a/packages/runtime/src/http-dispatcher.ts
+++ b/packages/runtime/src/http-dispatcher.ts
@@ -170,6 +170,15 @@ export class HttpDispatcher {
* before the legacy if-chain. See {@link DomainHandlerRegistry}.
*/
private readonly domainRegistry = new DomainHandlerRegistry();
+ /**
+ * Short-TTL memo for `/ready`'s driver probe (framework#3756). Kubernetes
+ * polls readiness every few seconds per replica; without this, every poll
+ * would be a database round-trip. One second is short enough that the
+ * verdict tracks an outage within a single probe interval and long enough
+ * that concurrent probes collapse onto one query.
+ */
+ private driverHealthMemo?: { at: number; unhealthy: string[] };
+ private static readonly DRIVER_HEALTH_TTL_MS = 1_000;
/**
* When `true`, scoped data-plane routes enforce a
* `sys_environment_member` lookup and return 403 for non-members.
@@ -281,6 +290,15 @@ export class HttpDispatcher {
*/
private registerBuiltinDomains(): void {
// GET /health — liveness probe (was branch "0b").
+ //
+ // Deliberately checks NOTHING beyond "this process is executing code",
+ // and must stay that way (framework#3756). A failing liveness probe
+ // makes the orchestrator RESTART the pod — which cannot fix an
+ // unreachable database, but would put every replica into a restart
+ // storm for the duration of the outage and kill in-flight requests
+ // that had nothing to do with the data layer. The dependency check
+ // belongs on `/ready`, whose failure mode (leave the LB rotation) is
+ // the one that actually helps.
this.domainRegistry.register({
prefix: '/health', match: 'exact', methods: ['GET'],
handler: async () => ({
@@ -294,19 +312,32 @@ export class HttpDispatcher {
}),
});
// GET /ready — k8s / load-balancer readiness probe (was branch "0b2").
- // 200 only when the kernel is fully running; 503 while booting
- // (idle/initializing) or shutting down (stopping/stopped) so a load
- // balancer stops routing to this replica BEFORE in-flight requests
- // are drained and the server closes (graceful rolling restart).
+ // 200 only when the kernel is fully running AND the data drivers can
+ // serve a query. 503 while booting (idle/initializing) or shutting down
+ // (stopping/stopped) so a load balancer stops routing to this replica
+ // BEFORE in-flight requests are drained and the server closes (graceful
+ // rolling restart) — and 503 when a driver is down, so a replica that
+ // would fail 100% of its requests leaves the rotation instead of
+ // absorbing traffic (framework#3756).
this.domainRegistry.register({
prefix: '/ready', match: 'exact', methods: ['GET'],
handler: async () => {
const state: string = typeof (this.kernel as any)?.getState === 'function'
? (this.kernel as any).getState()
: 'running';
- return state === 'running'
+ if (state !== 'running') {
+ return { handled: true, response: this.error('Service not ready', 503, { state }) };
+ }
+ const unhealthy = await this.unhealthyDrivers();
+ return unhealthy.length === 0
? { handled: true, response: this.success({ status: 'ready', state }) }
- : { handled: true, response: this.error('Service not ready', 503, { state }) };
+ : {
+ handled: true,
+ response: this.error('Data driver unavailable', 503, {
+ state,
+ drivers: unhealthy,
+ }),
+ };
},
});
this.domainRegistry.register(createAnalyticsDomain(this.domainDeps));
@@ -337,6 +368,46 @@ export class HttpDispatcher {
this.domainRegistry.register(route);
}
+ /**
+ * Names of the data drivers that cannot serve a query right now, for
+ * `/ready` (framework#3756). Empty means "no reason to leave the LB
+ * rotation" — which includes every case where we cannot tell.
+ *
+ * Fails OPEN by design, and the asymmetry is deliberate: readiness gates
+ * whether this replica receives ANY traffic, so an inconclusive probe must
+ * not black-hole a working deployment. A kernel with no data engine (lite
+ * kernels, edge, metadata-only hosts), an engine predating
+ * `checkDriversHealth`, or a probe that itself throws all read as ready —
+ * exactly as they did before this check existed. Only a driver that
+ * positively reports itself unhealthy takes the replica out.
+ */
+ private async unhealthyDrivers(): Promise {
+ const memo = this.driverHealthMemo;
+ if (memo && Date.now() - memo.at < HttpDispatcher.DRIVER_HEALTH_TTL_MS) {
+ return memo.unhealthy;
+ }
+ let unhealthy: string[] = [];
+ try {
+ let engine: any;
+ try {
+ engine = (this.kernel as any)?.getService?.('data');
+ } catch {
+ // 'data' not registered — no data plane to gate readiness on.
+ }
+ if (typeof engine?.checkDriversHealth === 'function') {
+ const results = await engine.checkDriversHealth();
+ unhealthy = (Array.isArray(results) ? results : [])
+ .filter((r: any) => r && r.healthy === false)
+ .map((r: any) => String(r.driverName));
+ }
+ } catch {
+ // The probe itself failed — inconclusive, not unhealthy. See above.
+ unhealthy = [];
+ }
+ this.driverHealthMemo = { at: Date.now(), unhealthy };
+ return unhealthy;
+ }
+
private resolveDefaultProject(): { environmentId: string; orgId?: string } | undefined {
if (this.defaultProject) return this.defaultProject;
try {