feat(runtime): /ready reports 503 when a data driver stops answering (#3756)

.changeset/ready-probe-driver-health.md

@@ -0,0 +1,48 @@
+---
+"@objectstack/objectql": minor
+"@objectstack/runtime": minor
+---
+
+feat(runtime): `/ready` reports 503 when a data driver stops answering (#3756)
+
+`/health` returned `{status: 'ok'}` unconditionally and `/ready` only checked
+whether the kernel state was `running` — a flag set once when bootstrap finishes
+and never revisited. Neither probe touched the data layer. So a database that
+went away *after* boot (restart, failover, network policy change, pool exhausted,
+credentials rotated) left both probes green: the load balancer kept routing to a
+replica that failed 100% of its requests, and the orchestrator saw nothing wrong.
+The driver's `checkHealth()` already existed and was cheap (`SELECT 1` /
+`db.command({ping:1})`) but was only consumed by `datasource-admin`'s
+`testConnection` — no probe path called it, and `ObjectQL` exposed no way to ask
+(`drivers` is private with no accessor).
+
+This is the runtime-side half of #3741, which fixed only the boot-time version
+of the same defect.
+
+- New `ObjectQL.checkDriversHealth({ timeoutMs })` pings every registered driver
+  and returns a `DriverHealth[]` verdict. Each probe is settled independently and
+  bounded (default 2s) — `checkHealth()` swallows its own errors, but on a dead
+  knex pool it does not return at all, waiting out `acquireConnectionTimeout`
+  (60s by default), and a probe that hangs is as useless as one that lies. A
+  driver implementing no `checkHealth()` is reported healthy: absence of a probe
+  is not evidence of failure.
+- `GET /ready` now returns 503 with the failing driver names when the kernel is
+  running but a driver is down, on top of the existing booting/shutting-down
+  cases. The result is memoized for ~1s so Kubernetes' few-second polling does
+  not become one database round-trip per probe per replica.
+- `GET /health` deliberately still checks nothing, and now says why in the code.
+  A failing *liveness* probe restarts the pod, which cannot fix an unreachable
+  database but would put every replica into a restart storm for the length of the
+  outage. Readiness — leave the rotation — is the failure mode that helps.
+
+The readiness check **fails open**: a kernel with no data engine (lite kernels,
+edge, metadata-only hosts), an engine predating `checkDriversHealth`, or a probe
+that itself throws all read as ready, exactly as before. Readiness gates whether
+a replica receives any traffic at all, so an inconclusive answer must not
+black-hole a working deployment. Only a driver that positively reports itself
+unhealthy takes the replica out.
+
+**Migration.** None. Deployments already wiring `/api/v1/ready` as their
+readiness probe get the stricter check automatically; deployments that pointed a
+*liveness* probe at `/ready` should move it to `/health`, which is the endpoint
+that never fails on a dependency.

content/docs/deployment/self-hosting.mdx

@@ -231,7 +231,21 @@ Every runtime exposes two probe endpoints — wire them into Docker
 | Endpoint | Meaning | Use as |
 |:---|:---|:---|
 | `GET /api/v1/health` | Process is up and serving HTTP | Liveness probe |
-| `GET /api/v1/ready` | Kernel booted, ready for traffic | Readiness probe |
+| `GET /api/v1/ready` | Kernel booted **and the data drivers answer** | Readiness probe |
+
+Wire each to the probe it is named for — they answer deliberately different
+questions:
+
+- **`/health` checks nothing but the process.** It never touches the database,
+  on purpose: a failing liveness probe makes the orchestrator *restart the pod*,
+  which cannot fix an unreachable database but would put every replica into a
+  restart storm for the length of the outage.
+- **`/ready` pings the data drivers** (bounded, and cached ~1s so frequent
+  polling costs no extra round-trips). A replica whose driver is down fails 100%
+  of its requests, so it returns 503 with the failing driver names and leaves
+  the load-balancer rotation until the database comes back. If the check is
+  inconclusive — no data engine at all, or the probe itself errors — the replica
+  stays ready rather than black-holing a working deployment.
 
 ### Kubernetes
 
@@ -278,9 +292,20 @@ spec:
       targetPort: 8080
 ```
 
-`/api/v1/ready` returns 503 while the kernel is booting *and* during graceful
-shutdown, so rolling restarts drain cleanly. Before setting `replicas > 1`,
-read the multi-node note below.
+`/api/v1/ready` returns 503 while the kernel is booting, during graceful
+shutdown, and whenever a data driver stops answering — so rolling restarts drain
+cleanly and a replica that lost its database stops receiving traffic instead of
+serving 500s. Before setting `replicas > 1`, read the multi-node note below.
+
+<Callout type="info">
+Drivers do **not** reconnect on their own
+([#3759](https://github.com/objectstack-ai/objectstack/issues/3759)). Once a
+connection is lost the process stays broken until it restarts, so on Kubernetes
+the readiness probe above is what removes the replica — pair it with your
+normal restart policy. Without an orchestrator (bare `os serve`, systemd, a
+single container with no health check), plan for a supervisor that restarts on
+a failing `/api/v1/ready`.
+</Callout>
 
 ## Reverse proxy & TLS
 

content/docs/protocol/kernel/lifecycle.mdx

@@ -573,18 +573,31 @@ ObjectStack includes **built-in health monitoring** to validate system state.
 ```
 GET /health
   → 200 with { status, version, uptime } if the process is alive (liveness)
+  → never fails on a dependency — see below
 
 GET /ready
-  → 200 if the kernel is running and ready to accept requests (readiness)
+  → 200 if the kernel is running AND every data driver answers (readiness)
   → 503 while still booting or shutting down
+  → 503 with { state, drivers: [...] } when a data driver stops answering
 ```
 
+The two probes answer deliberately different questions (#3756). `/health`
+checks nothing but the process, because a failing *liveness* probe makes the
+orchestrator restart the pod — which cannot fix an unreachable database, but
+would put every replica into a restart storm for the length of the outage.
+`/ready` pings the data drivers (bounded, memoized ~1s) because its failure mode
+— leave the load-balancer rotation — is the one that helps a replica that would
+otherwise fail 100% of its requests. The readiness check fails **open**: a
+kernel with no data engine, or a probe that itself errors, still reads as ready
+rather than black-holing a working deployment.
+
 <Callout type="info">
 `GET /health` returns a compact liveness body (`status`, `version`, `uptime`) and
 `GET /ready` returns readiness. The richer per-subsystem report and the
 plugin-declared custom checks below describe the internal health-monitor model
-(`PluginHealthMonitor`) — they are **not yet** exposed as a dedicated HTTP
-endpoint or as a declarative plugin field.
+(`PluginHealthMonitor`), which covers **plugins, not driver connections** — they
+are **not yet** exposed as a dedicated HTTP endpoint or as a declarative plugin
+field.
 </Callout>
 
 ### Health Status Response

packages/objectql/src/core.ts

@@ -43,7 +43,7 @@ export type { ObjectQLHostContext, HookHandler, HookEntry, OperationContext, Eng
 // fails (framework#3741). Embedders that boot the engine themselves can catch
 // it to render their own "database unreachable" message.
 export { DriverConnectError } from './driver-connect-errors.js';
-export type { DriverConnectFailure } from './driver-connect-errors.js';
+export type { DriverConnectFailure, DriverHealth } from './driver-connect-errors.js';
 
 // In-memory aggregation fallback
 export { applyInMemoryAggregation, bucketDateValue } from './in-memory-aggregation.js';

packages/objectql/src/driver-connect-errors.ts

@@ -6,6 +6,19 @@ export interface DriverConnectFailure {
   error: unknown;
 }
 
+/**
+ * One driver's verdict from `ObjectQL.checkDriversHealth()` (framework#3756) —
+ * whether it can serve a query right now, not whether it connected at boot.
+ */
+export interface DriverHealth {
+  driverName: string;
+  healthy: boolean;
+  /** Why it is unhealthy: `checkHealth()` returned false, threw, or timed out. */
+  error?: string;
+  /** True when the driver implements no `checkHealth()` and was assumed healthy. */
+  skipped?: boolean;
+}
+
 /** `error.message` when it is an Error, its string form otherwise. */
 function failureMessage(error: unknown): string {
   if (error instanceof Error) return error.message || error.name;

packages/objectql/src/engine-driver-health.test.ts

@@ -0,0 +1,115 @@
+// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license.
+//
+// framework#3756: `init()` answers "could the drivers connect at boot";
+// `checkDriversHealth()` answers "can they serve a query right now" — the
+// question a readiness probe needs the moment the database restarts or fails
+// over. The bound on each probe is load-bearing: `checkHealth()` swallows its
+// own errors, but on a dead pool it does not return at all.
+
+import { describe, it, expect } from 'vitest';
+import { ObjectQL } from './engine.js';
+
+function makeDriver(name: string, checkHealth?: () => Promise<boolean>) {
+  const driver: any = {
+    name,
+    version: '0.0.0',
+    supports: {},
+    async connect() {}, async disconnect() {},
+    async execute() { return null; },
+    async find() { return []; },
+    findStream() { throw new Error('ni'); },
+    async findOne() { return null; },
+    async create(_o: string, data: Record<string, unknown>) { return { id: 'r_1', ...data }; },
+    async update(_o: string, id: string, data: Record<string, unknown>) { return { ...data, id }; },
+    async delete() { return true; },
+    async count() { return 0; },
+    async bulkCreate() { return []; },
+    async bulkUpdate() { return []; },
+    async bulkDelete() {},
+    async beginTransaction() { return { __trx: true, commit: async () => {}, rollback: async () => {} }; },
+    async commit() {}, async rollback() {},
+  };
+  if (checkHealth) driver.checkHealth = checkHealth;
+  return driver;
+}
+
+const healthy = (name: string) => makeDriver(name, async () => true);
+const unhealthy = (name: string) => makeDriver(name, async () => false);
+const throwing = (name: string, msg: string) =>
+  makeDriver(name, async () => { throw new Error(msg); });
+/** Models a dead knex pool: `SELECT 1` neither resolves nor rejects. */
+const hanging = (name: string) => makeDriver(name, () => new Promise<boolean>(() => {}));
+
+function engineWith(...drivers: any[]) {
+  const engine = new ObjectQL({
+    logger: { debug() {}, info() {}, warn() {}, error() {} },
+  } as any);
+  drivers.forEach((d, i) => engine.registerDriver(d, i === 0));
+  return engine;
+}
+
+describe('ObjectQL.checkDriversHealth() (framework#3756)', () => {
+  it('reports every registered driver healthy when all pings succeed', async () => {
+    const engine = engineWith(healthy('sql'), healthy('mongodb'));
+
+    const results = await engine.checkDriversHealth();
+
+    expect(results).toEqual([
+      { driverName: 'sql', healthy: true },
+      { driverName: 'mongodb', healthy: true },
+    ]);
+  });
+
+  it('flags a driver whose checkHealth() returns false, naming it', async () => {
+    const engine = engineWith(healthy('sql'), unhealthy('mongodb'));
+
+    const results = await engine.checkDriversHealth();
+
+    expect(results.filter(r => !r.healthy).map(r => r.driverName)).toEqual(['mongodb']);
+    expect(results.find(r => r.driverName === 'mongodb')?.error).toContain('returned false');
+  });
+
+  it('flags a driver whose checkHealth() throws, carrying the message', async () => {
+    const engine = engineWith(throwing('sql', 'ECONNRESET'));
+
+    const [result] = await engine.checkDriversHealth();
+
+    expect(result).toMatchObject({ driverName: 'sql', healthy: false });
+    expect(result.error).toContain('ECONNRESET');
+  });
+
+  it('times out a hanging probe instead of awaiting it — a dead pool never settles', async () => {
+    const engine = engineWith(hanging('sql'));
+
+    const started = Date.now();
+    const [result] = await engine.checkDriversHealth({ timeoutMs: 50 });
+    const elapsed = Date.now() - started;
+
+    expect(result).toMatchObject({ driverName: 'sql', healthy: false });
+    expect(result.error).toContain('did not settle');
+    expect(elapsed).toBeLessThan(1_000); // would be knex's 60s acquire timeout
+  });
+
+  it('does not let one hanging driver hide a healthy one — probes settle independently', async () => {
+    const engine = engineWith(hanging('sql'), healthy('memory'));
+
+    const results = await engine.checkDriversHealth({ timeoutMs: 50 });
+
+    expect(results.find(r => r.driverName === 'memory')?.healthy).toBe(true);
+    expect(results.find(r => r.driverName === 'sql')?.healthy).toBe(false);
+  });
+
+  it('assumes a driver without checkHealth() is healthy rather than taking a replica out', async () => {
+    const engine = engineWith(makeDriver('legacy'));
+
+    const [result] = await engine.checkDriversHealth();
+
+    expect(result).toEqual({ driverName: 'legacy', healthy: true, skipped: true });
+  });
+
+  it('returns an empty verdict when no driver is registered', async () => {
+    const engine = engineWith();
+
+    await expect(engine.checkDriversHealth()).resolves.toEqual([]);
+  });
+});

packages/objectql/src/engine.ts

@@ -21,7 +21,7 @@ import {
 import { ExecutionContext, ExecutionContextInput, ExecutionContextSchema } from '@objectstack/spec/kernel';
 import { IDataDriver, IDataEngine, Logger, createLogger, withTransientRetry, type RetryOptions } from '@objectstack/core';
 import { SummaryRecomputeError, type SummaryRecomputeFailure } from './summary-errors.js';
-import { DriverConnectError, emitDegradedBootBanner, type DriverConnectFailure } from './driver-connect-errors.js';
+import { DriverConnectError, emitDegradedBootBanner, type DriverConnectFailure, type DriverHealth } from './driver-connect-errors.js';
 import { resolveAllowDriverConnectFailure } from '@objectstack/types';
 
 /**
@@ -1947,6 +1947,65 @@ export class ObjectQL implements IDataEngine {
     this.logger.info('ObjectQL engine initialization complete');
   }
 
+  /**
+   * Ping every registered driver and report which ones are usable RIGHT NOW
+   * (framework#3756).
+   *
+   * `init()` answers "could the drivers connect at boot"; this answers "can
+   * they serve a query at this instant", which is a different question the
+   * moment the database restarts, fails over, or drops the pool. Readiness
+   * probes are the intended caller — a replica whose driver is down fails 100%
+   * of its requests and must leave the load-balancer rotation.
+   *
+   * Every check is bounded by `timeoutMs` (default 2s) and settled
+   * independently. The bound is not optional: `IDataDriver.checkHealth()`
+   * swallows its own errors and returns `false`, but on a dead pool it does
+   * not return at all — knex's `SELECT 1` waits out `acquireConnectionTimeout`
+   * (60s by default). A probe that hangs is as useless as one that lies, so a
+   * timed-out driver is reported unhealthy rather than awaited.
+   *
+   * A driver that implements no `checkHealth()` is reported healthy: absence of
+   * a probe is not evidence of failure, and reporting "unhealthy" would take a
+   * working deployment out of rotation over a driver that simply never had the
+   * optional method.
+   */
+  async checkDriversHealth(opts?: { timeoutMs?: number }): Promise<DriverHealth[]> {
+    const timeoutMs = opts?.timeoutMs ?? 2_000;
+
+    return Promise.all(
+      Array.from(this.drivers, async ([driverName, driver]): Promise<DriverHealth> => {
+        if (typeof driver.checkHealth !== 'function') {
+          return { driverName, healthy: true, skipped: true };
+        }
+        let timer: ReturnType<typeof setTimeout> | undefined;
+        try {
+          const healthy = await Promise.race([
+            Promise.resolve(driver.checkHealth()),
+            new Promise<never>((_, reject) => {
+              timer = setTimeout(
+                () => reject(new Error(`checkHealth did not settle within ${timeoutMs}ms`)),
+                timeoutMs,
+              );
+              // Never hold the event loop open for a probe.
+              (timer as { unref?: () => void }).unref?.();
+            }),
+          ]);
+          return healthy
+            ? { driverName, healthy: true }
+            : { driverName, healthy: false, error: 'checkHealth() returned false' };
+        } catch (e) {
+          return {
+            driverName,
+            healthy: false,
+            error: e instanceof Error ? e.message : String(e),
+          };
+        } finally {
+          if (timer) clearTimeout(timer);
+        }
+      }),
+    );
+  }
+
   /**
    * Does this object declare a media field at all? Cached per schema object —
    * the registry hands back the same instance per object, so this scans a

packages/objectql/src/index.ts

@@ -39,7 +39,7 @@ export type { SummaryRecomputeFailure } from './summary-errors.js';
 // fails (framework#3741). Hosts that boot the engine themselves can catch it to
 // render their own "database unreachable" message.
 export { DriverConnectError } from './driver-connect-errors.js';
-export type { DriverConnectFailure } from './driver-connect-errors.js';
+export type { DriverConnectFailure, DriverHealth } from './driver-connect-errors.js';
 export type { InsertManyRowOutcome } from './engine.js';
 
 // Export in-memory aggregation fallback (used by engine.aggregate when the