feat(objectql)!: refuse to boot when a data driver fails to connect (#3741)

.changeset/mongodb-single-tenant-boot-guard.md

@@ -22,10 +22,11 @@ Rather than serve unisolated, the driver now fails fast at startup:
   `isolated`, including the posture derived from `OS_MULTI_ORG_ENABLED=true`),
   resolved through the shared `resolveTenancyPosture()` so the driver can never
   disagree with auth / the registry / the CLI about the mode. The check sits in
-  the constructor and not only in `connect()` because `ObjectQLEngine.init()`
-  *catches* a driver's connect rejection and boots anyway — a connect-only guard
-  would have degraded "refuses to start" into "starts, then throws at query
-  time". `connect()` re-checks in case a host flips the posture in between.
+  the constructor because that is the earliest seam — it fails before a host can
+  hand the driver anywhere — and `connect()` re-checks in case a host flips the
+  posture in between. (It originally had to live in the constructor because
+  `ObjectQLEngine.init()` *caught* a driver's connect rejection and booted
+  anyway; that is fixed in the same release, #3741, so both seams abort boot.)
 - `syncSchema()` / `syncSchemasBatch()` call `assertObjectsNotTenantScoped()` and
   refuse objects declaring `tenancy.enabled: true`, naming every offender in one
   message.

.changeset/objectql-driver-connect-failfast.md

@@ -0,0 +1,53 @@
+---
+"@objectstack/objectql": minor
+"@objectstack/types": minor
+"@objectstack/driver-mongodb": patch
+---
+
+feat(objectql)!: `init()` refuses to boot when a data driver fails to connect (#3741)
+
+`ObjectQLEngine.init()` wrapped every driver's `connect()` in a try/catch, logged
+one error line, and carried on. A server whose database was unreachable therefore
+"started successfully" — health endpoints could even stay green — and then failed
+every request with an error that reads nothing like *the database is down*. The
+warning it printed (`Operations may recover via lazy reconnection or fail at query
+time`) was half fiction: grep the repo and no reconnection exists in `driver-sql`
+or `driver-mongodb`, so only the "fail at query time" half was ever real. The
+caller made it worse — `ObjectQLPlugin.start()` runs `syncRegisteredSchemas()`
+immediately after `init()`, issuing DDL against a driver that isn't there.
+
+The structural half of the bug was worse than the operational one: the catch
+removed a driver's ability to **refuse startup at all**. Any fatal startup check —
+licence, server version, incompatible configuration, missing capability, not just
+an unreachable socket — is expressed by throwing from `connect()`, and every one
+of them was silently downgraded to a runtime error. That is why driver-mongodb's
+multi-tenancy guard (#3724 / #3734) had to be hoisted into its constructor.
+
+- `init()` now **throws** `DriverConnectError` (`code: 'ERR_DRIVER_CONNECT'`)
+  when any boot-registered driver's `connect()` rejects, aborting kernel
+  bootstrap. It still attempts every driver first, so one failed boot names all
+  of them. The message is self-contained — each failed driver and its cause —
+  because the CLI prints `error.message` alone; the first cause is also attached
+  as `error.cause`. Exported from both `@objectstack/objectql` and
+  `@objectstack/objectql/core`.
+- `connect()` is now a supported place for a driver to veto boot. Startup
+  validation that needs a live connection (server version, capability probes)
+  no longer has to be forced into a constructor.
+- The misleading "lazy reconnection" warning is gone.
+- New escape hatch `OS_ALLOW_DRIVER_CONNECT_FAILURE=1`
+  (`resolveAllowDriverConnectFailure()` in `@objectstack/types`) restores the old
+  lenient boot, but loudly: a `DEGRADED BOOT` banner names the failed drivers and
+  states that they are never retried or reconnected and that every query and
+  schema sync routed to them will fail for the process lifetime. The banner goes
+  to stderr as well as the logger, because `os serve` swallows all of stdout
+  during boot and `Logger` routes `warn` there — logger-only, the one message
+  that matters would be invisible in exactly the deployment the flag is for.
+  Defaults off.
+
+**Migration.** No code or config change is needed for a correctly configured
+deployment — a driver that connected before still connects. A deployment that was
+*silently* booting without its database now fails the boot instead, with the
+driver name and cause in the error; fix the datasource configuration (typically
+`OS_DATABASE_URL`, credentials, or network reachability). To keep booting without
+it — deliberately, and knowing every request that touches it will fail — set
+`OS_ALLOW_DRIVER_CONNECT_FAILURE=1`.

content/docs/data-modeling/drivers.mdx

@@ -71,6 +71,49 @@ actually connect to Turso.
 > Knex client name (`pg` / `mysql2` / `better-sqlite3`) when you instantiate
 > `SqlDriver`.
 
+## Startup: a driver that cannot connect aborts the boot
+
+`ObjectQLEngine.init()` connects every registered driver during kernel
+bootstrap. If any `connect()` rejects, the boot is **refused**
+([#3741](https://github.com/objectstack-ai/objectstack/issues/3741)) — the error
+names each failed driver and its cause, and `objectstack serve` exits 1.
+
+There is no lazy reconnection. A driver that did not connect at startup stays
+disconnected for the process lifetime, so booting anyway would produce a server
+that reports itself started, answers health checks, and then fails every request
+with an error nothing like *the database is unreachable* — while the schema sync
+that follows `init()` issues DDL against a datasource that isn't there.
+
+<Callout type="warn">
+`OS_ALLOW_DRIVER_CONNECT_FAILURE=1` boots anyway, in an explicitly degraded
+state announced by a `DEGRADED BOOT` banner. The failed drivers are never
+retried and never reconnected: every query and every schema sync routed to them
+fails until the process restarts. Do not set it in production.
+</Callout>
+
+### Writing a driver: `connect()` is where you refuse to start
+
+Because the rejection now propagates, throwing from `connect()` is the supported
+way for a driver to **veto the boot** — not only for an unreachable socket, but
+for any fatal startup condition: an unsupported server version, a missing
+capability, an incompatible deployment mode, a licence check. Validation that
+needs a live connection belongs here rather than in the constructor.
+
+```typescript
+async connect(): Promise<void> {
+  await this.pool.connect();
+  const { version } = await this.probeServerVersion();
+  if (major(version) < 14) {
+    // Aborts bootstrap — the operator sees this message, not a 500 per request.
+    throw new Error(`PostgreSQL ${version} is unsupported; this driver requires 14+.`);
+  }
+}
+```
+
+Checks that need no connection can still run in the constructor, which fails
+even earlier — that is where `MongoDBDriver` puts its
+[tenancy guard](#multi-tenancy-not-supported).
+
 ## PostgreSQL (via `@objectstack/driver-sql`)
 
 ```bash

content/docs/deployment/environment-variables.mdx

@@ -49,6 +49,7 @@ read at startup unless noted otherwise. Boolean variables accept `true` / `false
 |:---|:---|:---|:---|
 | `OS_DATABASE_URL` | url | — | Database connection string (e.g. `file:./data.sqlite`, `postgres://…`, `mongodb://…`, `memory://`). `libsql://` (Turso) is not supported. |
 | `OS_DATABASE_DRIVER` | enum | inferred | Force a specific driver when the URL is ambiguous. `memory` \| `sqlite` \| `sqlite-wasm` \| `postgres` \| `mongodb`. |
+| `OS_ALLOW_DRIVER_CONNECT_FAILURE` | boolean | `false` | Escape hatch for the driver-connect boot guard. By default a data driver that fails to connect at startup **refuses the boot** — a server that cannot reach its database must not report itself started and then fail every request. Set to `1` to boot anyway, in an explicitly degraded state logged loudly at startup. There is **no reconnection**: the drivers that failed stay dead for the process lifetime and every query and schema sync routed to them fails. |
 | `OS_STORAGE_ROOT` | path | `./.objectstack/data/uploads` | Root directory for the local file storage adapter, relative to the process cwd (used by `os serve`'s default `storage` capability wiring). |
 | `OS_ARTIFACT_PATH` | path | — | Path or `http(s)://` URL to a compiled `objectstack.json` artifact to boot the kernel from. |
 

content/docs/deployment/production-readiness.mdx

@@ -87,6 +87,11 @@ the [HARDENING.md recipes](https://github.com/objectstack-ai/objectstack/blob/ma
       deployment requesting multi-org without `@objectstack/organizations` now
       refuses to boot unless `OS_ALLOW_DEGRADED_TENANCY=1`; never set that flag
       in production. See [Tenancy Modes & Membership](/docs/deployment/tenancy-modes).
+- [ ] `OS_ALLOW_DRIVER_CONNECT_FAILURE` is **unset**. A data driver that cannot
+      connect at startup refuses the boot, so a bad `OS_DATABASE_URL`, a rotated
+      password, or a closed network path surfaces as a failed deploy instead of
+      a "started" server that 500s every request. Setting the flag boots without
+      the database and never reconnects; never set it in production.
 - [ ] Backup / restore drill documented and tested.
 - [ ] Data-retention windows reviewed (ADR-0057): the platform's default
       `lifecycle` declarations bound telemetry (activity 14d, job runs 30d,

packages/objectql/src/core.ts

@@ -39,6 +39,12 @@ export type { CompanionFieldMeta, CompanionObjectMeta } from './search-companion
 export { ObjectQL, ObjectRepository, ScopedContext } from './engine.js';
 export type { ObjectQLHostContext, HookHandler, HookEntry, OperationContext, EngineMiddleware } from './engine.js';
 
+// Boot guard: thrown by `ObjectQL.init()` when a registered driver's connect()
+// fails (framework#3741). Embedders that boot the engine themselves can catch
+// it to render their own "database unreachable" message.
+export { DriverConnectError } from './driver-connect-errors.js';
+export type { DriverConnectFailure } from './driver-connect-errors.js';
+
 // In-memory aggregation fallback
 export { applyInMemoryAggregation, bucketDateValue } from './in-memory-aggregation.js';
 

packages/objectql/src/driver-connect-errors.ts

@@ -0,0 +1,107 @@
+// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license.
+
+/** One boot-registered driver whose `connect()` rejected during engine init. */
+export interface DriverConnectFailure {
+  driverName: string;
+  error: unknown;
+}
+
+/** `error.message` when it is an Error, its string form otherwise. */
+function failureMessage(error: unknown): string {
+  if (error instanceof Error) return error.message || error.name;
+  return String(error);
+}
+
+/**
+ * Thrown by `ObjectQL.init()` when one or more boot-registered drivers fail to
+ * connect (framework#3741). Aborts kernel bootstrap: a server that cannot reach
+ * its database must not report itself started.
+ *
+ * Two failures are collapsed into one class on purpose, because `init()` cannot
+ * tell them apart and the correct response to both is the same — don't boot:
+ *
+ *  - the datasource is genuinely unreachable (wrong `OS_DATABASE_URL`, rotated
+ *    password, closed network path), and
+ *  - the driver is DELIBERATELY REFUSING to start (licence, server version,
+ *    incompatible configuration, missing capability). Throwing from `connect()`
+ *    is the supported way for a driver to veto boot; before this error existed,
+ *    such a veto was caught and downgraded to a query-time error, which is why
+ *    driver-mongodb's tenancy guard had to be hoisted into its constructor
+ *    (#3724 / #3734).
+ *
+ * The message is self-contained — it names every failed driver and its cause —
+ * because the CLI prints `error.message` alone (stack only under `DEBUG`).
+ * Identified by `code` rather than `instanceof` so it survives crossing package
+ * boundaries.
+ */
+export class DriverConnectError extends Error {
+  readonly code = 'ERR_DRIVER_CONNECT' as const;
+
+  /**
+   * The first failure's `Error`, so its stack stays reachable for `DEBUG`
+   * output and structured logging. Declared here rather than inherited: the
+   * repo compiles against `lib: ES2020`, which predates `Error.cause`.
+   */
+  readonly cause?: unknown;
+
+  constructor(
+    public readonly failures: DriverConnectFailure[],
+    public readonly totalDrivers: number,
+  ) {
+    const detail = failures
+      .map((f) => `  • ${f.driverName}: ${failureMessage(f.error)}`)
+      .join('\n');
+    super(
+      `${failures.length} of ${totalDrivers} data driver(s) failed to connect — refusing to boot.\n` +
+      `${detail}\n` +
+      `A driver that did not connect cannot serve queries (there is no lazy reconnection) and the ` +
+      `schema sync that runs right after init would issue DDL against it. Fix the datasource ` +
+      `configuration (e.g. OS_DATABASE_URL), or set OS_ALLOW_DRIVER_CONNECT_FAILURE=1 to boot anyway ` +
+      `in an explicitly degraded state where every query to those drivers fails.`,
+    );
+    this.name = 'DriverConnectError';
+    if (failures[0]?.error instanceof Error) {
+      (this as { cause?: unknown }).cause = failures[0].error;
+    }
+  }
+
+  /** Names of the drivers that failed, in registration order. */
+  get failedDrivers(): string[] {
+    return this.failures.map((f) => f.driverName);
+  }
+}
+
+/**
+ * Emit the degraded-boot banner on a channel the host cannot accidentally
+ * silence.
+ *
+ * `OS_ALLOW_DRIVER_CONNECT_FAILURE` only justifies itself if the state it opts
+ * into is impossible to miss — and a logger-only banner is missable: `os serve`
+ * swallows ALL of stdout while the kernel boots (its "boot-quiet" capture), and
+ * `Logger` routes `warn` to stdout, so the one message that matters would be
+ * invisible in exactly the situation it exists for. Writing to stderr as well
+ * is the same belt-and-braces the kernel already uses for plugin startup
+ * failures.
+ *
+ * Best-effort and never throws: falls back to `console.error`, then to silence
+ * on runtimes that have neither (the logger still carries the structured
+ * record either way).
+ */
+export function emitDegradedBootBanner(message: string): void {
+  const proc = (globalThis as {
+    process?: { stderr?: { write?: (chunk: string) => unknown } };
+  }).process;
+  try {
+    if (typeof proc?.stderr?.write === 'function') {
+      proc.stderr.write(`${message}\n`);
+      return;
+    }
+  } catch {
+    /* stderr unavailable / closed — fall through to console */
+  }
+  try {
+    (globalThis as { console?: { error?: (msg: string) => void } }).console?.error?.(message);
+  } catch {
+    /* no output channel at all — the logger record is the remaining trace */
+  }
+}