diff --git a/.changeset/mongodb-single-tenant-boot-guard.md b/.changeset/mongodb-single-tenant-boot-guard.md index 222246dadb..c8969b2ef9 100644 --- a/.changeset/mongodb-single-tenant-boot-guard.md +++ b/.changeset/mongodb-single-tenant-boot-guard.md @@ -22,10 +22,11 @@ Rather than serve unisolated, the driver now fails fast at startup: `isolated`, including the posture derived from `OS_MULTI_ORG_ENABLED=true`), resolved through the shared `resolveTenancyPosture()` so the driver can never disagree with auth / the registry / the CLI about the mode. The check sits in - the constructor and not only in `connect()` because `ObjectQLEngine.init()` - *catches* a driver's connect rejection and boots anyway — a connect-only guard - would have degraded "refuses to start" into "starts, then throws at query - time". `connect()` re-checks in case a host flips the posture in between. + the constructor because that is the earliest seam — it fails before a host can + hand the driver anywhere — and `connect()` re-checks in case a host flips the + posture in between. (It originally had to live in the constructor because + `ObjectQLEngine.init()` *caught* a driver's connect rejection and booted + anyway; that is fixed in the same release, #3741, so both seams abort boot.) - `syncSchema()` / `syncSchemasBatch()` call `assertObjectsNotTenantScoped()` and refuse objects declaring `tenancy.enabled: true`, naming every offender in one message. diff --git a/.changeset/objectql-driver-connect-failfast.md b/.changeset/objectql-driver-connect-failfast.md new file mode 100644 index 0000000000..ea4c258cec --- /dev/null +++ b/.changeset/objectql-driver-connect-failfast.md @@ -0,0 +1,53 @@ +--- +"@objectstack/objectql": minor +"@objectstack/types": minor +"@objectstack/driver-mongodb": patch +--- + +feat(objectql)!: `init()` refuses to boot when a data driver fails to connect (#3741) + +`ObjectQLEngine.init()` wrapped every driver's `connect()` in a try/catch, logged +one error line, and carried on. A server whose database was unreachable therefore +"started successfully" — health endpoints could even stay green — and then failed +every request with an error that reads nothing like *the database is down*. The +warning it printed (`Operations may recover via lazy reconnection or fail at query +time`) was half fiction: grep the repo and no reconnection exists in `driver-sql` +or `driver-mongodb`, so only the "fail at query time" half was ever real. The +caller made it worse — `ObjectQLPlugin.start()` runs `syncRegisteredSchemas()` +immediately after `init()`, issuing DDL against a driver that isn't there. + +The structural half of the bug was worse than the operational one: the catch +removed a driver's ability to **refuse startup at all**. Any fatal startup check — +licence, server version, incompatible configuration, missing capability, not just +an unreachable socket — is expressed by throwing from `connect()`, and every one +of them was silently downgraded to a runtime error. That is why driver-mongodb's +multi-tenancy guard (#3724 / #3734) had to be hoisted into its constructor. + +- `init()` now **throws** `DriverConnectError` (`code: 'ERR_DRIVER_CONNECT'`) + when any boot-registered driver's `connect()` rejects, aborting kernel + bootstrap. It still attempts every driver first, so one failed boot names all + of them. The message is self-contained — each failed driver and its cause — + because the CLI prints `error.message` alone; the first cause is also attached + as `error.cause`. Exported from both `@objectstack/objectql` and + `@objectstack/objectql/core`. +- `connect()` is now a supported place for a driver to veto boot. Startup + validation that needs a live connection (server version, capability probes) + no longer has to be forced into a constructor. +- The misleading "lazy reconnection" warning is gone. +- New escape hatch `OS_ALLOW_DRIVER_CONNECT_FAILURE=1` + (`resolveAllowDriverConnectFailure()` in `@objectstack/types`) restores the old + lenient boot, but loudly: a `DEGRADED BOOT` banner names the failed drivers and + states that they are never retried or reconnected and that every query and + schema sync routed to them will fail for the process lifetime. The banner goes + to stderr as well as the logger, because `os serve` swallows all of stdout + during boot and `Logger` routes `warn` there — logger-only, the one message + that matters would be invisible in exactly the deployment the flag is for. + Defaults off. + +**Migration.** No code or config change is needed for a correctly configured +deployment — a driver that connected before still connects. A deployment that was +*silently* booting without its database now fails the boot instead, with the +driver name and cause in the error; fix the datasource configuration (typically +`OS_DATABASE_URL`, credentials, or network reachability). To keep booting without +it — deliberately, and knowing every request that touches it will fail — set +`OS_ALLOW_DRIVER_CONNECT_FAILURE=1`. diff --git a/content/docs/data-modeling/drivers.mdx b/content/docs/data-modeling/drivers.mdx index 6b1d7cbe9b..2ed7381b33 100644 --- a/content/docs/data-modeling/drivers.mdx +++ b/content/docs/data-modeling/drivers.mdx @@ -71,6 +71,49 @@ actually connect to Turso. > Knex client name (`pg` / `mysql2` / `better-sqlite3`) when you instantiate > `SqlDriver`. +## Startup: a driver that cannot connect aborts the boot + +`ObjectQLEngine.init()` connects every registered driver during kernel +bootstrap. If any `connect()` rejects, the boot is **refused** +([#3741](https://github.com/objectstack-ai/objectstack/issues/3741)) — the error +names each failed driver and its cause, and `objectstack serve` exits 1. + +There is no lazy reconnection. A driver that did not connect at startup stays +disconnected for the process lifetime, so booting anyway would produce a server +that reports itself started, answers health checks, and then fails every request +with an error nothing like *the database is unreachable* — while the schema sync +that follows `init()` issues DDL against a datasource that isn't there. + + +`OS_ALLOW_DRIVER_CONNECT_FAILURE=1` boots anyway, in an explicitly degraded +state announced by a `DEGRADED BOOT` banner. The failed drivers are never +retried and never reconnected: every query and every schema sync routed to them +fails until the process restarts. Do not set it in production. + + +### Writing a driver: `connect()` is where you refuse to start + +Because the rejection now propagates, throwing from `connect()` is the supported +way for a driver to **veto the boot** — not only for an unreachable socket, but +for any fatal startup condition: an unsupported server version, a missing +capability, an incompatible deployment mode, a licence check. Validation that +needs a live connection belongs here rather than in the constructor. + +```typescript +async connect(): Promise { + await this.pool.connect(); + const { version } = await this.probeServerVersion(); + if (major(version) < 14) { + // Aborts bootstrap — the operator sees this message, not a 500 per request. + throw new Error(`PostgreSQL ${version} is unsupported; this driver requires 14+.`); + } +} +``` + +Checks that need no connection can still run in the constructor, which fails +even earlier — that is where `MongoDBDriver` puts its +[tenancy guard](#multi-tenancy-not-supported). + ## PostgreSQL (via `@objectstack/driver-sql`) ```bash diff --git a/content/docs/deployment/environment-variables.mdx b/content/docs/deployment/environment-variables.mdx index 8d234c9125..83da5b19eb 100644 --- a/content/docs/deployment/environment-variables.mdx +++ b/content/docs/deployment/environment-variables.mdx @@ -49,6 +49,7 @@ read at startup unless noted otherwise. Boolean variables accept `true` / `false |:---|:---|:---|:---| | `OS_DATABASE_URL` | url | — | Database connection string (e.g. `file:./data.sqlite`, `postgres://…`, `mongodb://…`, `memory://`). `libsql://` (Turso) is not supported. | | `OS_DATABASE_DRIVER` | enum | inferred | Force a specific driver when the URL is ambiguous. `memory` \| `sqlite` \| `sqlite-wasm` \| `postgres` \| `mongodb`. | +| `OS_ALLOW_DRIVER_CONNECT_FAILURE` | boolean | `false` | Escape hatch for the driver-connect boot guard. By default a data driver that fails to connect at startup **refuses the boot** — a server that cannot reach its database must not report itself started and then fail every request. Set to `1` to boot anyway, in an explicitly degraded state logged loudly at startup. There is **no reconnection**: the drivers that failed stay dead for the process lifetime and every query and schema sync routed to them fails. | | `OS_STORAGE_ROOT` | path | `./.objectstack/data/uploads` | Root directory for the local file storage adapter, relative to the process cwd (used by `os serve`'s default `storage` capability wiring). | | `OS_ARTIFACT_PATH` | path | — | Path or `http(s)://` URL to a compiled `objectstack.json` artifact to boot the kernel from. | diff --git a/content/docs/deployment/production-readiness.mdx b/content/docs/deployment/production-readiness.mdx index d56754a147..c27d9fdf39 100644 --- a/content/docs/deployment/production-readiness.mdx +++ b/content/docs/deployment/production-readiness.mdx @@ -87,6 +87,11 @@ the [HARDENING.md recipes](https://github.com/objectstack-ai/objectstack/blob/ma deployment requesting multi-org without `@objectstack/organizations` now refuses to boot unless `OS_ALLOW_DEGRADED_TENANCY=1`; never set that flag in production. See [Tenancy Modes & Membership](/docs/deployment/tenancy-modes). +- [ ] `OS_ALLOW_DRIVER_CONNECT_FAILURE` is **unset**. A data driver that cannot + connect at startup refuses the boot, so a bad `OS_DATABASE_URL`, a rotated + password, or a closed network path surfaces as a failed deploy instead of + a "started" server that 500s every request. Setting the flag boots without + the database and never reconnects; never set it in production. - [ ] Backup / restore drill documented and tested. - [ ] Data-retention windows reviewed (ADR-0057): the platform's default `lifecycle` declarations bound telemetry (activity 14d, job runs 30d, diff --git a/packages/objectql/src/core.ts b/packages/objectql/src/core.ts index 85bc050f60..d154ad8b69 100644 --- a/packages/objectql/src/core.ts +++ b/packages/objectql/src/core.ts @@ -39,6 +39,12 @@ export type { CompanionFieldMeta, CompanionObjectMeta } from './search-companion export { ObjectQL, ObjectRepository, ScopedContext } from './engine.js'; export type { ObjectQLHostContext, HookHandler, HookEntry, OperationContext, EngineMiddleware } from './engine.js'; +// Boot guard: thrown by `ObjectQL.init()` when a registered driver's connect() +// fails (framework#3741). Embedders that boot the engine themselves can catch +// it to render their own "database unreachable" message. +export { DriverConnectError } from './driver-connect-errors.js'; +export type { DriverConnectFailure } from './driver-connect-errors.js'; + // In-memory aggregation fallback export { applyInMemoryAggregation, bucketDateValue } from './in-memory-aggregation.js'; diff --git a/packages/objectql/src/driver-connect-errors.ts b/packages/objectql/src/driver-connect-errors.ts new file mode 100644 index 0000000000..52d2e1dd05 --- /dev/null +++ b/packages/objectql/src/driver-connect-errors.ts @@ -0,0 +1,107 @@ +// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. + +/** One boot-registered driver whose `connect()` rejected during engine init. */ +export interface DriverConnectFailure { + driverName: string; + error: unknown; +} + +/** `error.message` when it is an Error, its string form otherwise. */ +function failureMessage(error: unknown): string { + if (error instanceof Error) return error.message || error.name; + return String(error); +} + +/** + * Thrown by `ObjectQL.init()` when one or more boot-registered drivers fail to + * connect (framework#3741). Aborts kernel bootstrap: a server that cannot reach + * its database must not report itself started. + * + * Two failures are collapsed into one class on purpose, because `init()` cannot + * tell them apart and the correct response to both is the same — don't boot: + * + * - the datasource is genuinely unreachable (wrong `OS_DATABASE_URL`, rotated + * password, closed network path), and + * - the driver is DELIBERATELY REFUSING to start (licence, server version, + * incompatible configuration, missing capability). Throwing from `connect()` + * is the supported way for a driver to veto boot; before this error existed, + * such a veto was caught and downgraded to a query-time error, which is why + * driver-mongodb's tenancy guard had to be hoisted into its constructor + * (#3724 / #3734). + * + * The message is self-contained — it names every failed driver and its cause — + * because the CLI prints `error.message` alone (stack only under `DEBUG`). + * Identified by `code` rather than `instanceof` so it survives crossing package + * boundaries. + */ +export class DriverConnectError extends Error { + readonly code = 'ERR_DRIVER_CONNECT' as const; + + /** + * The first failure's `Error`, so its stack stays reachable for `DEBUG` + * output and structured logging. Declared here rather than inherited: the + * repo compiles against `lib: ES2020`, which predates `Error.cause`. + */ + readonly cause?: unknown; + + constructor( + public readonly failures: DriverConnectFailure[], + public readonly totalDrivers: number, + ) { + const detail = failures + .map((f) => ` • ${f.driverName}: ${failureMessage(f.error)}`) + .join('\n'); + super( + `${failures.length} of ${totalDrivers} data driver(s) failed to connect — refusing to boot.\n` + + `${detail}\n` + + `A driver that did not connect cannot serve queries (there is no lazy reconnection) and the ` + + `schema sync that runs right after init would issue DDL against it. Fix the datasource ` + + `configuration (e.g. OS_DATABASE_URL), or set OS_ALLOW_DRIVER_CONNECT_FAILURE=1 to boot anyway ` + + `in an explicitly degraded state where every query to those drivers fails.`, + ); + this.name = 'DriverConnectError'; + if (failures[0]?.error instanceof Error) { + (this as { cause?: unknown }).cause = failures[0].error; + } + } + + /** Names of the drivers that failed, in registration order. */ + get failedDrivers(): string[] { + return this.failures.map((f) => f.driverName); + } +} + +/** + * Emit the degraded-boot banner on a channel the host cannot accidentally + * silence. + * + * `OS_ALLOW_DRIVER_CONNECT_FAILURE` only justifies itself if the state it opts + * into is impossible to miss — and a logger-only banner is missable: `os serve` + * swallows ALL of stdout while the kernel boots (its "boot-quiet" capture), and + * `Logger` routes `warn` to stdout, so the one message that matters would be + * invisible in exactly the situation it exists for. Writing to stderr as well + * is the same belt-and-braces the kernel already uses for plugin startup + * failures. + * + * Best-effort and never throws: falls back to `console.error`, then to silence + * on runtimes that have neither (the logger still carries the structured + * record either way). + */ +export function emitDegradedBootBanner(message: string): void { + const proc = (globalThis as { + process?: { stderr?: { write?: (chunk: string) => unknown } }; + }).process; + try { + if (typeof proc?.stderr?.write === 'function') { + proc.stderr.write(`${message}\n`); + return; + } + } catch { + /* stderr unavailable / closed — fall through to console */ + } + try { + (globalThis as { console?: { error?: (msg: string) => void } }).console?.error?.(message); + } catch { + /* no output channel at all — the logger record is the remaining trace */ + } +} diff --git a/packages/objectql/src/engine-driver-connect-failfast.test.ts b/packages/objectql/src/engine-driver-connect-failfast.test.ts new file mode 100644 index 0000000000..987f3b2937 --- /dev/null +++ b/packages/objectql/src/engine-driver-connect-failfast.test.ts @@ -0,0 +1,170 @@ +// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. +// +// framework#3741: `ObjectQL.init()` used to try/catch every driver's connect() +// and boot anyway, so a server that could not reach its database still reported +// itself started (and then 500'd every request), and a driver had no way to +// veto boot — any fatal startup check thrown from connect() was downgraded to a +// runtime error. init() now fails fast; the lenient path is opt-in via +// OS_ALLOW_DRIVER_CONNECT_FAILURE. + +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { ObjectQL } from './engine.js'; +import { DriverConnectError } from './driver-connect-errors.js'; + +/** Minimal driver stub — only `connect` behaviour matters for init(). */ +function makeDriver(name: string, connect: () => Promise) { + let connected = false; + const driver: any = { + name, + version: '0.0.0', + supports: {}, + get connected() { return connected; }, + async connect() { await connect(); connected = true; }, + async disconnect() { connected = false; }, + async checkHealth() { return connected; }, + async execute() { return null; }, + async find() { return []; }, + findStream() { throw new Error('ni'); }, + async findOne() { return null; }, + async create(_o: string, data: Record) { return { id: 'r_1', ...data }; }, + async update(_o: string, id: string, data: Record) { return { ...data, id }; }, + async delete() { return true; }, + async count() { return 0; }, + async bulkCreate() { return []; }, + async bulkUpdate() { return []; }, + async bulkDelete() {}, + async beginTransaction() { return { __trx: true, commit: async () => {}, rollback: async () => {} }; }, + async commit() {}, async rollback() {}, + }; + return driver; +} + +const ok = (name: string) => makeDriver(name, async () => {}); +const fails = (name: string, message: string) => + makeDriver(name, async () => { throw new Error(message); }); + +describe('ObjectQL.init() — driver connect fail-fast (framework#3741)', () => { + const ENV = 'OS_ALLOW_DRIVER_CONNECT_FAILURE'; + let saved: string | undefined; + + beforeEach(() => { saved = process.env[ENV]; delete process.env[ENV]; }); + afterEach(() => { + if (saved === undefined) delete process.env[ENV]; + else process.env[ENV] = saved; + }); + + it('throws DriverConnectError when the only driver cannot connect', async () => { + const engine = new ObjectQL(); + engine.registerDriver(fails('sql', 'connect ECONNREFUSED 127.0.0.1:5432'), true); + + await expect(engine.init()).rejects.toBeInstanceOf(DriverConnectError); + }); + + it('names every failed driver and its cause in the message (the CLI prints message only)', async () => { + const engine = new ObjectQL(); + engine.registerDriver(fails('sql', 'connect ECONNREFUSED 127.0.0.1:5432'), true); + engine.registerDriver(ok('memory')); + engine.registerDriver(fails('mongodb', 'multi-tenant mode is not supported')); + + const err = await engine.init().then( + () => { throw new Error('init() resolved but should have thrown'); }, + (e: unknown) => e as DriverConnectError, + ); + + expect(err).toBeInstanceOf(DriverConnectError); + expect(err.code).toBe('ERR_DRIVER_CONNECT'); + expect(err.failedDrivers).toEqual(['sql', 'mongodb']); + expect(err.totalDrivers).toBe(3); + expect(err.message).toContain('2 of 3'); + expect(err.message).toContain('sql: connect ECONNREFUSED 127.0.0.1:5432'); + expect(err.message).toContain('mongodb: multi-tenant mode is not supported'); + expect(err.message).toContain('OS_ALLOW_DRIVER_CONNECT_FAILURE=1'); + // First failure is reachable as `cause` for DEBUG / structured logging. + expect((err.cause as Error)?.message).toBe('connect ECONNREFUSED 127.0.0.1:5432'); + }); + + it('lets a driver veto boot by throwing a semantic (non-network) error from connect()', async () => { + // The #3734 case: driver-mongodb refuses multi-tenant mode. Before this fix + // the veto was caught and downgraded, which is why the guard had to live in + // the constructor. + const engine = new ObjectQL(); + engine.registerDriver( + fails('mongodb', 'MongoDriver refuses to start: OS_TENANCY_POSTURE=isolated is not supported'), + true, + ); + + await expect(engine.init()).rejects.toThrow(/refuses to start/); + }); + + it('still connects every healthy driver and resolves when none fail', async () => { + const engine = new ObjectQL(); + const a = ok('sql'); + const b = ok('memory'); + engine.registerDriver(a, true); + engine.registerDriver(b); + + await expect(engine.init()).resolves.toBeUndefined(); + expect(a.connected).toBe(true); + expect(b.connected).toBe(true); + }); + + it('attempts every driver before throwing (does not stop at the first failure)', async () => { + const engine = new ObjectQL(); + const healthy = ok('memory'); + engine.registerDriver(fails('sql', 'down'), true); + engine.registerDriver(healthy); + + await expect(engine.init()).rejects.toBeInstanceOf(DriverConnectError); + expect(healthy.connected).toBe(true); + }); + + it('boots degraded when OS_ALLOW_DRIVER_CONNECT_FAILURE opts in, and says so loudly', async () => { + process.env[ENV] = '1'; + const warnings: string[] = []; + const engine = new ObjectQL({ + logger: { + debug() {}, info() {}, error() {}, + warn: (msg: string) => { warnings.push(msg); }, + }, + } as any); + engine.registerDriver(fails('sql', 'down'), true); + + await expect(engine.init()).resolves.toBeUndefined(); + const warned = warnings.join('\n'); + expect(warned).toContain('DEGRADED BOOT'); + expect(warned).toContain('NOT reconnected'); + expect(warned).toContain('sql'); + }); + + it('repeats the degraded banner on stderr, which `os serve` boot-quiet cannot swallow', async () => { + // `os serve` replaces process.stdout.write for the whole boot, and Logger + // sends `warn` to stdout — so a logger-only banner is invisible in exactly + // the deployment this flag exists for. + process.env[ENV] = '1'; + const written: string[] = []; + const realWrite = process.stderr.write; + (process.stderr as { write: unknown }).write = (chunk: any) => { + written.push(String(chunk)); + return true; + }; + try { + const engine = new ObjectQL({ + logger: { debug() {}, info() {}, warn() {}, error() {} }, + } as any); + engine.registerDriver(fails('sql', 'down'), true); + await engine.init(); + } finally { + (process.stderr as { write: unknown }).write = realWrite; + } + + expect(written.join('')).toContain('DEGRADED BOOT'); + }); + + it('treats a falsy opt-in value as off — still fail-fast', async () => { + process.env[ENV] = 'false'; + const engine = new ObjectQL(); + engine.registerDriver(fails('sql', 'down'), true); + + await expect(engine.init()).rejects.toBeInstanceOf(DriverConnectError); + }); +}); diff --git a/packages/objectql/src/engine.ts b/packages/objectql/src/engine.ts index ede1d5e1f2..129359f782 100644 --- a/packages/objectql/src/engine.ts +++ b/packages/objectql/src/engine.ts @@ -21,6 +21,8 @@ import { import { ExecutionContext, ExecutionContextSchema } from '@objectstack/spec/kernel'; import { IDataDriver, IDataEngine, Logger, createLogger, withTransientRetry, type RetryOptions } from '@objectstack/core'; import { SummaryRecomputeError, type SummaryRecomputeFailure } from './summary-errors.js'; +import { DriverConnectError, emitDegradedBootBanner, type DriverConnectFailure } from './driver-connect-errors.js'; +import { resolveAllowDriverConnectFailure } from '@objectstack/types'; /** * Per-row outcome of {@link ObjectQL.insertMany} (framework#3172). One entry @@ -1859,33 +1861,65 @@ export class ObjectQL implements IDataEngine { } /** - * Initialize the engine and all registered drivers + * Initialize the engine and all registered drivers. + * + * **Fail-fast by default** (framework#3741): if any boot-registered driver's + * `connect()` rejects, this throws {@link DriverConnectError} and kernel + * bootstrap aborts. Two reasons, both load-bearing: + * + * 1. A driver that did not connect never recovers — there is no lazy + * reconnection anywhere in `driver-sql` / `driver-mongodb`. Booting + * anyway produces a server that reports itself started, may even answer + * health checks, and then 500s every request with an error that reads + * nothing like "the database is unreachable". The caller immediately + * makes it worse: `ObjectQLPlugin.start()` runs `syncRegisteredSchemas()` + * right after this, issuing DDL against a driver that isn't there. + * 2. Swallowing the rejection removed a driver's ability to REFUSE STARTUP. + * Any fatal startup check — licence, server version, incompatible + * configuration, missing capability, not just an unreachable socket — is + * expressed by throwing from `connect()`, and every one of them used to + * be silently downgraded to a runtime error. (That is why + * driver-mongodb's multi-tenancy guard had to be hoisted into its + * constructor in #3734; `connect()` is a supported place for it now.) + * + * Operators who need the old lenient behaviour opt in explicitly with + * `OS_ALLOW_DRIVER_CONNECT_FAILURE=1`, which boots in a state that is warned + * about loudly rather than assumed. */ async init() { - this.logger.info('Initializing ObjectQL engine', { + this.logger.info('Initializing ObjectQL engine', { driverCount: this.drivers.size, drivers: Array.from(this.drivers.keys()) }); - - const failedDrivers: string[] = []; + + const failures: DriverConnectFailure[] = []; for (const [name, driver] of this.drivers) { try { await driver.connect(); this.logger.info('Driver connected successfully', { driverName: name }); } catch (e) { - failedDrivers.push(name); + failures.push({ driverName: name, error: e }); this.logger.error('Failed to connect driver', e as Error, { driverName: name }); } } - if (failedDrivers.length > 0) { - this.logger.warn( - `${failedDrivers.length} of ${this.drivers.size} driver(s) failed initial connect. ` + - `Operations may recover via lazy reconnection or fail at query time.`, - { failedDrivers } - ); + if (failures.length > 0) { + if (!resolveAllowDriverConnectFailure()) { + throw new DriverConnectError(failures, this.drivers.size); + } + const failedDrivers = failures.map(f => f.driverName); + const banner = + `⚠️ DEGRADED BOOT: ${failures.length} of ${this.drivers.size} driver(s) failed to connect ` + + `(${failedDrivers.join(', ')}), but OS_ALLOW_DRIVER_CONNECT_FAILURE is set — starting anyway. ` + + `These drivers are NOT retried and NOT reconnected: every query and every schema sync routed ` + + `to them fails for the lifetime of this process. Unset OS_ALLOW_DRIVER_CONNECT_FAILURE to ` + + `restore fail-fast boot.`; + this.logger.warn(banner, { failedDrivers }); + // …and again on a channel the host cannot silence — see the helper's note + // on `os serve`'s boot-quiet stdout capture. + emitDegradedBootBanner(banner); } - + this.logger.info('ObjectQL engine initialization complete'); } diff --git a/packages/objectql/src/index.ts b/packages/objectql/src/index.ts index 7273bce60a..ce81f6446c 100644 --- a/packages/objectql/src/index.ts +++ b/packages/objectql/src/index.ts @@ -35,6 +35,11 @@ export { ObjectQL, ObjectRepository, ScopedContext } from './engine.js'; export type { ObjectQLHostContext, HookHandler, HookEntry, OperationContext, EngineMiddleware } from './engine.js'; export { SummaryRecomputeError } from './summary-errors.js'; export type { SummaryRecomputeFailure } from './summary-errors.js'; +// Boot guard: thrown by `ObjectQL.init()` when a registered driver's connect() +// fails (framework#3741). Hosts that boot the engine themselves can catch it to +// render their own "database unreachable" message. +export { DriverConnectError } from './driver-connect-errors.js'; +export type { DriverConnectFailure } from './driver-connect-errors.js'; export type { InsertManyRowOutcome } from './engine.js'; // Export in-memory aggregation fallback (used by engine.aggregate when the diff --git a/packages/plugins/driver-mongodb/src/mongodb-driver.ts b/packages/plugins/driver-mongodb/src/mongodb-driver.ts index b85d0ed48e..65918eb552 100644 --- a/packages/plugins/driver-mongodb/src/mongodb-driver.ts +++ b/packages/plugins/driver-mongodb/src/mongodb-driver.ts @@ -129,11 +129,11 @@ export class MongoDBDriver implements IDataDriver { constructor(config: MongoDBDriverConfig) { // Refuse to even EXIST in a multi-tenant deployment (#3724). The check is - // repeated in `connect()`, but construction is the only seam guaranteed to - // fail loudly: `ObjectQLEngine.init()` catches a driver's connect rejection - // and logs it, then boots anyway ("may recover via lazy reconnection"), so - // a connect-only guard would degrade from "refuses to start" to "starts, - // then throws at query time". + // repeated in `connect()`; construction just fails earliest, before a host + // can hand this driver to anything. (Originally the constructor was the + // ONLY seam that failed loudly, because `ObjectQLEngine.init()` caught a + // driver's connect rejection and booted anyway — fixed in framework#3741, + // so `connect()` now aborts boot too.) assertSingleTenantPosture(); this.config = config; diff --git a/packages/plugins/driver-mongodb/src/mongodb-tenancy-guard.test.ts b/packages/plugins/driver-mongodb/src/mongodb-tenancy-guard.test.ts index f9ea6990dd..db14011e98 100644 --- a/packages/plugins/driver-mongodb/src/mongodb-tenancy-guard.test.ts +++ b/packages/plugins/driver-mongodb/src/mongodb-tenancy-guard.test.ts @@ -145,8 +145,8 @@ describe('multi-tenancy boot guard (#3724)', () => { describe('MongoDBDriver wiring', () => { it('the constructor refuses in multi-tenant mode', () => { process.env.OS_MULTI_ORG_ENABLED = 'true'; - // Construction is the seam that fails loudly: ObjectQLEngine.init() - // catches a connect() rejection and boots anyway. + // Construction is the earliest seam — it fails before a host can hand + // the driver anywhere. (`connect()` also aborts boot since #3741.) expect(() => makeDriver()).toThrow(MongoDBMultiTenantUnsupportedError); }); diff --git a/packages/plugins/driver-mongodb/src/mongodb-tenancy-guard.ts b/packages/plugins/driver-mongodb/src/mongodb-tenancy-guard.ts index 233c0a9309..707c761bd6 100644 --- a/packages/plugins/driver-mongodb/src/mongodb-tenancy-guard.ts +++ b/packages/plugins/driver-mongodb/src/mongodb-tenancy-guard.ts @@ -22,9 +22,12 @@ * 1. The deployment's tenancy posture is not `single` (deployment-level signal) * → {@link assertSingleTenantPosture}, called from the `MongoDBDriver` * **constructor** and re-checked in `connect()` before a socket is opened. - * The constructor is the seam that fails loudly: `ObjectQLEngine.init()` - * catches a driver's connect rejection and boots anyway, so a connect-only - * guard would degrade into a query-time failure. + * Both seams now abort boot: `ObjectQLEngine.init()` propagates a driver's + * connect rejection since framework#3741 (it used to catch it and boot + * anyway, which is why the check was hoisted into the constructor). The + * constructor check is kept because it fails earliest — before a host can + * hand the driver to anything — and the `connect()` re-check covers a host + * that flips the posture between construction and connect. * 2. An object declares `tenancy.enabled: true` (metadata-level signal) → * {@link assertObjectsNotTenantScoped}, called from the `syncSchema` / * `syncSchemasBatch` paths. diff --git a/packages/types/src/env.ts b/packages/types/src/env.ts index ed97d054ee..c507351260 100644 --- a/packages/types/src/env.ts +++ b/packages/types/src/env.ts @@ -166,6 +166,29 @@ export function resolveAllowDegradedTenancy(): boolean { return ['1', 'true', 'on', 'yes'].includes(String(raw).trim().toLowerCase()); } +/** + * Escape hatch for the driver-connect boot guard (framework#3741). + * + * `ObjectQLEngine.init()` connects every boot-registered driver and, by + * default, refuses to boot when any of them fails — a server whose database is + * unreachable must not report itself started and then 500 every request with an + * error that reads nothing like "the database is down". Failing there is also + * what gives a driver the ability to REFUSE STARTUP at all: any fatal startup + * check a driver wants to run (licence, server version, incompatible + * configuration, missing capability) can simply throw from `connect()`. + * + * Setting this to a truthy value (`true`/`1`/`on`/`yes`, case-insensitive) + * boots anyway, in an explicitly degraded state that is logged loudly at + * startup. There is NO reconnection: the drivers that failed stay dead for the + * process lifetime and every query routed to them fails. Defaults OFF — an + * unset flag means "fail fast". + */ +export function resolveAllowDriverConnectFailure(): boolean { + const raw = readEnvWithDeprecation('OS_ALLOW_DRIVER_CONNECT_FAILURE', [], { silent: true }); + if (raw == null) return false; + return ['1', 'true', 'on', 'yes'].includes(String(raw).trim().toLowerCase()); +} + /** * SINGLE decision point for "is the MCP HTTP surface (`/api/v1/mcp`) on?". *