Skip to content

Commit 030125b

Browse files
authored
feat(objectql)!: refuse to boot when a data driver fails to connect (#3741) (#3751)
`ObjectQLEngine.init()` wrapped every driver's `connect()` in a try/catch, logged one error line, and carried on. A server whose database was unreachable "started successfully" and then failed every request with an error that reads nothing like *the database is down*. The warning it printed — "Operations may recover via lazy reconnection or fail at query time" — was half fiction: no reconnection exists in driver-sql or driver-mongodb. The caller made it worse: `ObjectQLPlugin.start()` runs `syncRegisteredSchemas()` immediately after `init()`, issuing DDL against a driver that isn't there. The structural half was worse: the catch removed a driver's ability to refuse startup at all. Any fatal startup check (licence, server version, incompatible configuration, missing capability) is expressed by throwing from `connect()`, and every one was silently downgraded to a runtime error — which is why driver-mongodb's tenancy guard had to be hoisted into its constructor in #3734. - `init()` now throws `DriverConnectError` (`code: 'ERR_DRIVER_CONNECT'`) when any boot-registered driver's `connect()` rejects, aborting kernel bootstrap. It attempts every driver first, so one failed boot names all of them. The message is self-contained because the CLI prints `error.message` alone. - `connect()` is now a supported place for a driver to veto boot. - Removed the misleading "lazy reconnection" claim. - Escape hatch `OS_ALLOW_DRIVER_CONNECT_FAILURE=1` restores the lenient boot, defaults off, and announces itself with a DEGRADED BOOT banner written to stderr as well as the logger — `os serve` swallows all of stdout during boot and `Logger` routes `warn` there, so logger-only it would be invisible in exactly the deployment the flag is for. - Documented the connect-time boot contract for operators and driver authors in `data-modeling/drivers.mdx`.
1 parent dac6a08 commit 030125b

14 files changed

Lines changed: 477 additions & 26 deletions

.changeset/mongodb-single-tenant-boot-guard.md

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -22,10 +22,11 @@ Rather than serve unisolated, the driver now fails fast at startup:
2222
`isolated`, including the posture derived from `OS_MULTI_ORG_ENABLED=true`),
2323
resolved through the shared `resolveTenancyPosture()` so the driver can never
2424
disagree with auth / the registry / the CLI about the mode. The check sits in
25-
the constructor and not only in `connect()` because `ObjectQLEngine.init()`
26-
*catches* a driver's connect rejection and boots anyway — a connect-only guard
27-
would have degraded "refuses to start" into "starts, then throws at query
28-
time". `connect()` re-checks in case a host flips the posture in between.
25+
the constructor because that is the earliest seam — it fails before a host can
26+
hand the driver anywhere — and `connect()` re-checks in case a host flips the
27+
posture in between. (It originally had to live in the constructor because
28+
`ObjectQLEngine.init()` *caught* a driver's connect rejection and booted
29+
anyway; that is fixed in the same release, #3741, so both seams abort boot.)
2930
- `syncSchema()` / `syncSchemasBatch()` call `assertObjectsNotTenantScoped()` and
3031
refuse objects declaring `tenancy.enabled: true`, naming every offender in one
3132
message.
Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
---
2+
"@objectstack/objectql": minor
3+
"@objectstack/types": minor
4+
"@objectstack/driver-mongodb": patch
5+
---
6+
7+
feat(objectql)!: `init()` refuses to boot when a data driver fails to connect (#3741)
8+
9+
`ObjectQLEngine.init()` wrapped every driver's `connect()` in a try/catch, logged
10+
one error line, and carried on. A server whose database was unreachable therefore
11+
"started successfully" — health endpoints could even stay green — and then failed
12+
every request with an error that reads nothing like *the database is down*. The
13+
warning it printed (`Operations may recover via lazy reconnection or fail at query
14+
time`) was half fiction: grep the repo and no reconnection exists in `driver-sql`
15+
or `driver-mongodb`, so only the "fail at query time" half was ever real. The
16+
caller made it worse — `ObjectQLPlugin.start()` runs `syncRegisteredSchemas()`
17+
immediately after `init()`, issuing DDL against a driver that isn't there.
18+
19+
The structural half of the bug was worse than the operational one: the catch
20+
removed a driver's ability to **refuse startup at all**. Any fatal startup check —
21+
licence, server version, incompatible configuration, missing capability, not just
22+
an unreachable socket — is expressed by throwing from `connect()`, and every one
23+
of them was silently downgraded to a runtime error. That is why driver-mongodb's
24+
multi-tenancy guard (#3724 / #3734) had to be hoisted into its constructor.
25+
26+
- `init()` now **throws** `DriverConnectError` (`code: 'ERR_DRIVER_CONNECT'`)
27+
when any boot-registered driver's `connect()` rejects, aborting kernel
28+
bootstrap. It still attempts every driver first, so one failed boot names all
29+
of them. The message is self-contained — each failed driver and its cause —
30+
because the CLI prints `error.message` alone; the first cause is also attached
31+
as `error.cause`. Exported from both `@objectstack/objectql` and
32+
`@objectstack/objectql/core`.
33+
- `connect()` is now a supported place for a driver to veto boot. Startup
34+
validation that needs a live connection (server version, capability probes)
35+
no longer has to be forced into a constructor.
36+
- The misleading "lazy reconnection" warning is gone.
37+
- New escape hatch `OS_ALLOW_DRIVER_CONNECT_FAILURE=1`
38+
(`resolveAllowDriverConnectFailure()` in `@objectstack/types`) restores the old
39+
lenient boot, but loudly: a `DEGRADED BOOT` banner names the failed drivers and
40+
states that they are never retried or reconnected and that every query and
41+
schema sync routed to them will fail for the process lifetime. The banner goes
42+
to stderr as well as the logger, because `os serve` swallows all of stdout
43+
during boot and `Logger` routes `warn` there — logger-only, the one message
44+
that matters would be invisible in exactly the deployment the flag is for.
45+
Defaults off.
46+
47+
**Migration.** No code or config change is needed for a correctly configured
48+
deployment — a driver that connected before still connects. A deployment that was
49+
*silently* booting without its database now fails the boot instead, with the
50+
driver name and cause in the error; fix the datasource configuration (typically
51+
`OS_DATABASE_URL`, credentials, or network reachability). To keep booting without
52+
it — deliberately, and knowing every request that touches it will fail — set
53+
`OS_ALLOW_DRIVER_CONNECT_FAILURE=1`.

content/docs/data-modeling/drivers.mdx

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,49 @@ actually connect to Turso.
7171
> Knex client name (`pg` / `mysql2` / `better-sqlite3`) when you instantiate
7272
> `SqlDriver`.
7373
74+
## Startup: a driver that cannot connect aborts the boot
75+
76+
`ObjectQLEngine.init()` connects every registered driver during kernel
77+
bootstrap. If any `connect()` rejects, the boot is **refused**
78+
([#3741](https://github.com/objectstack-ai/objectstack/issues/3741)) — the error
79+
names each failed driver and its cause, and `objectstack serve` exits 1.
80+
81+
There is no lazy reconnection. A driver that did not connect at startup stays
82+
disconnected for the process lifetime, so booting anyway would produce a server
83+
that reports itself started, answers health checks, and then fails every request
84+
with an error nothing like *the database is unreachable* — while the schema sync
85+
that follows `init()` issues DDL against a datasource that isn't there.
86+
87+
<Callout type="warn">
88+
`OS_ALLOW_DRIVER_CONNECT_FAILURE=1` boots anyway, in an explicitly degraded
89+
state announced by a `DEGRADED BOOT` banner. The failed drivers are never
90+
retried and never reconnected: every query and every schema sync routed to them
91+
fails until the process restarts. Do not set it in production.
92+
</Callout>
93+
94+
### Writing a driver: `connect()` is where you refuse to start
95+
96+
Because the rejection now propagates, throwing from `connect()` is the supported
97+
way for a driver to **veto the boot** — not only for an unreachable socket, but
98+
for any fatal startup condition: an unsupported server version, a missing
99+
capability, an incompatible deployment mode, a licence check. Validation that
100+
needs a live connection belongs here rather than in the constructor.
101+
102+
```typescript
103+
async connect(): Promise<void> {
104+
await this.pool.connect();
105+
const { version } = await this.probeServerVersion();
106+
if (major(version) < 14) {
107+
// Aborts bootstrap — the operator sees this message, not a 500 per request.
108+
throw new Error(`PostgreSQL ${version} is unsupported; this driver requires 14+.`);
109+
}
110+
}
111+
```
112+
113+
Checks that need no connection can still run in the constructor, which fails
114+
even earlier — that is where `MongoDBDriver` puts its
115+
[tenancy guard](#multi-tenancy-not-supported).
116+
74117
## PostgreSQL (via `@objectstack/driver-sql`)
75118
76119
```bash

content/docs/deployment/environment-variables.mdx

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@ read at startup unless noted otherwise. Boolean variables accept `true` / `false
4949
|:---|:---|:---|:---|
5050
| `OS_DATABASE_URL` | url || Database connection string (e.g. `file:./data.sqlite`, `postgres://…`, `mongodb://…`, `memory://`). `libsql://` (Turso) is not supported. |
5151
| `OS_DATABASE_DRIVER` | enum | inferred | Force a specific driver when the URL is ambiguous. `memory` \| `sqlite` \| `sqlite-wasm` \| `postgres` \| `mongodb`. |
52+
| `OS_ALLOW_DRIVER_CONNECT_FAILURE` | boolean | `false` | Escape hatch for the driver-connect boot guard. By default a data driver that fails to connect at startup **refuses the boot** — a server that cannot reach its database must not report itself started and then fail every request. Set to `1` to boot anyway, in an explicitly degraded state logged loudly at startup. There is **no reconnection**: the drivers that failed stay dead for the process lifetime and every query and schema sync routed to them fails. |
5253
| `OS_STORAGE_ROOT` | path | `./.objectstack/data/uploads` | Root directory for the local file storage adapter, relative to the process cwd (used by `os serve`'s default `storage` capability wiring). |
5354
| `OS_ARTIFACT_PATH` | path || Path or `http(s)://` URL to a compiled `objectstack.json` artifact to boot the kernel from. |
5455

content/docs/deployment/production-readiness.mdx

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,11 @@ the [HARDENING.md recipes](https://github.com/objectstack-ai/objectstack/blob/ma
8787
deployment requesting multi-org without `@objectstack/organizations` now
8888
refuses to boot unless `OS_ALLOW_DEGRADED_TENANCY=1`; never set that flag
8989
in production. See [Tenancy Modes & Membership](/docs/deployment/tenancy-modes).
90+
- [ ] `OS_ALLOW_DRIVER_CONNECT_FAILURE` is **unset**. A data driver that cannot
91+
connect at startup refuses the boot, so a bad `OS_DATABASE_URL`, a rotated
92+
password, or a closed network path surfaces as a failed deploy instead of
93+
a "started" server that 500s every request. Setting the flag boots without
94+
the database and never reconnects; never set it in production.
9095
- [ ] Backup / restore drill documented and tested.
9196
- [ ] Data-retention windows reviewed (ADR-0057): the platform's default
9297
`lifecycle` declarations bound telemetry (activity 14d, job runs 30d,

packages/objectql/src/core.ts

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,12 @@ export type { CompanionFieldMeta, CompanionObjectMeta } from './search-companion
3939
export { ObjectQL, ObjectRepository, ScopedContext } from './engine.js';
4040
export type { ObjectQLHostContext, HookHandler, HookEntry, OperationContext, EngineMiddleware } from './engine.js';
4141

42+
// Boot guard: thrown by `ObjectQL.init()` when a registered driver's connect()
43+
// fails (framework#3741). Embedders that boot the engine themselves can catch
44+
// it to render their own "database unreachable" message.
45+
export { DriverConnectError } from './driver-connect-errors.js';
46+
export type { DriverConnectFailure } from './driver-connect-errors.js';
47+
4248
// In-memory aggregation fallback
4349
export { applyInMemoryAggregation, bucketDateValue } from './in-memory-aggregation.js';
4450

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license.
2+
3+
/** One boot-registered driver whose `connect()` rejected during engine init. */
4+
export interface DriverConnectFailure {
5+
driverName: string;
6+
error: unknown;
7+
}
8+
9+
/** `error.message` when it is an Error, its string form otherwise. */
10+
function failureMessage(error: unknown): string {
11+
if (error instanceof Error) return error.message || error.name;
12+
return String(error);
13+
}
14+
15+
/**
16+
* Thrown by `ObjectQL.init()` when one or more boot-registered drivers fail to
17+
* connect (framework#3741). Aborts kernel bootstrap: a server that cannot reach
18+
* its database must not report itself started.
19+
*
20+
* Two failures are collapsed into one class on purpose, because `init()` cannot
21+
* tell them apart and the correct response to both is the same — don't boot:
22+
*
23+
* - the datasource is genuinely unreachable (wrong `OS_DATABASE_URL`, rotated
24+
* password, closed network path), and
25+
* - the driver is DELIBERATELY REFUSING to start (licence, server version,
26+
* incompatible configuration, missing capability). Throwing from `connect()`
27+
* is the supported way for a driver to veto boot; before this error existed,
28+
* such a veto was caught and downgraded to a query-time error, which is why
29+
* driver-mongodb's tenancy guard had to be hoisted into its constructor
30+
* (#3724 / #3734).
31+
*
32+
* The message is self-contained — it names every failed driver and its cause —
33+
* because the CLI prints `error.message` alone (stack only under `DEBUG`).
34+
* Identified by `code` rather than `instanceof` so it survives crossing package
35+
* boundaries.
36+
*/
37+
export class DriverConnectError extends Error {
38+
readonly code = 'ERR_DRIVER_CONNECT' as const;
39+
40+
/**
41+
* The first failure's `Error`, so its stack stays reachable for `DEBUG`
42+
* output and structured logging. Declared here rather than inherited: the
43+
* repo compiles against `lib: ES2020`, which predates `Error.cause`.
44+
*/
45+
readonly cause?: unknown;
46+
47+
constructor(
48+
public readonly failures: DriverConnectFailure[],
49+
public readonly totalDrivers: number,
50+
) {
51+
const detail = failures
52+
.map((f) => ` • ${f.driverName}: ${failureMessage(f.error)}`)
53+
.join('\n');
54+
super(
55+
`${failures.length} of ${totalDrivers} data driver(s) failed to connect — refusing to boot.\n` +
56+
`${detail}\n` +
57+
`A driver that did not connect cannot serve queries (there is no lazy reconnection) and the ` +
58+
`schema sync that runs right after init would issue DDL against it. Fix the datasource ` +
59+
`configuration (e.g. OS_DATABASE_URL), or set OS_ALLOW_DRIVER_CONNECT_FAILURE=1 to boot anyway ` +
60+
`in an explicitly degraded state where every query to those drivers fails.`,
61+
);
62+
this.name = 'DriverConnectError';
63+
if (failures[0]?.error instanceof Error) {
64+
(this as { cause?: unknown }).cause = failures[0].error;
65+
}
66+
}
67+
68+
/** Names of the drivers that failed, in registration order. */
69+
get failedDrivers(): string[] {
70+
return this.failures.map((f) => f.driverName);
71+
}
72+
}
73+
74+
/**
75+
* Emit the degraded-boot banner on a channel the host cannot accidentally
76+
* silence.
77+
*
78+
* `OS_ALLOW_DRIVER_CONNECT_FAILURE` only justifies itself if the state it opts
79+
* into is impossible to miss — and a logger-only banner is missable: `os serve`
80+
* swallows ALL of stdout while the kernel boots (its "boot-quiet" capture), and
81+
* `Logger` routes `warn` to stdout, so the one message that matters would be
82+
* invisible in exactly the situation it exists for. Writing to stderr as well
83+
* is the same belt-and-braces the kernel already uses for plugin startup
84+
* failures.
85+
*
86+
* Best-effort and never throws: falls back to `console.error`, then to silence
87+
* on runtimes that have neither (the logger still carries the structured
88+
* record either way).
89+
*/
90+
export function emitDegradedBootBanner(message: string): void {
91+
const proc = (globalThis as {
92+
process?: { stderr?: { write?: (chunk: string) => unknown } };
93+
}).process;
94+
try {
95+
if (typeof proc?.stderr?.write === 'function') {
96+
proc.stderr.write(`${message}\n`);
97+
return;
98+
}
99+
} catch {
100+
/* stderr unavailable / closed — fall through to console */
101+
}
102+
try {
103+
(globalThis as { console?: { error?: (msg: string) => void } }).console?.error?.(message);
104+
} catch {
105+
/* no output channel at all — the logger record is the remaining trace */
106+
}
107+
}

0 commit comments

Comments
 (0)