diff --git a/.changeset/datasource-bound-connect-failfast.md b/.changeset/datasource-bound-connect-failfast.md new file mode 100644 index 0000000000..79a2ebd1cb --- /dev/null +++ b/.changeset/datasource-bound-connect-failfast.md @@ -0,0 +1,60 @@ +--- +"@objectstack/service-datasource": minor +"@objectstack/types": patch +"@objectstack/objectql": patch +--- + +fix(datasource)!: a declared datasource that objects bind to must connect, or the boot fails (#3758) + +`DatasourceConnectionService.handleFailure()` fail-fasted only for an `external` +datasource with `validation.onMismatch: 'fail'`. Everything else degraded to one +`warn` line — including the case the D2 auto-connect gate itself flags as having +**no fallback path**: a datasource that objects bind to explicitly via +`object.datasource`. Those objects never fall through to the `default` driver; +`engine.getDriver` throws `Datasource 'x' is not registered` for them. + +So an app declaring `datasource: 'analytics'` with 20 objects bound to it, booted +against a wrong `ANALYTICS_URL`, started clean and exited zero — and then failed +every read and write of those 20 objects with an error that reads nothing like +*the analytics database is unreachable*. The rest of the app worked, which made it +**harder** to locate than a total outage: it looks like "some pages are broken", +not like a misconfigured datasource. This is the same decision #3741/#3751 fixed +one layer up in `ObjectQLEngine.init()`; the boundary here was still drawn in the +old place. + +- **Fail-fast is now keyed on "no fallback path", not on `onMismatch` alone.** At + the `declared-auto` (boot) trigger, a connect failure aborts the boot when the + datasource is `external` + `onMismatch: 'fail'` **or** when ≥1 object binds to + it explicitly. `autoConnect: true` with nothing bound stays lenient — that is + "connect it if you can", and nothing declares a dependency on it. The + runtime-admin create/update and boot-rehydration triggers are unchanged and + still always degrade: a UI action must never brick a running server. +- **Every failure mode counts**, not just an unreachable socket: an unresolvable + `external.credentialsRef` (D3) and an unsupported `driver` leave the bound + objects exactly as dead, so they take the same verdict. +- **The error names the bound objects** (up to 10, then `+N more`) alongside the + underlying cause, so the message points at the real problem instead of just the + datasource name. The service already receives the list for post-connect + `syncObjectSchema`. +- **`connectDeclared()` attempts every gated datasource before throwing**, and + aggregates, so one failed boot reports all the misconfigured ones rather than + one per restart — the same shape as `ObjectQLEngine.init()`'s + `DriverConnectError`. +- **The escape hatch is shared with the engine guard**: + `OS_ALLOW_DRIVER_CONNECT_FAILURE=1` now also covers this path (and covers + `onMismatch: 'fail'`, which previously had no opt-out). The operator intent is + identical — "I know the database is unreachable, boot anyway" — and two flags + would only guarantee one of them gets missed. When set, boot continues and a + `DEGRADED BOOT` banner goes to stderr as well as the logger, because `os serve` + swallows stdout during boot. `emitDegradedBootBanner` moved to + `@objectstack/types` so both call sites share one implementation; + `@objectstack/objectql` re-exports it unchanged. + +ADR-0062 D5 is amended with the new criterion and the shared flag. + +**Migration.** No change for a correctly configured deployment — a datasource that +connected before still connects. A deployment that was *silently* booting with a +dead, explicitly-bound datasource now fails the boot instead, naming the +datasource, the cause, and the objects that depend on it; fix the datasource +configuration. To keep booting without it — deliberately, knowing every request +touching those objects will fail — set `OS_ALLOW_DRIVER_CONNECT_FAILURE=1`. diff --git a/content/docs/data-modeling/drivers.mdx b/content/docs/data-modeling/drivers.mdx index 08261a07ac..d84180c9e5 100644 --- a/content/docs/data-modeling/drivers.mdx +++ b/content/docs/data-modeling/drivers.mdx @@ -91,6 +91,10 @@ comes back is the **boot sequence that was skipped**: nothing re-runs the schema sync, so those objects can be left with no tables even after the database returns. +The same guard covers **declared datasources** whose objects have no fallback — +see [When auto-connect fails](/docs/data-modeling/external-datasources#when-auto-connect-fails) +([#3758](https://github.com/objectstack-ai/objectstack/issues/3758)). + `OS_ALLOW_DRIVER_CONNECT_FAILURE=1` boots anyway, in an explicitly degraded state announced by a `DEGRADED BOOT` banner. Queries to a failed driver fail diff --git a/content/docs/data-modeling/external-datasources.mdx b/content/docs/data-modeling/external-datasources.mdx index 3f13c8e8f6..e05f628e39 100644 --- a/content/docs/data-modeling/external-datasources.mdx +++ b/content/docs/data-modeling/external-datasources.mdx @@ -124,6 +124,36 @@ A connected external datasource is also visible in **Setup → Datasources** (st `origin: code`, read-only in the UI) and via `GET /api/v1/datasources` and `GET /api/v1/meta/datasource`, where an admin can run the "Sync objects" wizard. +### When auto-connect fails + +An object that binds explicitly via `datasource: '…'` has **no fallback** — it +never falls through to the `default` driver, so an unconnected datasource means +every read and write of that object fails. The boot therefore **refuses to start** +when a datasource in that position cannot be connected +([#3758](https://github.com/objectstack-ai/objectstack/issues/3758)), rather than +leaving a server that looks healthy and errors only on the affected pages: + +| Gate | Connect fails at boot | +|:---|:---| +| **external** with `validation.onMismatch: 'fail'` | refuses the boot | +| **objects bind explicitly** via `object.datasource` | refuses the boot — the error names them | +| **`autoConnect: true`** with nothing bound | warning; left unconnected | + +This covers every reason a connect can fail — unreachable database, unresolvable +`credentialsRef`, unsupported `driver` — because the bound objects are equally +dead in each case. Every gated datasource is attempted before the boot aborts, so +one failed start reports *all* the misconfigured ones. + + +`OS_ALLOW_DRIVER_CONNECT_FAILURE=1` boots anyway — the same flag as the +[engine-level driver-connect guard](/docs/data-modeling/drivers#startup-a-driver-that-cannot-connect-aborts-the-boot) — +in an explicitly degraded state announced by a `DEGRADED BOOT` banner. The +datasource stays unconnected for the process lifetime: nothing re-runs the +connect, so queries against its objects keep failing with +`Datasource 'x' is not registered` even after the database comes back. Do not set +it in production. + + ## 4. Credentials Never inline a password. Put a reference in `external.credentialsRef` and store the @@ -133,10 +163,11 @@ cleartext **at connect, before the driver is built**. Resolution is **fail-closed**: if a `credentialsRef` is declared but no secret store is configured, or the secret cannot be resolved/decrypted, the datasource is left -**unconnected with a clear error** — never connected without the credential. For a -code-defined external datasource with `validation.onMismatch: 'fail'` this fails -the boot; otherwise it degrades with a warning. (Credential-less drivers such as -SQLite simply have no `credentialsRef`.) +**unconnected with a clear error** — never connected without the credential. An +unresolvable credential is a connect failure like any other, so it fails the boot +for the datasources listed under [When auto-connect fails](#when-auto-connect-fails) +and degrades with a warning otherwise. (Credential-less drivers such as SQLite +simply have no `credentialsRef`.) ## 5. Writes (double opt-in) diff --git a/content/docs/deployment/environment-variables.mdx b/content/docs/deployment/environment-variables.mdx index 83da5b19eb..726e89e521 100644 --- a/content/docs/deployment/environment-variables.mdx +++ b/content/docs/deployment/environment-variables.mdx @@ -49,7 +49,7 @@ read at startup unless noted otherwise. Boolean variables accept `true` / `false |:---|:---|:---|:---| | `OS_DATABASE_URL` | url | — | Database connection string (e.g. `file:./data.sqlite`, `postgres://…`, `mongodb://…`, `memory://`). `libsql://` (Turso) is not supported. | | `OS_DATABASE_DRIVER` | enum | inferred | Force a specific driver when the URL is ambiguous. `memory` \| `sqlite` \| `sqlite-wasm` \| `postgres` \| `mongodb`. | -| `OS_ALLOW_DRIVER_CONNECT_FAILURE` | boolean | `false` | Escape hatch for the driver-connect boot guard. By default a data driver that fails to connect at startup **refuses the boot** — a server that cannot reach its database must not report itself started and then fail every request. Set to `1` to boot anyway, in an explicitly degraded state logged loudly at startup. There is **no reconnection**: the drivers that failed stay dead for the process lifetime and every query and schema sync routed to them fails. | +| `OS_ALLOW_DRIVER_CONNECT_FAILURE` | boolean | `false` | Escape hatch for the driver-connect boot guard. By default a data driver that fails to connect at startup **refuses the boot** — a server that cannot reach its database must not report itself started and then fail every request. The same guard covers a **declared datasource** that objects bind to via `datasource: '…'`, or an `external` one with `validation.onMismatch: 'fail'`: those objects have no fallback datasource, so an unconnected one means they are all dead. Set to `1` to boot anyway, in an explicitly degraded state logged loudly at startup. There is **no reconnection**: whatever failed stays dead for the process lifetime and every query and schema sync routed to it fails. | | `OS_STORAGE_ROOT` | path | `./.objectstack/data/uploads` | Root directory for the local file storage adapter, relative to the process cwd (used by `os serve`'s default `storage` capability wiring). | | `OS_ARTIFACT_PATH` | path | — | Path or `http(s)://` URL to a compiled `objectstack.json` artifact to boot the kernel from. | diff --git a/content/docs/deployment/production-readiness.mdx b/content/docs/deployment/production-readiness.mdx index c27d9fdf39..3aeb36d9bd 100644 --- a/content/docs/deployment/production-readiness.mdx +++ b/content/docs/deployment/production-readiness.mdx @@ -90,8 +90,12 @@ the [HARDENING.md recipes](https://github.com/objectstack-ai/objectstack/blob/ma - [ ] `OS_ALLOW_DRIVER_CONNECT_FAILURE` is **unset**. A data driver that cannot connect at startup refuses the boot, so a bad `OS_DATABASE_URL`, a rotated password, or a closed network path surfaces as a failed deploy instead of - a "started" server that 500s every request. Setting the flag boots without - the database and never reconnects; never set it in production. + a "started" server that 500s every request. The same applies to a declared + **datasource** that objects bind to explicitly (`datasource: 'analytics'`): + those objects have no fallback, so a misconfigured `analytics` URL fails + the deploy instead of leaving a server where most pages work and that one + subset errors. Setting the flag boots without the database and never + reconnects; never set it in production. - [ ] Backup / restore drill documented and tested. - [ ] Data-retention windows reviewed (ADR-0057): the platform's default `lifecycle` declarations bound telemetry (activity 14d, job runs 30d, diff --git a/docs/adr/0062-external-datasource-runtime.md b/docs/adr/0062-external-datasource-runtime.md index 33b73191fe..282073a7b4 100644 --- a/docs/adr/0062-external-datasource-runtime.md +++ b/docs/adr/0062-external-datasource-runtime.md @@ -69,10 +69,26 @@ Code-defined datasources surface in `GET /api/v1/datasources`, `GET /api/v1/meta ### D5 — Lifecycle, health & ordering for N datasources -`DatasourceConnectionService` owns connect/disconnect (graceful shutdown), pool config per datasource, and an optional health probe surfaced in the admin list (`status`). **Ordering**: all declared datasources connect **before** the `kernel:ready` external-validation gate (ADR-0015 §5.2) and before first query — i.e. during plugin init/start, not in a `kernel:ready` handler. Connect failure policy is **fail-fast for `external` with `validation.onMismatch: 'fail'`**, **degrade-with-warning** otherwise (a connectivity blip on an optional analytics replica should not brick boot). +`DatasourceConnectionService` owns connect/disconnect (graceful shutdown), pool config per datasource, and an optional health probe surfaced in the admin list (`status`). **Ordering**: all declared datasources connect **before** the `kernel:ready` external-validation gate (ADR-0015 §5.2) and before first query — i.e. during plugin init/start, not in a `kernel:ready` handler. Connect failure policy is **fail-fast when the datasource has no fallback path**, **degrade-with-warning** otherwise (a connectivity blip on an optional analytics replica should not brick boot). > **Phase 1 implementation notes (#2163).** *Ordering* is satisfied by the kernel's two-phase boot (init-all → start-all): the connection service is registered as the `'datasource-connection'` kernel service during the datasource-admin plugin's `init()`, and declared datasources are auto-connected from `AppPlugin.start()` — which runs before the `kernel:ready` validation gate. Because boot schema-sync runs before the external driver exists, `connect()` calls `engine.syncObjectSchema()` for each bound federated object (DDL-free), so they are queryable with zero app code. *Fail-fast* is scoped to the **declared-auto** trigger; the runtime-admin create/update + boot-rehydration triggers always degrade-with-warning, preserving the pre-ADR-0062 admin behavior (a UI action never bricks the running server). *Connect policy* (the epic #2163 seam): a host-injectable `DatasourceConnectPolicy` is consulted before every connect; the open-core default allows (subject to the D2 gate), and a multi-tenant host binds a stricter, fail-closed policy for egress isolation — one connect path, no cloud fork. +> **Amendment (#3758) — "no fallback path" is the fail-fast criterion, not `onMismatch:'fail'` alone.** The original wording scoped fail-fast to `external` + `validation.onMismatch: 'fail'`, which drew the line in the wrong place: **an explicit `object.datasource` binding is a hard dependency too**, and it is one the D2 note above already identifies as having *no fallback whatsoever* — such objects never resolve to the `default` driver, `engine.getDriver` throws `Datasource 'x' is not registered` for them. So a declared datasource that 20 objects bind to, failing to connect at boot, produced the worst possible shape: a server that starts clean, serves most of the app, and fails every read/write of those 20 objects with an error that reads nothing like "the analytics database is unreachable" — harder to locate than a total outage. The **declared-auto** fail-fast set is therefore: +> +> | | Gate | Boot connect failure | +> |:---|:---|:---| +> | (a) | `external` (`schemaMode !== 'managed'`) with `validation.onMismatch: 'fail'` | **fail-fast** | +> | (b) | ≥1 object binds explicitly via `object.datasource` | **fail-fast** — no fallback path | +> | (c) | `autoConnect: true`, nothing bound | degrade-with-warning — "connect it if you can"; nothing declares a dependency | +> +> This covers every reason a connect can fail, not just an unreachable socket: an unresolvable `external.credentialsRef` (D3) and an unsupported `driver` leave the bound objects exactly as dead, so they take the same verdict. `onMismatch` keeps its own meaning (what to do about a *schema* mismatch) and is no longer overloaded as the only way to say "this datasource is required". +> +> The escape hatch is **shared with the engine-level guard**, `OS_ALLOW_DRIVER_CONNECT_FAILURE=1` (framework#3741/#3751): the operator intent is identical ("I know the database is unreachable — boot anyway"), and two flags would only guarantee one of them gets missed. It now covers (a) as well, which previously had no opt-out. When set, the boot continues and the degraded state is announced through `emitDegradedBootBanner` (stderr as well as the logger, because `os serve`'s boot-quiet capture swallows stdout). +> +> The error message names the **bound objects** (up to 10, then `+N more`), not just the datasource — the service already receives them for post-connect `syncObjectSchema`. And `connectDeclared` attempts **every** gated datasource before throwing an aggregate, so one boot reports every misconfiguration rather than one per restart — the same shape as `ObjectQLEngine.init()`'s `DriverConnectError`. +> +> **A `DatasourceConnectPolicy` denial is not a connect failure** and stays metadata-only, unchanged. A multi-tenant host that blocks egress for a tenant's plan is making a deliberate decision about a datasource it knows it is refusing; fail-fasting there would turn a policy verdict into a boot outage for every tenant on the shared runtime. The fail-fast set covers *failures* — unreachable, unauthenticated, unsupported — not *refusals*. + ### D6 — Native-analytics SQL honors the remote table/columns The analytics native-SQL strategy compiles its own `FROM ""` / column references outside the driver (ADR-0015 §18 noted this). It must resolve an external object's physical table (`remoteName`/`remoteSchema`) and columns (`columnMap`) the same way `SqlDriver` now does — reusing the driver's resolution (e.g. an exposed `physicalTableFor(object)` / `physicalColumnFor(object, field)`), not a second copy. Until then, analytics over external objects stays disabled rather than silently querying the wrong table. diff --git a/examples/app-showcase/src/system/datasources/showcase-external.datasource.ts b/examples/app-showcase/src/system/datasources/showcase-external.datasource.ts index 9e40a22bd9..a2c3d95ab8 100644 --- a/examples/app-showcase/src/system/datasources/showcase-external.datasource.ts +++ b/examples/app-showcase/src/system/datasources/showcase-external.datasource.ts @@ -11,8 +11,15 @@ import { defineDatasource } from '@objectstack/spec/data'; * queryable through the normal ObjectQL/REST surface. * * `schemaMode: 'external'` ⇒ ObjectStack never runs DDL here (it's a guest in a - * database it does not own). `onMismatch: 'warn'` keeps a fixture hiccup from - * bricking the whole showcase boot — the rest of the demo still loads. + * database it does not own). `onMismatch: 'warn'` keeps a *schema drift* in the + * fixture (a renamed column, an extra table) from bricking the showcase boot — + * the rest of the demo still loads. + * + * It does **not** make a failed *connection* survivable, and shouldn't: the + * `customer` / `order` objects bind to this datasource explicitly, so they have + * no fallback driver and every query against them would fail (#3758). If the + * fixture file cannot be opened at all, the boot stops with that as the reason + * rather than serving a showcase whose federation pages are quietly dead. * * It also shows up in **Setup → Integrations → Datasources** and via * `GET /api/v1/meta/datasource`, where an admin can run the runtime diff --git a/packages/objectql/src/driver-connect-errors.ts b/packages/objectql/src/driver-connect-errors.ts index 10a5297cf9..85407a1362 100644 --- a/packages/objectql/src/driver-connect-errors.ts +++ b/packages/objectql/src/driver-connect-errors.ts @@ -85,36 +85,10 @@ export class DriverConnectError extends Error { } /** - * Emit the degraded-boot banner on a channel the host cannot accidentally - * silence. - * - * `OS_ALLOW_DRIVER_CONNECT_FAILURE` only justifies itself if the state it opts - * into is impossible to miss — and a logger-only banner is missable: `os serve` - * swallows ALL of stdout while the kernel boots (its "boot-quiet" capture), and - * `Logger` routes `warn` to stdout, so the one message that matters would be - * invisible in exactly the situation it exists for. Writing to stderr as well - * is the same belt-and-braces the kernel already uses for plugin startup - * failures. - * - * Best-effort and never throws: falls back to `console.error`, then to silence - * on runtimes that have neither (the logger still carries the structured - * record either way). + * The degraded-boot banner now lives in `@objectstack/types` because + * `DatasourceConnectionService` owes the operator the same banner for the same + * flag (framework#3758) and must not depend on the whole query engine to print + * it. Re-exported here so this module stays the single import site for + * engine-side driver-connect failure reporting. */ -export function emitDegradedBootBanner(message: string): void { - const proc = (globalThis as { - process?: { stderr?: { write?: (chunk: string) => unknown } }; - }).process; - try { - if (typeof proc?.stderr?.write === 'function') { - proc.stderr.write(`${message}\n`); - return; - } - } catch { - /* stderr unavailable / closed — fall through to console */ - } - try { - (globalThis as { console?: { error?: (msg: string) => void } }).console?.error?.(message); - } catch { - /* no output channel at all — the logger record is the remaining trace */ - } -} +export { emitDegradedBootBanner } from '@objectstack/types'; diff --git a/packages/runtime/src/app-plugin.ts b/packages/runtime/src/app-plugin.ts index ee2d52db05..5bf49a7bcd 100644 --- a/packages/runtime/src/app-plugin.ts +++ b/packages/runtime/src/app-plugin.ts @@ -455,11 +455,14 @@ export class AppPlugin implements Plugin { } } } catch (err) { - // A fail-fast (external + onMismatch:'fail') connect error propagates - // to brick boot as intended (ADR-0062 D5); other errors are already - // degraded inside the connection service. Re-throw so the kernel - // surfaces the real cause. (Single-string message: the context - // logger types `error(message, error?)`, not a meta object.) + // A fail-fast connect error propagates to brick boot as intended + // (ADR-0062 D5): the datasource has no fallback path — `external` + + // onMismatch:'fail', or objects that bind to it explicitly and would + // otherwise fail every query with "Datasource 'x' is not registered" + // (#3758). Other errors are already degraded inside the connection + // service. Re-throw so the kernel surfaces the real cause. + // (Single-string message: the context logger types + // `error(message, error?)`, not a meta object.) ctx.logger.error( `[AppPlugin] declared-datasource auto-connect failed for app '${appId}': ${(err as Error)?.message ?? String(err)}`, ); diff --git a/packages/runtime/src/datasource-autoconnect.test.ts b/packages/runtime/src/datasource-autoconnect.test.ts index 30c177fb22..bd857dd255 100644 --- a/packages/runtime/src/datasource-autoconnect.test.ts +++ b/packages/runtime/src/datasource-autoconnect.test.ts @@ -11,7 +11,7 @@ // driver, so the full AppPlugin → `datasource-connection` → engine path runs // without any native driver dependency. -import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import { describe, it, expect, beforeAll, afterAll, beforeEach, afterEach } from 'vitest'; import { Runtime } from './runtime.js'; import { DriverPlugin } from './driver-plugin.js'; import { AppPlugin } from './app-plugin.js'; @@ -180,6 +180,82 @@ describe('ADR-0062 credentials fail-closed (D3)', () => { }, BOOT_TIMEOUT); }); +describe('ADR-0062 D5 — an explicitly-bound datasource that cannot connect bricks boot (#3758)', () => { + // A MANAGED datasource with no `onMismatch:'fail'` to lean on, that two + // objects bind to explicitly. Those objects never fall back to the `default` + // driver — `engine.getDriver` throws for them — so leaving this at a warning + // produced a server that started clean and failed every query against them. + function boundArtifact() { + return { + manifest: { id: 'com.test.ds-bound', name: 'DS Bound', version: '1.0.0' }, + objects: [ + { name: 'visit', label: 'Visit', datasource: 'analytics', fields: { id: { type: 'text' } } }, + { name: 'session', label: 'Session', datasource: 'analytics', fields: { id: { type: 'text' } } }, + { name: 'note', label: 'Note', fields: { title: { type: 'text' } } }, + ], + datasources: [ + { + name: 'analytics', + label: 'Analytics', + // No factory supports this driver — the same dead end as an + // unreachable host, from the bound objects' point of view. + driver: 'not-a-real-driver', + schemaMode: 'managed', + origin: 'code', + config: {}, + active: true, + }, + ], + }; + } + + async function bootBound() { + const { ObjectQLPlugin } = await import('@objectstack/objectql'); + const { InMemoryDriver } = await import('@objectstack/driver-memory'); + const { DatasourceAdminServicePlugin, createDefaultDatasourceDriverFactory } = await import( + '@objectstack/service-datasource' + ); + const runtime = new Runtime({ cluster: false }); + const kernel = runtime.getKernel(); + await kernel.use(new DriverPlugin(new InMemoryDriver())); + await kernel.use(new ObjectQLPlugin()); + await kernel.use(new AppPlugin(boundArtifact())); + await kernel.use( + new DatasourceAdminServicePlugin({ driverFactory: createDefaultDatasourceDriverFactory() }), + ); + return kernel; + } + + const ENV = 'OS_ALLOW_DRIVER_CONNECT_FAILURE'; + let saved: string | undefined; + beforeEach(() => { saved = process.env[ENV]; delete process.env[ENV]; }); + afterEach(() => { + if (saved === undefined) delete process.env[ENV]; + else process.env[ENV] = saved; + }); + + it('refuses the boot, naming the datasource and the objects that depend on it', async () => { + const kernel = await bootBound(); + const err = await kernel.bootstrap().then( + () => { throw new Error('bootstrap() resolved but should have thrown'); }, + (e: unknown) => e as Error, + ); + expect(err.message).toMatch(/analytics/); + expect(err.message).toMatch(/visit/); + expect(err.message).toMatch(/session/); + try { await (kernel as any)?.stop?.(); } catch { /* noop */ } + }, BOOT_TIMEOUT); + + it('boots degraded when the operator sets OS_ALLOW_DRIVER_CONNECT_FAILURE', async () => { + process.env[ENV] = '1'; + const kernel = await bootBound(); + await expect(kernel.bootstrap()).resolves.not.toThrow(); + const engine = kernel.getService<{ getDriverByName(n: string): unknown }>('data'); + expect(engine.getDriverByName('analytics')).toBeUndefined(); // still unconnected + try { await (kernel as any)?.stop?.(); } catch { /* noop */ } + }, BOOT_TIMEOUT); +}); + describe('ADR-0062 connect policy seam', () => { it('a deny policy leaves the external datasource unconnected (cloud egress isolation)', async () => { const denyExternal: DatasourceConnectPolicy = { diff --git a/packages/services/service-datasource/package.json b/packages/services/service-datasource/package.json index 6b5e2e2492..f5b5aa4ed1 100644 --- a/packages/services/service-datasource/package.json +++ b/packages/services/service-datasource/package.json @@ -27,7 +27,8 @@ }, "dependencies": { "@objectstack/core": "workspace:*", - "@objectstack/spec": "workspace:*" + "@objectstack/spec": "workspace:*", + "@objectstack/types": "workspace:*" }, "devDependencies": { "@objectstack/driver-memory": "workspace:*", diff --git a/packages/services/service-datasource/src/__tests__/datasource-connection-service.test.ts b/packages/services/service-datasource/src/__tests__/datasource-connection-service.test.ts index 5ae82da386..ffa1b02838 100644 --- a/packages/services/service-datasource/src/__tests__/datasource-connection-service.test.ts +++ b/packages/services/service-datasource/src/__tests__/datasource-connection-service.test.ts @@ -1,6 +1,6 @@ // Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. -import { describe, it, expect, vi } from 'vitest'; +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; import { DatasourceConnectionService, isDatasourceAddressed, @@ -62,13 +62,15 @@ function svc(over: { } = {}) { const engine = over.engine === undefined ? fakeEngine() : over.engine; const factory = over.factory === undefined ? fakeFactory() : over.factory; + const warnings: string[] = []; const service = new DatasourceConnectionService({ factory: () => factory ?? undefined, engine: () => engine ?? undefined, policy: over.policy, secrets: over.secrets, + logger: { warn: (msg: string) => { warnings.push(msg); } }, }); - return { service, engine: engine as ReturnType | undefined, factory }; + return { service, engine: engine as ReturnType | undefined, factory, warnings }; } const externalDs: ConnectableDatasource = { @@ -189,12 +191,151 @@ describe('DatasourceConnectionService.connect', () => { expect(result.status).toBe('failed-degraded'); }); - it('degrade: external + onMismatch:warn degrades even at boot', async () => { + it('degrade: external + onMismatch:warn degrades even at boot (nothing binds to it)', async () => { const { service } = svc({ factory: fakeFactory({ connectThrows: true }) }); const result = await service.connect(externalDs, { context: { trigger: 'declared-auto' } }); expect(result.status).toBe('failed-degraded'); }); }); + + // framework#3758: a datasource that objects bind to explicitly has NO fallback + // — `engine.getDriver` throws for them rather than resolving `default` — so a + // boot-time connect failure used to leave a clean-looking server whose bound + // objects all 500 at query time with "Datasource 'x' is not registered". + describe('D5 fail-fast for explicitly-bound datasources (framework#3758)', () => { + const ENV = 'OS_ALLOW_DRIVER_CONNECT_FAILURE'; + let saved: string | undefined; + beforeEach(() => { saved = process.env[ENV]; delete process.env[ENV]; }); + afterEach(() => { + if (saved === undefined) delete process.env[ENV]; + else process.env[ENV] = saved; + }); + + /** A plain managed datasource — no `onMismatch:'fail'` to lean on. */ + const analytics: ConnectableDatasource = { + name: 'analytics', + driver: 'sqlite', + schemaMode: 'managed', + config: {}, + }; + + it('fail-fast: bound objects at boot brick the boot, even for a managed datasource', async () => { + const { service } = svc({ factory: fakeFactory({ connectThrows: true }) }); + await expect( + service.connect(analytics, { objects: ['visit', 'session'], context: { trigger: 'declared-auto' } }), + ).rejects.toThrow(/fail-fast/); + }); + + it('names the bound objects and the real cause — not just the datasource', async () => { + const { service } = svc({ factory: fakeFactory({ connectThrows: true }) }); + const err = await service + .connect(analytics, { objects: ['visit', 'session'], context: { trigger: 'declared-auto' } }) + .then( + () => { throw new Error('connect() resolved but should have thrown'); }, + (e: unknown) => e as Error, + ); + expect(err.message).toContain("datasource 'analytics'"); + expect(err.message).toContain('connection refused'); // the underlying cause + expect(err.message).toContain('2 object(s) bind to it explicitly'); + expect(err.message).toContain('visit, session'); + expect(err.message).toContain('OS_ALLOW_DRIVER_CONNECT_FAILURE=1'); + }); + + it('truncates a long binding list to 10 names + a count', async () => { + const { service } = svc({ factory: fakeFactory({ connectThrows: true }) }); + const objects = Array.from({ length: 20 }, (_, i) => `obj_${i}`); + const err = await service + .connect(analytics, { objects, context: { trigger: 'declared-auto' } }) + .then(() => undefined, (e: Error) => e); + expect(err!.message).toContain('20 object(s) bind to it explicitly'); + expect(err!.message).toContain('obj_0, obj_1'); + expect(err!.message).toContain('+10 more'); + expect(err!.message).not.toContain('obj_15'); + }); + + it('a credential failure on a bound datasource is fatal too (D3 ∩ D5)', async () => { + const { service } = svc({ secrets: { resolve: async () => undefined } }); + await expect( + service.connect( + { ...analytics, external: { credentialsRef: 'sys_secret:abc' } }, + { objects: ['visit'], context: { trigger: 'declared-auto' } }, + ), + ).rejects.toThrow(/fail-fast/); + }); + + it('an unsupported driver on a bound datasource is fatal too — the objects are just as dead', async () => { + const { service } = svc({ factory: fakeFactory({ supports: () => false }) }); + await expect( + service.connect(analytics, { objects: ['visit'], context: { trigger: 'declared-auto' } }), + ).rejects.toThrow(/no driver factory supports/); + }); + + it('degrade: autoConnect:true with nothing bound stays lenient ("connect it if you can")', async () => { + const { service } = svc({ factory: fakeFactory({ connectThrows: true }) }); + const result = await service.connect( + { ...analytics, autoConnect: true }, + { context: { trigger: 'declared-auto' } }, + ); + expect(result.status).toBe('failed-degraded'); + }); + + it('degrade: the SAME bound datasource via runtime-admin never bricks the running server', async () => { + const { service } = svc({ factory: fakeFactory({ connectThrows: true }) }); + const result = await service.connect(analytics, { + objects: ['visit', 'session'], + context: { trigger: 'runtime-admin' }, + }); + expect(result.status).toBe('failed-degraded'); + }); + + it('boots degraded when OS_ALLOW_DRIVER_CONNECT_FAILURE opts in, and says so loudly', async () => { + process.env[ENV] = '1'; + const { service, warnings } = svc({ factory: fakeFactory({ connectThrows: true }) }); + const result = await service.connect(analytics, { + objects: ['visit'], + context: { trigger: 'declared-auto' }, + }); + expect(result.status).toBe('failed-degraded'); + const warned = warnings.join('\n'); + expect(warned).toContain('DEGRADED BOOT'); + expect(warned).toContain('visit'); + }); + + it('repeats the degraded banner on stderr, which `os serve` boot-quiet cannot swallow', async () => { + process.env[ENV] = '1'; + const written: string[] = []; + const realWrite = process.stderr.write; + (process.stderr as { write: unknown }).write = (chunk: any) => { + written.push(String(chunk)); + return true; + }; + try { + const { service } = svc({ factory: fakeFactory({ connectThrows: true }) }); + await service.connect(analytics, { objects: ['visit'], context: { trigger: 'declared-auto' } }); + } finally { + (process.stderr as { write: unknown }).write = realWrite; + } + expect(written.join('')).toContain('DEGRADED BOOT'); + }); + + it('treats a falsy opt-in value as off — still fail-fast', async () => { + process.env[ENV] = 'false'; + const { service } = svc({ factory: fakeFactory({ connectThrows: true }) }); + await expect( + service.connect(analytics, { objects: ['visit'], context: { trigger: 'declared-auto' } }), + ).rejects.toThrow(/fail-fast/); + }); + + it('the shared flag also unblocks the (a) external + onMismatch:fail path', async () => { + process.env[ENV] = '1'; + const { service } = svc({ factory: fakeFactory({ connectThrows: true }) }); + const result = await service.connect( + { ...externalDs, external: { validation: { onMismatch: 'fail' } } }, + { context: { trigger: 'declared-auto' } }, + ); + expect(result.status).toBe('failed-degraded'); + }); + }); }); describe('D3 credential resolution — fail-closed', () => { @@ -291,4 +432,57 @@ describe('DatasourceConnectionService.connectDeclared', () => { expect(results).toEqual([]); expect(engine!.drivers.size).toBe(0); }); + + // framework#3758 — the operator should not have to reboot once per broken + // datasource to discover them, so every gated datasource is attempted before + // the aggregate throw (same shape as ObjectQL.init()'s DriverConnectError). + it('attempts every gated datasource, then reports ALL fatal failures in one error', async () => { + const ENV = 'OS_ALLOW_DRIVER_CONNECT_FAILURE'; + const saved = process.env[ENV]; + delete process.env[ENV]; + try { + const { service } = svc({ factory: fakeFactory({ connectThrows: true }) }); + const err = await service + .connectDeclared({ + datasources: [ + { name: 'analytics', driver: 'sqlite', schemaMode: 'managed', config: {} }, + { name: 'billing', driver: 'sqlite', schemaMode: 'managed', config: {} }, + ], + objects: [ + { name: 'visit', datasource: 'analytics' }, + { name: 'invoice', datasource: 'billing' }, + ], + }) + .then( + () => { throw new Error('connectDeclared() resolved but should have thrown'); }, + (e: unknown) => e as Error, + ); + expect(err.message).toContain('2 declared datasource(s) failed to connect'); + expect(err.message).toContain("datasource 'analytics'"); + expect(err.message).toContain("datasource 'billing'"); // not stopped at the first + expect(err.message).toContain('visit'); + expect(err.message).toContain('invoice'); + } finally { + if (saved === undefined) delete process.env[ENV]; + else process.env[ENV] = saved; + } + }); + + it('a single fatal failure propagates as-is (no aggregate wrapper to read past)', async () => { + const ENV = 'OS_ALLOW_DRIVER_CONNECT_FAILURE'; + const saved = process.env[ENV]; + delete process.env[ENV]; + try { + const { service } = svc({ factory: fakeFactory({ connectThrows: true }) }); + await expect( + service.connectDeclared({ + datasources: [{ name: 'analytics', driver: 'sqlite', schemaMode: 'managed', config: {} }], + objects: [{ name: 'visit', datasource: 'analytics' }], + }), + ).rejects.toThrow(/^datasource 'analytics': connect failed/); + } finally { + if (saved === undefined) delete process.env[ENV]; + else process.env[ENV] = saved; + } + }); }); diff --git a/packages/services/service-datasource/src/datasource-connection-service.ts b/packages/services/service-datasource/src/datasource-connection-service.ts index db0357149a..3c459e7dd7 100644 --- a/packages/services/service-datasource/src/datasource-connection-service.ts +++ b/packages/services/service-datasource/src/datasource-connection-service.ts @@ -24,6 +24,10 @@ * D8) and auto-connect never double-register. */ +import { + emitDegradedBootBanner, + resolveAllowDriverConnectFailure, +} from '@objectstack/types'; import type { IDatasourceDriverFactory, DatasourceConnectionSpec, @@ -135,6 +139,11 @@ export interface ConnectResult { * backward-compat guarantee). External datasources and explicit * `object.datasource` bindings never resolved to `default` (they throw when * unregistered), so auto-connecting them is a strict improvement, not a change. + * + * That same "no fallback" property is why gate (b) is also a **fail-fast** + * trigger when the connect fails (framework#3758) — see + * {@link DatasourceConnectionService.handleFailure}. Gate (c) is not: nothing + * declares a dependency on an `autoConnect` datasource. */ export function isDatasourceAddressed( ds: Pick, @@ -162,6 +171,12 @@ export class DatasourceConnectionService { * Called from `AppPlugin.start()` with the app bundle's datasources + objects. * Each connected external datasource also has its bound objects' read metadata * synced so they are immediately queryable with zero app code. + * + * Throws when any datasource hits the D5 fail-fast verdict (see + * {@link handleFailure}) — but only after attempting **all** of them, so one + * boot names every misconfigured datasource instead of one per restart. This + * mirrors `ObjectQLEngine.init()`'s aggregate `DriverConnectError` + * (framework#3741); the same operator is reading both. */ async connectDeclared(input: { datasources: readonly ConnectableDatasource[]; @@ -169,6 +184,7 @@ export class DatasourceConnectionService { }): Promise { const objects = input.objects ?? []; const results: ConnectResult[] = []; + const fatal: Error[] = []; for (const ds of input.datasources) { if (!ds?.name) continue; if (ds.active === false) continue; @@ -176,8 +192,20 @@ export class DatasourceConnectionService { const bound = objects .filter((o) => o?.datasource === ds.name && typeof o?.name === 'string') .map((o) => o.name as string); - results.push( - await this.connect(ds, { objects: bound, context: { origin: ds.origin ?? 'code', trigger: 'declared-auto' } }), + try { + results.push( + await this.connect(ds, { objects: bound, context: { origin: ds.origin ?? 'code', trigger: 'declared-auto' } }), + ); + } catch (err) { + fatal.push(err instanceof Error ? err : new Error(String(err))); + results.push({ name: ds.name, status: 'failed-degraded', reason: errMsg(err) }); + } + } + if (fatal.length === 1) throw fatal[0]; + if (fatal.length > 1) { + throw new Error( + `${fatal.length} declared datasource(s) failed to connect — refusing to boot.\n` + + fatal.map((e) => ` • ${e.message}`).join('\n'), ); } return results; @@ -187,10 +215,12 @@ export class DatasourceConnectionService { * Build + connect + register a single datasource's live driver. The shared * core used by both auto-connect and the runtime-admin pool registration. * - * Failure policy (ADR-0062 D5): an `external` datasource with - * `validation.onMismatch: 'fail'` fails fast (re-throws, bricking boot as - * intended); everything else degrades with a warning so an optional replica's - * connectivity blip never bricks boot. + * Failure policy (ADR-0062 D5): at boot, a datasource with **no fallback + * path** fails fast (re-throws, bricking boot as intended) — `external` with + * `validation.onMismatch: 'fail'`, or one that `opts.objects` shows is + * explicitly bound by objects. Everything else degrades with a warning so an + * optional replica's connectivity blip never bricks boot. See + * {@link handleFailure}. */ async connect( record: ConnectableDatasource, @@ -231,6 +261,7 @@ export class DatasourceConnectionService { 'skipped-unsupported', `no driver factory supports driver '${record.driver}'`, opts.context, + opts.objects, ); } @@ -252,12 +283,13 @@ export class DatasourceConnectionService { 'failed-credentials', `requires credential '${credentialsRef}' but no secret store (SecretBinder/ICryptoProvider) is configured`, opts.context, + opts.objects, ); } try { secret = await resolver(credentialsRef); } catch (err) { - return this.handleFailure(record, 'failed-credentials', `resolving credential '${credentialsRef}' threw: ${errMsg(err)}`, opts.context); + return this.handleFailure(record, 'failed-credentials', `resolving credential '${credentialsRef}' threw: ${errMsg(err)}`, opts.context, opts.objects); } if (secret == null || secret === '') { return this.handleFailure( @@ -265,6 +297,7 @@ export class DatasourceConnectionService { 'failed-credentials', `credential '${credentialsRef}' could not be resolved or decrypted (missing sys_secret row, or the encryption key changed)`, opts.context, + opts.objects, ); } } @@ -302,7 +335,7 @@ export class DatasourceConnectionService { this.logger?.info?.(`datasource '${name}': connected (driver=${record.driver}, schemaMode=${record.schemaMode ?? 'managed'})`); return { name, status: 'connected' }; } catch (err) { - return this.handleFailure(record, 'failed-degraded', errMsg(err), opts.context); + return this.handleFailure(record, 'failed-degraded', errMsg(err), opts.context, opts.objects); } } @@ -319,36 +352,89 @@ export class DatasourceConnectionService { } /** - * Apply the D5 connect-failure policy (also covers D3 credential failures). A - * code-defined `external` datasource with `onMismatch:'fail'` auto-connected at - * boot re-throws (fail-fast, bricking boot as intended). Runtime-admin - * create/update + boot rehydration always degrade-with-warning — a UI action - * or a replica blip must never brick the running server (preserves the - * pre-ADR-0062 admin behavior). Either way the datasource is left unconnected - * with a clear message — never a silent skip. + * Apply the D5 connect-failure policy (also covers D3 credential failures). + * + * A boot-time (`declared-auto`) connect failure is **fatal** when the + * datasource has no fallback path, which is true in two cases: + * + * - **(a)** it is `external` with `validation.onMismatch:'fail'` — the author + * asked for a hard stop explicitly; or + * - **(b)** objects bind to it explicitly via `object.datasource` — those + * objects have **no fallback whatsoever**: `engine.getDriver` throws + * `Datasource 'x' is not registered` for them, it never resolves `default` + * (framework#3758). Leaving this at a warning produced the worst possible + * shape: a server that boots clean, serves most of the app, and fails every + * read/write of the bound objects with an error that reads nothing like + * "the analytics database is unreachable". + * + * Anything else degrades with a warning: `autoConnect:true` means "connect it + * if you can" with nothing declaring a dependency on it, and runtime-admin + * create/update + boot rehydration must never brick a running server over a + * UI action or a replica blip (preserves the pre-ADR-0062 admin behavior). + * + * The fatal path shares the engine's escape hatch, + * `OS_ALLOW_DRIVER_CONNECT_FAILURE` (framework#3741) — the operator intent is + * identical ("I know the database is unreachable, boot anyway") and two flags + * would only mean one of them gets missed. When it is set the boot continues + * and the degraded state is announced on a channel `os serve`'s boot-quiet + * stdout capture cannot swallow. + * + * Either way the datasource is left unconnected with a clear message — never + * a silent skip. */ private handleFailure( record: ConnectableDatasource, status: ConnectStatus, reason: string, context?: DatasourceConnectContext, + boundObjects: readonly string[] = [], ): ConnectResult { const isExternal = record.schemaMode && record.schemaMode !== 'managed'; - const failFast = - context?.trigger === 'declared-auto' && - isExternal && - record.external?.validation?.onMismatch === 'fail'; const msg = `datasource '${record.name}': connect failed — ${reason}`; - if (failFast) { + + const causes: string[] = []; + if (context?.trigger === 'declared-auto') { + if (isExternal && record.external?.validation?.onMismatch === 'fail') { + causes.push(`schemaMode=${record.schemaMode}, validation.onMismatch='fail'`); + } + if (boundObjects.length > 0) { + causes.push( + `${boundObjects.length} object(s) bind to it explicitly (${formatObjectList(boundObjects)}) ` + + `and have no fallback datasource — every read/write of them would fail`, + ); + } + } + if (causes.length === 0) { + this.logger?.warn?.(`${msg} — degrading (datasource left unconnected)`); + return { name: record.name, status, reason }; + } + + const why = causes.join('; '); + if (!resolveAllowDriverConnectFailure()) { throw new Error( - `${msg}. (schemaMode=${record.schemaMode}, validation.onMismatch='fail' ⇒ fail-fast per ADR-0062 D5)`, + `${msg}. (${why} ⇒ fail-fast per ADR-0062 D5). Fix the datasource configuration, or set ` + + `OS_ALLOW_DRIVER_CONNECT_FAILURE=1 to boot anyway and serve errors until it is reachable.`, ); } - this.logger?.warn?.(`${msg} — degrading (datasource left unconnected)`); + const banner = + `⚠️ DEGRADED BOOT: ${msg} (${why}), but OS_ALLOW_DRIVER_CONNECT_FAILURE is set — starting ` + + `anyway. Queries against the objects bound to it fail with "Datasource '${record.name}' is ` + + `not registered" until it is reachable AND the server is restarted: nothing re-runs the ` + + `connect. Unset OS_ALLOW_DRIVER_CONNECT_FAILURE to restore fail-fast boot.`; + this.logger?.warn?.(banner); + // …and again on a channel the host cannot silence — see the helper's note + // on `os serve`'s boot-quiet stdout capture. + emitDegradedBootBanner(banner); return { name: record.name, status, reason }; } } +/** Up to 10 bound object names, then `+N more` — a name list, not a wall of text. */ +function formatObjectList(names: readonly string[]): string { + const head = names.slice(0, 10).join(', '); + return names.length > 10 ? `${head}, +${names.length - 10} more` : head; +} + function toSpec(record: ConnectableDatasource): DatasourceConnectionSpec { return { name: record.name, diff --git a/packages/types/src/degraded-boot.ts b/packages/types/src/degraded-boot.ts new file mode 100644 index 0000000000..c08f2b6ca6 --- /dev/null +++ b/packages/types/src/degraded-boot.ts @@ -0,0 +1,53 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * Degraded-boot reporting, shared by every subsystem that can be told to boot + * without a datasource it needs. + * + * Two of them exist today and they opt in through the *same* operator flag + * (`OS_ALLOW_DRIVER_CONNECT_FAILURE`, see {@link resolveAllowDriverConnectFailure}): + * + * - `ObjectQLEngine.init()` — a boot-registered driver whose `connect()` + * rejected (framework#3741). + * - `DatasourceConnectionService` — a declared datasource that objects bind to + * explicitly, or an `external` one with `validation.onMismatch:'fail'`, + * that could not be connected (framework#3758). + * + * They live in different packages but owe the operator the same thing: the + * degraded state must be impossible to miss. + */ + +/** + * Emit the degraded-boot banner on a channel the host cannot accidentally + * silence. + * + * `OS_ALLOW_DRIVER_CONNECT_FAILURE` only justifies itself if the state it opts + * into is impossible to miss — and a logger-only banner is missable: `os serve` + * swallows ALL of stdout while the kernel boots (its "boot-quiet" capture), and + * `Logger` routes `warn` to stdout, so the one message that matters would be + * invisible in exactly the situation it exists for. Writing to stderr as well + * is the same belt-and-braces the kernel already uses for plugin startup + * failures. + * + * Best-effort and never throws: falls back to `console.error`, then to silence + * on runtimes that have neither (the logger still carries the structured + * record either way). + */ +export function emitDegradedBootBanner(message: string): void { + const proc = (globalThis as { + process?: { stderr?: { write?: (chunk: string) => unknown } }; + }).process; + try { + if (typeof proc?.stderr?.write === 'function') { + proc.stderr.write(`${message}\n`); + return; + } + } catch { + /* stderr unavailable / closed — fall through to console */ + } + try { + (globalThis as { console?: { error?: (msg: string) => void } }).console?.error?.(message); + } catch { + /* no output channel at all — the logger record is the remaining trace */ + } +} diff --git a/packages/types/src/index.ts b/packages/types/src/index.ts index ca4ca3701d..121395d7ad 100644 --- a/packages/types/src/index.ts +++ b/packages/types/src/index.ts @@ -1,5 +1,6 @@ // Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. +export * from './degraded-boot.js'; export * from './env.js'; export * from './module-not-found.js'; diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index cdfc1d0467..a523c0d695 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -1936,6 +1936,9 @@ importers: '@objectstack/spec': specifier: workspace:* version: link:../../spec + '@objectstack/types': + specifier: workspace:* + version: link:../../types devDependencies: '@objectstack/driver-memory': specifier: workspace:*