fix(datasource): an explicitly-bound datasource that cannot connect refuses the boot (#3758)

.changeset/datasource-bound-connect-failfast.md

@@ -0,0 +1,60 @@
+---
+"@objectstack/service-datasource": minor
+"@objectstack/types": patch
+"@objectstack/objectql": patch
+---
+
+fix(datasource)!: a declared datasource that objects bind to must connect, or the boot fails (#3758)
+
+`DatasourceConnectionService.handleFailure()` fail-fasted only for an `external`
+datasource with `validation.onMismatch: 'fail'`. Everything else degraded to one
+`warn` line — including the case the D2 auto-connect gate itself flags as having
+**no fallback path**: a datasource that objects bind to explicitly via
+`object.datasource`. Those objects never fall through to the `default` driver;
+`engine.getDriver` throws `Datasource 'x' is not registered` for them.
+
+So an app declaring `datasource: 'analytics'` with 20 objects bound to it, booted
+against a wrong `ANALYTICS_URL`, started clean and exited zero — and then failed
+every read and write of those 20 objects with an error that reads nothing like
+*the analytics database is unreachable*. The rest of the app worked, which made it
+**harder** to locate than a total outage: it looks like "some pages are broken",
+not like a misconfigured datasource. This is the same decision #3741/#3751 fixed
+one layer up in `ObjectQLEngine.init()`; the boundary here was still drawn in the
+old place.
+
+- **Fail-fast is now keyed on "no fallback path", not on `onMismatch` alone.** At
+  the `declared-auto` (boot) trigger, a connect failure aborts the boot when the
+  datasource is `external` + `onMismatch: 'fail'` **or** when ≥1 object binds to
+  it explicitly. `autoConnect: true` with nothing bound stays lenient — that is
+  "connect it if you can", and nothing declares a dependency on it. The
+  runtime-admin create/update and boot-rehydration triggers are unchanged and
+  still always degrade: a UI action must never brick a running server.
+- **Every failure mode counts**, not just an unreachable socket: an unresolvable
+  `external.credentialsRef` (D3) and an unsupported `driver` leave the bound
+  objects exactly as dead, so they take the same verdict.
+- **The error names the bound objects** (up to 10, then `+N more`) alongside the
+  underlying cause, so the message points at the real problem instead of just the
+  datasource name. The service already receives the list for post-connect
+  `syncObjectSchema`.
+- **`connectDeclared()` attempts every gated datasource before throwing**, and
+  aggregates, so one failed boot reports all the misconfigured ones rather than
+  one per restart — the same shape as `ObjectQLEngine.init()`'s
+  `DriverConnectError`.
+- **The escape hatch is shared with the engine guard**:
+  `OS_ALLOW_DRIVER_CONNECT_FAILURE=1` now also covers this path (and covers
+  `onMismatch: 'fail'`, which previously had no opt-out). The operator intent is
+  identical — "I know the database is unreachable, boot anyway" — and two flags
+  would only guarantee one of them gets missed. When set, boot continues and a
+  `DEGRADED BOOT` banner goes to stderr as well as the logger, because `os serve`
+  swallows stdout during boot. `emitDegradedBootBanner` moved to
+  `@objectstack/types` so both call sites share one implementation;
+  `@objectstack/objectql` re-exports it unchanged.
+
+ADR-0062 D5 is amended with the new criterion and the shared flag.
+
+**Migration.** No change for a correctly configured deployment — a datasource that
+connected before still connects. A deployment that was *silently* booting with a
+dead, explicitly-bound datasource now fails the boot instead, naming the
+datasource, the cause, and the objects that depend on it; fix the datasource
+configuration. To keep booting without it — deliberately, knowing every request
+touching those objects will fail — set `OS_ALLOW_DRIVER_CONNECT_FAILURE=1`.

content/docs/data-modeling/drivers.mdx

@@ -91,6 +91,10 @@ comes back is the **boot sequence that was skipped**: nothing re-runs the schema
 sync, so those objects can be left with no tables even after the database
 returns.
 
+The same guard covers **declared datasources** whose objects have no fallback —
+see [When auto-connect fails](/docs/data-modeling/external-datasources#when-auto-connect-fails)
+([#3758](https://github.com/objectstack-ai/objectstack/issues/3758)).
+
 <Callout type="warn">
 `OS_ALLOW_DRIVER_CONNECT_FAILURE=1` boots anyway, in an explicitly degraded
 state announced by a `DEGRADED BOOT` banner. Queries to a failed driver fail

content/docs/data-modeling/external-datasources.mdx

@@ -124,6 +124,36 @@ A connected external datasource is also visible in **Setup → Datasources** (st
 `origin: code`, read-only in the UI) and via `GET /api/v1/datasources` and
 `GET /api/v1/meta/datasource`, where an admin can run the "Sync objects" wizard.
 
+### When auto-connect fails
+
+An object that binds explicitly via `datasource: '…'` has **no fallback** — it
+never falls through to the `default` driver, so an unconnected datasource means
+every read and write of that object fails. The boot therefore **refuses to start**
+when a datasource in that position cannot be connected
+([#3758](https://github.com/objectstack-ai/objectstack/issues/3758)), rather than
+leaving a server that looks healthy and errors only on the affected pages:
+
+| Gate | Connect fails at boot |
+|:---|:---|
+| **external** with `validation.onMismatch: 'fail'` | refuses the boot |
+| **objects bind explicitly** via `object.datasource` | refuses the boot — the error names them |
+| **`autoConnect: true`** with nothing bound | warning; left unconnected |
+
+This covers every reason a connect can fail — unreachable database, unresolvable
+`credentialsRef`, unsupported `driver` — because the bound objects are equally
+dead in each case. Every gated datasource is attempted before the boot aborts, so
+one failed start reports *all* the misconfigured ones.
+
+<Callout type="warn">
+`OS_ALLOW_DRIVER_CONNECT_FAILURE=1` boots anyway — the same flag as the
+[engine-level driver-connect guard](/docs/data-modeling/drivers#startup-a-driver-that-cannot-connect-aborts-the-boot) —
+in an explicitly degraded state announced by a `DEGRADED BOOT` banner. The
+datasource stays unconnected for the process lifetime: nothing re-runs the
+connect, so queries against its objects keep failing with
+`Datasource 'x' is not registered` even after the database comes back. Do not set
+it in production.
+</Callout>
+
 ## 4. Credentials
 
 Never inline a password. Put a reference in `external.credentialsRef` and store the
@@ -133,10 +163,11 @@ cleartext **at connect, before the driver is built**.
 
 Resolution is **fail-closed**: if a `credentialsRef` is declared but no secret store
 is configured, or the secret cannot be resolved/decrypted, the datasource is left
-**unconnected with a clear error** — never connected without the credential. For a
-code-defined external datasource with `validation.onMismatch: 'fail'` this fails
-the boot; otherwise it degrades with a warning. (Credential-less drivers such as
-SQLite simply have no `credentialsRef`.)
+**unconnected with a clear error** — never connected without the credential. An
+unresolvable credential is a connect failure like any other, so it fails the boot
+for the datasources listed under [When auto-connect fails](#when-auto-connect-fails)
+and degrades with a warning otherwise. (Credential-less drivers such as SQLite
+simply have no `credentialsRef`.)
 
 ## 5. Writes (double opt-in)
 

content/docs/deployment/environment-variables.mdx

@@ -49,7 +49,7 @@ read at startup unless noted otherwise. Boolean variables accept `true` / `false
 |:---|:---|:---|:---|
 | `OS_DATABASE_URL` | url | — | Database connection string (e.g. `file:./data.sqlite`, `postgres://…`, `mongodb://…`, `memory://`). `libsql://` (Turso) is not supported. |
 | `OS_DATABASE_DRIVER` | enum | inferred | Force a specific driver when the URL is ambiguous. `memory` \| `sqlite` \| `sqlite-wasm` \| `postgres` \| `mongodb`. |
-| `OS_ALLOW_DRIVER_CONNECT_FAILURE` | boolean | `false` | Escape hatch for the driver-connect boot guard. By default a data driver that fails to connect at startup **refuses the boot** — a server that cannot reach its database must not report itself started and then fail every request. Set to `1` to boot anyway, in an explicitly degraded state logged loudly at startup. There is **no reconnection**: the drivers that failed stay dead for the process lifetime and every query and schema sync routed to them fails. |
+| `OS_ALLOW_DRIVER_CONNECT_FAILURE` | boolean | `false` | Escape hatch for the driver-connect boot guard. By default a data driver that fails to connect at startup **refuses the boot** — a server that cannot reach its database must not report itself started and then fail every request. The same guard covers a **declared datasource** that objects bind to via `datasource: '…'`, or an `external` one with `validation.onMismatch: 'fail'`: those objects have no fallback datasource, so an unconnected one means they are all dead. Set to `1` to boot anyway, in an explicitly degraded state logged loudly at startup. There is **no reconnection**: whatever failed stays dead for the process lifetime and every query and schema sync routed to it fails. |
 | `OS_STORAGE_ROOT` | path | `./.objectstack/data/uploads` | Root directory for the local file storage adapter, relative to the process cwd (used by `os serve`'s default `storage` capability wiring). |
 | `OS_ARTIFACT_PATH` | path | — | Path or `http(s)://` URL to a compiled `objectstack.json` artifact to boot the kernel from. |
 

content/docs/deployment/production-readiness.mdx

@@ -90,8 +90,12 @@ the [HARDENING.md recipes](https://github.com/objectstack-ai/objectstack/blob/ma
 - [ ] `OS_ALLOW_DRIVER_CONNECT_FAILURE` is **unset**. A data driver that cannot
       connect at startup refuses the boot, so a bad `OS_DATABASE_URL`, a rotated
       password, or a closed network path surfaces as a failed deploy instead of
-      a "started" server that 500s every request. Setting the flag boots without
-      the database and never reconnects; never set it in production.
+      a "started" server that 500s every request. The same applies to a declared
+      **datasource** that objects bind to explicitly (`datasource: 'analytics'`):
+      those objects have no fallback, so a misconfigured `analytics` URL fails
+      the deploy instead of leaving a server where most pages work and that one
+      subset errors. Setting the flag boots without the database and never
+      reconnects; never set it in production.
 - [ ] Backup / restore drill documented and tested.
 - [ ] Data-retention windows reviewed (ADR-0057): the platform's default
       `lifecycle` declarations bound telemetry (activity 14d, job runs 30d,

docs/adr/0062-external-datasource-runtime.md

@@ -69,10 +69,26 @@ Code-defined datasources surface in `GET /api/v1/datasources`, `GET /api/v1/meta
 
 ### D5 — Lifecycle, health & ordering for N datasources
 
-`DatasourceConnectionService` owns connect/disconnect (graceful shutdown), pool config per datasource, and an optional health probe surfaced in the admin list (`status`). **Ordering**: all declared datasources connect **before** the `kernel:ready` external-validation gate (ADR-0015 §5.2) and before first query — i.e. during plugin init/start, not in a `kernel:ready` handler. Connect failure policy is **fail-fast for `external` with `validation.onMismatch: 'fail'`**, **degrade-with-warning** otherwise (a connectivity blip on an optional analytics replica should not brick boot).
+`DatasourceConnectionService` owns connect/disconnect (graceful shutdown), pool config per datasource, and an optional health probe surfaced in the admin list (`status`). **Ordering**: all declared datasources connect **before** the `kernel:ready` external-validation gate (ADR-0015 §5.2) and before first query — i.e. during plugin init/start, not in a `kernel:ready` handler. Connect failure policy is **fail-fast when the datasource has no fallback path**, **degrade-with-warning** otherwise (a connectivity blip on an optional analytics replica should not brick boot).
 
 > **Phase 1 implementation notes (#2163).** *Ordering* is satisfied by the kernel's two-phase boot (init-all → start-all): the connection service is registered as the `'datasource-connection'` kernel service during the datasource-admin plugin's `init()`, and declared datasources are auto-connected from `AppPlugin.start()` — which runs before the `kernel:ready` validation gate. Because boot schema-sync runs before the external driver exists, `connect()` calls `engine.syncObjectSchema()` for each bound federated object (DDL-free), so they are queryable with zero app code. *Fail-fast* is scoped to the **declared-auto** trigger; the runtime-admin create/update + boot-rehydration triggers always degrade-with-warning, preserving the pre-ADR-0062 admin behavior (a UI action never bricks the running server). *Connect policy* (the epic #2163 seam): a host-injectable `DatasourceConnectPolicy` is consulted before every connect; the open-core default allows (subject to the D2 gate), and a multi-tenant host binds a stricter, fail-closed policy for egress isolation — one connect path, no cloud fork.
 
+> **Amendment (#3758) — "no fallback path" is the fail-fast criterion, not `onMismatch:'fail'` alone.** The original wording scoped fail-fast to `external` + `validation.onMismatch: 'fail'`, which drew the line in the wrong place: **an explicit `object.datasource` binding is a hard dependency too**, and it is one the D2 note above already identifies as having *no fallback whatsoever* — such objects never resolve to the `default` driver, `engine.getDriver` throws `Datasource 'x' is not registered` for them. So a declared datasource that 20 objects bind to, failing to connect at boot, produced the worst possible shape: a server that starts clean, serves most of the app, and fails every read/write of those 20 objects with an error that reads nothing like "the analytics database is unreachable" — harder to locate than a total outage. The **declared-auto** fail-fast set is therefore:
+>
+> | | Gate | Boot connect failure |
+> |:---|:---|:---|
+> | (a) | `external` (`schemaMode !== 'managed'`) with `validation.onMismatch: 'fail'` | **fail-fast** |
+> | (b) | ≥1 object binds explicitly via `object.datasource` | **fail-fast** — no fallback path |
+> | (c) | `autoConnect: true`, nothing bound | degrade-with-warning — "connect it if you can"; nothing declares a dependency |
+>
+> This covers every reason a connect can fail, not just an unreachable socket: an unresolvable `external.credentialsRef` (D3) and an unsupported `driver` leave the bound objects exactly as dead, so they take the same verdict. `onMismatch` keeps its own meaning (what to do about a *schema* mismatch) and is no longer overloaded as the only way to say "this datasource is required".
+>
+> The escape hatch is **shared with the engine-level guard**, `OS_ALLOW_DRIVER_CONNECT_FAILURE=1` (framework#3741/#3751): the operator intent is identical ("I know the database is unreachable — boot anyway"), and two flags would only guarantee one of them gets missed. It now covers (a) as well, which previously had no opt-out. When set, the boot continues and the degraded state is announced through `emitDegradedBootBanner` (stderr as well as the logger, because `os serve`'s boot-quiet capture swallows stdout).
+>
+> The error message names the **bound objects** (up to 10, then `+N more`), not just the datasource — the service already receives them for post-connect `syncObjectSchema`. And `connectDeclared` attempts **every** gated datasource before throwing an aggregate, so one boot reports every misconfiguration rather than one per restart — the same shape as `ObjectQLEngine.init()`'s `DriverConnectError`.
+>
+> **A `DatasourceConnectPolicy` denial is not a connect failure** and stays metadata-only, unchanged. A multi-tenant host that blocks egress for a tenant's plan is making a deliberate decision about a datasource it knows it is refusing; fail-fasting there would turn a policy verdict into a boot outage for every tenant on the shared runtime. The fail-fast set covers *failures* — unreachable, unauthenticated, unsupported — not *refusals*.
+
 ### D6 — Native-analytics SQL honors the remote table/columns
 
 The analytics native-SQL strategy compiles its own `FROM "<table>"` / column references outside the driver (ADR-0015 §18 noted this). It must resolve an external object's physical table (`remoteName`/`remoteSchema`) and columns (`columnMap`) the same way `SqlDriver` now does — reusing the driver's resolution (e.g. an exposed `physicalTableFor(object)` / `physicalColumnFor(object, field)`), not a second copy. Until then, analytics over external objects stays disabled rather than silently querying the wrong table.

examples/app-showcase/src/system/datasources/showcase-external.datasource.ts

@@ -11,8 +11,15 @@ import { defineDatasource } from '@objectstack/spec/data';
  * queryable through the normal ObjectQL/REST surface.
  *
  * `schemaMode: 'external'` ⇒ ObjectStack never runs DDL here (it's a guest in a
- * database it does not own). `onMismatch: 'warn'` keeps a fixture hiccup from
- * bricking the whole showcase boot — the rest of the demo still loads.
+ * database it does not own). `onMismatch: 'warn'` keeps a *schema drift* in the
+ * fixture (a renamed column, an extra table) from bricking the showcase boot —
+ * the rest of the demo still loads.
+ *
+ * It does **not** make a failed *connection* survivable, and shouldn't: the
+ * `customer` / `order` objects bind to this datasource explicitly, so they have
+ * no fallback driver and every query against them would fail (#3758). If the
+ * fixture file cannot be opened at all, the boot stops with that as the reason
+ * rather than serving a showcase whose federation pages are quietly dead.
  *
  * It also shows up in **Setup → Integrations → Datasources** and via
  * `GET /api/v1/meta/datasource`, where an admin can run the runtime

packages/objectql/src/driver-connect-errors.ts

@@ -85,36 +85,10 @@ export class DriverConnectError extends Error {
 }
 
 /**
- * Emit the degraded-boot banner on a channel the host cannot accidentally
- * silence.
- *
- * `OS_ALLOW_DRIVER_CONNECT_FAILURE` only justifies itself if the state it opts
- * into is impossible to miss — and a logger-only banner is missable: `os serve`
- * swallows ALL of stdout while the kernel boots (its "boot-quiet" capture), and
- * `Logger` routes `warn` to stdout, so the one message that matters would be
- * invisible in exactly the situation it exists for. Writing to stderr as well
- * is the same belt-and-braces the kernel already uses for plugin startup
- * failures.
- *
- * Best-effort and never throws: falls back to `console.error`, then to silence
- * on runtimes that have neither (the logger still carries the structured
- * record either way).
+ * The degraded-boot banner now lives in `@objectstack/types` because
+ * `DatasourceConnectionService` owes the operator the same banner for the same
+ * flag (framework#3758) and must not depend on the whole query engine to print
+ * it. Re-exported here so this module stays the single import site for
+ * engine-side driver-connect failure reporting.
  */
-export function emitDegradedBootBanner(message: string): void {
-  const proc = (globalThis as {
-    process?: { stderr?: { write?: (chunk: string) => unknown } };
-  }).process;
-  try {
-    if (typeof proc?.stderr?.write === 'function') {
-      proc.stderr.write(`${message}\n`);
-      return;
-    }
-  } catch {
-    /* stderr unavailable / closed — fall through to console */
-  }
-  try {
-    (globalThis as { console?: { error?: (msg: string) => void } }).console?.error?.(message);
-  } catch {
-    /* no output channel at all — the logger record is the remaining trace */
-  }
-}
+export { emitDegradedBootBanner } from '@objectstack/types';