feat(datasource): a datasource that is down is visible, and says why when queried (#3827, #3828)

.changeset/datasource-availability-observability.md

@@ -0,0 +1,79 @@
+---
+"@objectstack/service-datasource": minor
+"@objectstack/objectql": minor
+"@objectstack/rest": patch
+"@objectstack/runtime": patch
+---
+
+feat(datasource): a datasource that is down is visible, and says why when queried (#3827, #3828)
+
+#3816 made an explicitly-bound datasource that cannot connect refuse the boot. Two
+gaps survived that fix, both in the cases that still boot — a policy denial, an
+`autoConnect` datasource, or any failure the operator waved through with
+`OS_ALLOW_DRIVER_CONNECT_FAILURE`:
+
+- **It was invisible.** `DatasourceSummary.status` was the literal `'unvalidated'`
+  for every row — the contract declared three states and the implementation only
+  ever emitted one — so a dead datasource looked exactly like a healthy-untested
+  one. `checkDriversHealth()` could not help either: it iterates registered
+  drivers, and a datasource that never connected was never registered, so it is
+  *absent* from the probe rather than unhealthy. The only trace was a warning
+  that scrolled past at boot, which made the diagnostic procedure "restart the
+  server and re-read the logs".
+- **The query-time error said nothing.** `getDriver()` answered four different
+  situations with one sentence, `Datasource 'x' is not registered.`: refused by
+  policy, failed to connect under the escape hatch, a misspelled name, and
+  `active: false`. Only the third is an authoring bug, so the other three sent
+  the reader hunting for a typo that does not exist.
+
+Both come from the same root: `connect()` already produced a `ConnectResult` for
+every attempt and every caller threw it away.
+
+- **`DatasourceConnectionService` retains the last verdict per datasource**, with a
+  coarse `availability` (`available` / `blocked` / `failed` / `unattempted`) beside
+  the raw status. New `getConnectionState(name)` / `listConnectionStates()`.
+  `disconnect()` drops it, so a removed pool stops explaining itself.
+- **`DatasourceSummary.status` tells the truth**: `ok` | `error` | `blocked` |
+  `unvalidated`, with a new operator-facing `statusReason`. `blocked` is new and
+  deliberate — a policy denial is a decision, not a fault, and will not clear on
+  its own. Reported in **Setup → Datasources**, `GET /api/v1/datasources`, and the
+  summary returned from create/update, so a "Save" whose pool failed to open is no
+  longer presented as success.
+- **`ERR_DATASOURCE_UNAVAILABLE` (HTTP 503)**: new `DatasourceUnavailableError`
+  from `@objectstack/objectql`, thrown by `getDriver()` when the connection layer
+  recorded *why* a declared datasource has no driver. An undeclared name keeps the
+  original message — there is genuinely nothing to add. 503 rather than 500/400:
+  nothing about the request is wrong, and the state may clear.
+- **A privileged/public split for the reason.** The error **never** carries the
+  underlying cause — connect failures routinely contain hosts, ports and DSNs, and
+  a policy's `reason` is written for operators. Those stay in the logs and the
+  (admin-gated) datasource list. `DatasourceConnectDecision` gains an opt-in
+  `publicReason` for hosts that want to tell tenants something specific
+  (e.g. `'External datasources require the Scale plan.'`); it is the only string
+  that reaches an end user.
+- **Readiness is deliberately not gated on this.** `/ready` still reflects
+  registered-driver health only: an optional datasource being down must not pull an
+  otherwise-working replica out of the load balancer.
+
+Also lands a drift guard for **#3826**, and corrects ADR-0062's status while doing
+it. The ADR claimed D1 ("exactly one definition → live driver path") as
+implemented; only the *construction* half converged. The `default` driver is still
+registered as a `driver.*` kernel service and connected by `ObjectQLEngine.init()`,
+with its own failure verdict, pool teardown, and no connect policy. What blocks the
+merge is an input-shape mismatch, not ordering: `connect()` takes a datasource
+*definition* and builds the driver, while `default` arrives pre-built, and routing
+it through the service would make `ObjectQLPlugin`'s boot depend on an optional
+higher-layer service. Until that is designed, `degraded-boot-parity.test.ts` pins
+both paths to the same operator-visible contract (fail-fast by default, identical
+`OS_ALLOW_DRIVER_CONNECT_FAILURE` parsing, `DEGRADED BOOT` on stderr) so a change
+to one that forgets the other fails CI — #3741 → #3758 was exactly that miss, and
+it cost three months and a second bug report.
+
+**Migration.** Additive. `DatasourceSummary.status` gains a `'blocked'` member: a
+consumer exhaustively switching on it needs a case (the admin UI shows it as a
+distinct state). Nothing that was `'ok'` or `'error'` changes meaning; rows that
+were reported `'unvalidated'` now report their real state. Query-time errors for a
+datasource the connection layer recorded change from a generic `Error` to
+`DatasourceUnavailableError` (503 instead of the previous catch-all status);
+matching on the old `is not registered` text still works for the undeclared-name
+case, which is the only one that was ever accurate.

content/docs/api/error-catalog.mdx

@@ -13,7 +13,7 @@ ObjectStack uses a structured error system with **9 error categories** and **51
 </Callout>
 
 <Callout type="warn">
-**Spec vs. wire format:** The codes below are `StandardErrorCode` — the standardized, lowercase snake_case contract that plugin/hook authors should throw with (see [Server-Side Error Handling](/docs/api/error-handling-server)). The kernel REST server (`@objectstack/rest`) that serves `/api/v1/data/*` today emits a flatter envelope with SCREAMING_SNAKE_CASE codes instead — e.g. `VALIDATION_FAILED`, `PERMISSION_DENIED`, `RECORD_NOT_FOUND`, `CONCURRENT_UPDATE` — not the codes in this catalog. See the [API Overview](/docs/api#error-handling) for both wire formats in use and [Wire Format](/docs/api/wire-format#7-error-response-format) for JSON examples.
+**Spec vs. wire format:** The codes below are `StandardErrorCode` — the standardized, lowercase snake_case contract that plugin/hook authors should throw with (see [Server-Side Error Handling](/docs/api/error-handling-server)). The kernel REST server (`@objectstack/rest`) that serves `/api/v1/data/*` today emits a flatter envelope with SCREAMING_SNAKE_CASE codes instead — e.g. `VALIDATION_FAILED`, `PERMISSION_DENIED`, `RECORD_NOT_FOUND`, `CONCURRENT_UPDATE`, `ERR_DATASOURCE_UNAVAILABLE` — not the codes in this catalog. See the [API Overview](/docs/api#error-handling) for both wire formats in use and [Wire Format](/docs/api/wire-format#7-error-response-format) for JSON examples.
 </Callout>
 
 ---

content/docs/api/wire-format.mdx

@@ -380,6 +380,28 @@ Returned when an `If-Match` / `expectedVersion` token no longer matches the stor
 }
 ```
 
+### Datasource Unavailable — `503 Service Unavailable`
+
+Returned when the object's declared `datasource` has no live driver: the host's
+connect policy refused it, or it failed to connect at startup and the server was
+started with `OS_ALLOW_DRIVER_CONNECT_FAILURE`. Nothing about the request is
+wrong, and the state may clear — so this is a `503`, not a `400` or a `500`.
+
+`reason` is the class (`blocked` | `failed`). The underlying cause is
+deliberately **not** included — it routinely contains hosts, ports and DSNs; it
+stays in the server logs and **Setup → Datasources**. See
+[External Datasources](/docs/data-modeling/external-datasources#what-a-query-against-an-unusable-datasource-says).
+
+```json
+{
+  "error": "[ObjectQL] Datasource 'analytics' configured for object 'visit' is declared but not connected: the host's datasource connect policy refused it. See the startup logs or Setup → Datasources for the cause.",
+  "code": "ERR_DATASOURCE_UNAVAILABLE",
+  "datasource": "analytics",
+  "reason": "blocked",
+  "object": "visit"
+}
+```
+
 <Callout type="info">
 **Error Codes:** See the [Error Catalog](/docs/api/error-catalog) for the complete list of error codes and their meanings.
 </Callout>

content/docs/data-modeling/external-datasources.mdx

@@ -149,11 +149,66 @@ one failed start reports *all* the misconfigured ones.
 [engine-level driver-connect guard](/docs/data-modeling/drivers#startup-a-driver-that-cannot-connect-aborts-the-boot) —
 in an explicitly degraded state announced by a `DEGRADED BOOT` banner. The
 datasource stays unconnected for the process lifetime: nothing re-runs the
-connect, so queries against its objects keep failing with
-`Datasource 'x' is not registered` even after the database comes back. Do not set
-it in production.
+connect, so queries against its objects keep failing until the server is
+restarted. Do not set it in production.
 </Callout>
 
+### Seeing the state of a datasource
+
+Every connect attempt's verdict is retained, so a datasource that is down no
+longer has to be diagnosed by restarting the server and re-reading boot logs
+([#3827](https://github.com/objectstack-ai/objectstack/issues/3827)).
+
+**Setup → Datasources** and `GET /api/v1/datasources` report a `status` per
+datasource:
+
+| `status` | Meaning |
+|:---|:---|
+| `ok` | A live driver is registered and routable. |
+| `error` | A connect was attempted and failed. `statusReason` carries the cause. |
+| `blocked` | The host's connect policy refused it — a decision, not a fault. It will not clear on its own. |
+| `unvalidated` | No connect attempted: a `managed` datasource left metadata-only by the gate above, or a runtime row nobody has tested. |
+
+`statusReason` is **operator-facing** and may name hosts, ports, or internal
+plans — this surface is already admin-gated.
+
+Note that `checkDriversHealth()` (and therefore `/ready`) cannot see these: a
+datasource that never connected was never registered as a driver, so it is
+*absent* from the health probe rather than reported unhealthy. Readiness is
+deliberately **not** gated on them — an optional datasource being down must not
+pull an otherwise-working replica out of the load balancer.
+
+### What a query against an unusable datasource says
+
+An object bound to a datasource with no live driver fails with
+`ERR_DATASOURCE_UNAVAILABLE` (HTTP **503**), and the message says which situation
+it is ([#3828](https://github.com/objectstack-ai/objectstack/issues/3828)):
+refused by policy, or a connect that failed under
+`OS_ALLOW_DRIVER_CONNECT_FAILURE`. A datasource name that was never declared
+still gets the original `is not registered` — that one really is an authoring
+bug, and there is nothing to add.
+
+The error **never carries the underlying cause**: connect errors routinely
+contain hosts, ports and DSNs, and a policy's `reason` is written for operators.
+Both stay in the logs and the admin list. A host that wants to tell tenants
+something specific sets `publicReason` on its connect decision — opt-in, and the
+only string that reaches an end user:
+
+```typescript
+const policy: DatasourceConnectPolicy = {
+  canConnect: (ds) =>
+    allowedFor(tenant, ds)
+      ? { allow: true }
+      : {
+          allow: false,
+          // operator-facing: logs + Setup → Datasources only
+          reason: `egress allow-list miss for ${ds.name} (${tenant.id}, plan=${tenant.plan})`,
+          // tenant-facing: appended to the query-time error
+          publicReason: 'External datasources require the Scale plan.',
+        },
+};
+```
+
 ## 4. Credentials
 
 Never inline a password. Put a reference in `external.credentialsRef` and store the

docs/adr/0062-external-datasource-runtime.md

@@ -1,6 +1,6 @@
 # ADR-0062: External Datasource Runtime — connection lifecycle, credentials, visibility & query completeness
 
-**Status**: Accepted (2026-06-22) — D1–D8 implemented (`service-datasource` connection service + opt-in-safe gate + fail-closed connect policy; native-SQL declines external per D6; D7 lint in `validate-expressions.ts`).
+**Status**: Accepted (2026-06-22) — D2–D8 implemented (`service-datasource` connection service + opt-in-safe gate + fail-closed connect policy; native-SQL declines external per D6; D7 lint in `validate-expressions.ts`). **D1 partially implemented**: declared datasources auto-connect through the one service, but the `default` driver still has its own connect + failure path — see the status correction under D1 (#3826).
 
 **Supersedes the runtime portions of**: ADR-0015 §18 addendum (kept as the historical record). ADR-0015 remains the canonical spec/binding decision; this ADR is the canonical *runtime* decision.
 
@@ -53,6 +53,21 @@ R2/R3/R8 are the same change seen from three angles — you cannot "auto-connect
 
 Introduce a single service that, given a datasource definition, builds a driver via the **injected** driver factory (the same `createDefaultDatasourceDriverFactory` used by the runtime-admin path), connects it, and registers it into the ObjectQL engine under the datasource name. `app-plugin` calls it for every declared datasource (in addition to today's `registerInMemory` for visibility); `standalone-stack`'s single-`default`-driver bootstrap is refactored to go through the same service so there is exactly one "definition → live driver" code path. `engine.registerDriver` remains the sink. The `onEnable` bridge becomes unnecessary for the common case (kept as an escape hatch — D8).
 
+> **Status correction (#3826) — the `default` refactor is NOT done; the "exactly one code path" claim is half-true.** The *construction* half converged: `standalone-stack` builds the `default` driver through `createDefaultDatasourceDriverFactory` (`standalone-stack.ts`, "the SAME `create({driver,config})` used for declared/runtime datasources"). The *connect + failure-verdict* half did not, and this ADR's header has been claiming D1 as implemented while two independent implementations coexist:
+>
+> | | `default` | declared datasource |
+> |:---|:---|:---|
+> | build | shared factory ✅ | shared factory ✅ |
+> | register | `DriverPlugin` → a `driver.*` kernel service, discovered in `ObjectQLPlugin.start()` | `engine.registerDriver()` inside `connect()` |
+> | connect | `ObjectQLEngine.init()` | `DatasourceConnectionService.connect()` |
+> | failure verdict | `DriverConnectError`, aggregate (#3741) | `handleFailure()` (#3758) |
+> | pool teardown | kernel shutdown via `DriverPlugin` | `DatasourceConnectionService.disconnect()` |
+> | connect policy | not consulted | `DatasourceConnectPolicy` |
+>
+> **What actually blocks the merge is an input-shape mismatch, not ordering.** The kernel's init-all-then-start-all means the connection service *does* exist by `ObjectQLPlugin.start()`, so timing is available. The obstacles are: (1) `DatasourceConnectionService.connect()` takes a datasource *definition* and **builds** the driver, while `default` arrives as an already-constructed driver instance published as a `driver.*` kernel service — there is no "adopt this driver" entry point; and (2) routing `default` through the service would make `ObjectQLPlugin`'s boot depend on an **optional service from a higher layer** (`service-datasource`), inverting the layering for the one driver every app needs. Closing this means either adding an `adoptDriver()` seam to the connection service, or making the standalone `default` a real declared datasource definition — a design decision, not a mechanical move, and still the riskiest single step per §Risk.
+>
+> Until then the divergence is guarded rather than assumed: `packages/runtime/src/degraded-boot-parity.test.ts` pins both paths to the same operator-visible contract (fail-fast by default, identical `OS_ALLOW_DRIVER_CONNECT_FAILURE` parsing, `DEGRADED BOOT` on stderr), so a change to one that forgets the other fails CI instead of shipping. #3741 → #3758 was exactly that miss, and it cost three months and a second bug report.
+
 ### D2 — Connect is opt-in-safe: existing managed apps are byte-for-byte unchanged
 
 Auto-connect must not change apps that today declare datasources that are *decorative* or routed via `datasourceMapping` (e.g. `examples/app-crm`'s `crm_primary`/`crm_analytics`). Gate auto-connect so a declared datasource is only connected when it is meaningfully addressed: **(a)** it is `external` (`schemaMode !== 'managed'`), or **(b)** an object/`datasourceMapping` actually routes to it, or **(c)** it sets an explicit `autoConnect: true`. A managed datasource that nothing routes to stays metadata-only (today's behavior). The `default` datasource keeps its current dedicated bootstrap. This is the load-bearing backward-compat decision.

packages/objectql/src/core.ts

@@ -42,8 +42,13 @@ export type { ObjectQLHostContext, HookHandler, HookEntry, OperationContext, Eng
 // Boot guard: thrown by `ObjectQL.init()` when a registered driver's connect()
 // fails (framework#3741). Embedders that boot the engine themselves can catch
 // it to render their own "database unreachable" message.
-export { DriverConnectError } from './driver-connect-errors.js';
-export type { DriverConnectFailure, DriverHealth } from './driver-connect-errors.js';
+export { DriverConnectError, DatasourceUnavailableError } from './driver-connect-errors.js';
+export type {
+  DriverConnectFailure,
+  DriverHealth,
+  DatasourceUnavailableInfo,
+  DatasourceUnavailableKind,
+} from './driver-connect-errors.js';
 
 // In-memory aggregation fallback
 export { applyInMemoryAggregation, bucketDateValue } from './in-memory-aggregation.js';