feat(driver-sql): run file-backed SQLite in WAL journal mode (#3941)

.changeset/sqlite-wal-journal-mode.md

@@ -0,0 +1,65 @@
+---
+'@objectstack/driver-sql': minor
+'@objectstack/driver-sqlite-wasm': patch
+'@objectstack/cli': patch
+---
+
+File-backed SQLite now runs `journal_mode = WAL` (#3941).
+
+`SqlDriver.connect()` set `auto_vacuum` and left the journal mode alone, so
+every ObjectStack SQLite database ran SQLite's built-in default — a rollback
+journal. That is the worst mode for the shape this platform actually has, which
+is **several processes on one file**: a dev server, `os migrate`,
+`os meta resync`, a test run. Measured, on the same file:
+
+| | rollback journal | WAL |
+|:---|:---|:---|
+| writer while another process holds a read open | `SQLITE_BUSY` — committing needs an exclusive lock | proceeds |
+| idle attached connection visible to SQL | no — a lock lasts only as long as its transaction | yes (`locking_mode = EXCLUSIVE` + `BEGIN IMMEDIATE` reports busy) |
+
+The second row is why the `os migrate` occupancy check had to inspect file
+descriptors to see a live server at all (#3940): under a rollback journal there
+was nothing in the database to see. That signal stays — it names the process,
+which WAL's lock probe cannot — but the SQL probe is now authoritative for
+databases ObjectStack created rather than a fallback that was blind in practice.
+Concurrent *writers* still serialize; SQLite allows one at a time in any mode.
+
+Journal mode is a persistent property of the file, so an existing database is
+converted in place on the next connect (a header change — no rows are touched)
+and stays converted. Two consequences to plan for:
+
+- `app.db-wal` / `app.db-shm` exist beside the database while a connection is
+  attached, and `app.db-wal` can hold committed transactions. A clean shutdown
+  checkpoints them away; a naive copy of `app.db` alone while a server runs does
+  not. Use `sqlite3 app.db ".backup …"`.
+- **WAL does not work on network filesystems** (NFS/SMB). Opt out with
+  `OS_DATABASE_SQLITE_JOURNAL_MODE=delete`, or per datasource with
+  `sqliteJournalMode: 'delete'` in the driver config (which outranks the env
+  var). Either form *applies* `delete`, so it also converts a database that
+  already adopted WAL back — skipping would have stranded it.
+
+Nothing here fails a boot, and nothing is assumed: `PRAGMA journal_mode = X`
+answers with the mode actually in force rather than raising on refusal, so the
+reply is read back; and because a filesystem can accept WAL and then fail the
+first read *through* it, the mode is proven with a read and rolled back to
+`delete` if that fails — with a warning naming the file and the escape hatch.
+`synchronous` is untouched, so durability is exactly what it was. `:memory:`
+databases are left alone, as is `auto_vacuum = INCREMENTAL`, which keeps
+reclaiming under WAL (ADR-0057).
+
+`os db clean` now counts `-wal` / `-shm` as part of the database when it measures
+what a `VACUUM` reclaimed, so bytes that were sitting in the log do not read as a
+reclaim of zero.
+
+`@objectstack/driver-sqlite-wasm` deliberately stays out of WAL. Its live
+database is in the WASM heap and what reaches disk is a byte image it exports, so
+nothing reads the database across processes and the pragma buys it nothing —
+while still being a persistent header change in the operator's file. sql.js
+*accepts* the pragma (its VFS is memory-backed), so this had to be declared
+rather than discovered.
+
+It also now parks a `-wal` left behind by an unclean native-driver exit rather
+than loading the image beside it: wasm SQLite cannot read that log, and leaving
+it next to a freshly rewritten image would let a later real SQLite replay frames
+that no longer belong to it. The warning names the file it parked and how to
+recover what was in it.

content/docs/data-modeling/drivers.mdx

@@ -279,6 +279,64 @@ export OS_DATABASE_URL=file:./data/app.db
 export OS_DATABASE_URL=:memory:
 ```
 
+### Journal mode: WAL, and cross-process access
+
+A file-backed SQLite database is switched to **WAL** (write-ahead logging) the
+first time ObjectStack connects to it. SQLite's own default is a rollback
+journal (`journal_mode = delete`), which is the wrong trade for the way this
+platform is actually used — **several processes on one file**: a dev server,
+`os migrate`, `os meta resync`, a test run.
+
+| | rollback journal (SQLite default) | WAL (ObjectStack default) |
+|:---|:---|:---|
+| Reader while another process writes | allowed until the writer commits | always allowed (last committed snapshot) |
+| Writer while another process reads | **blocked** — committing needs an exclusive lock (`SQLITE_BUSY`) | allowed |
+| Idle connection visible to other processes | no — a lock lasts only as long as its transaction | yes, which is what makes the [`os migrate` occupancy check](/docs/deployment/cli#occupancy-check-sqlite) reliable |
+| Files on disk | `app.db` (plus a transient `app.db-journal`) | `app.db`, plus `app.db-wal` / `app.db-shm` while a connection is attached |
+
+Concurrent *writers* still serialize — SQLite allows one at a time in any mode.
+
+Two consequences worth knowing:
+
+- **Journal mode is stored in the file.** One connection sets it and it stays
+  set, so an existing database is converted in place on the next boot (a header
+  change; no rows are touched) and stays in WAL even when opened by other tools.
+- **`app.db-wal` can hold committed data.** A clean shutdown checkpoints and
+  removes it, but do not copy or restore `app.db` on its own while a server is
+  attached — use `sqlite3 app.db ".backup …"`, as
+  [Backup & restore](/docs/deployment/backup-restore#sqlite-single-host)
+  describes.
+
+**Opting out.** WAL requires shared memory beside the database and therefore
+**does not work on network filesystems** (NFS/SMB). Set the journal mode back
+per deployment:
+
+```bash
+export OS_DATABASE_SQLITE_JOURNAL_MODE=delete
+```
+
+…or per datasource, which outranks the env var:
+
+```typescript
+new SqlDriver({
+  client: 'better-sqlite3',
+  connection: { filename: '/mnt/share/app.db' },
+  useNullAsDefault: true,
+  sqliteJournalMode: 'delete',
+});
+```
+
+Either form *applies* `delete`, so it also converts a database that already
+adopted WAL back. The driver never assumes the switch worked: it reads the mode
+back and, if WAL is accepted but cannot be read through (the usual network-share
+symptom), reverts to `delete` and says so. A database that stays on a rollback
+journal is slower under concurrency, never broken.
+
+`:memory:` databases and the WASM SQLite driver are left on their own journal
+mode. Neither is shared between processes: the WASM driver's live database sits
+in the WASM heap and what reaches disk is a byte image it exports, so there is no
+cross-process concurrency for WAL to buy.
+
 ### Space Reclamation & Table Rotation (ADR-0057)
 
 The SQL driver connects SQLite with `auto_vacuum=INCREMENTAL` and exposes

content/docs/deployment/backup-restore.mdx

@@ -57,6 +57,17 @@ sqlite3 /srv/data/app.db ".backup '/backups/app-$(date +%F).db'"
 The default location when `OS_DATABASE_URL` is unset is
 `<home>/data/objectstack.db` under the ObjectStack home directory.
 
+<Callout type="warn">
+ObjectStack keeps file-backed SQLite in
+[WAL mode](/docs/data-modeling/drivers#journal-mode-wal-and-cross-process-access),
+so `app.db-wal` beside the database can hold committed transactions that are not
+yet in `app.db`. `.backup` accounts for that; copying `app.db` on its own does
+not. If you must copy files, stop the server first (a clean shutdown checkpoints
+and removes `-wal`/`-shm`), or copy `app.db`, `app.db-wal` and `app.db-shm`
+together — a database restored beside a *stale* log is worse than one restored
+without it.
+</Callout>
+
 ### Managed databases (hosted Postgres, MongoDB Atlas, …)
 
 Use the provider's snapshot, branching, or point-in-time-restore facilities.
@@ -87,7 +98,9 @@ Fresh container or VM. Install Node 22+ and `@objectstack/cli`.
 
 ```bash
 pg_restore --clean --if-exists -d "$OS_DATABASE_URL" backup-2026-07-14.dump
-# or copy the SQLite file / restore the provider snapshot
+# or restore the provider snapshot
+# SQLite: put the `.backup` file in place with the server stopped, and make sure
+# no stale app.db-wal / app.db-shm is left beside it (see the warning above)
 ```
 
 ### Provide the original secrets

content/docs/deployment/cli.mdx

@@ -541,15 +541,16 @@ schema cookie that the migration invalidates, and its writes can collide as
 `SQLITE_BUSY`. Before booting, `os migrate` checks two things:
 
 1. **Which processes hold the file open** (`/proc` on Linux, `lsof` on macOS).
-   This is the signal that fires in practice. ObjectStack's sqlite driver runs
-   `journal_mode = delete`, and a rollback-journal database keeps no record of
-   who is attached — an idle server holds no lock at all, so no SQL probe can
-   see it. An open file descriptor is what it does leave, and it names the
-   process to go and stop.
+   The signal that works in every journal mode, and the only one that names the
+   process to go and stop. It is also the only one that sees an *idle* server on
+   a rollback-journal database, where a lock lasts no longer than the
+   transaction that took it.
 2. **A SQL lock probe** (`PRAGMA locking_mode = EXCLUSIVE` under
-   `busy_timeout = 0`). On a WAL database this catches an attached connection
-   even when step 1 cannot run — a platform without process inspection, or a
-   database held by another user's process.
+   `busy_timeout = 0`). ObjectStack keeps file-backed SQLite in
+   [WAL mode](/docs/data-modeling/drivers#journal-mode-wal-and-cross-process-access),
+   where this catches any attached connection — idle or not — including the cases
+   step 1 cannot reach: a platform without process inspection, or a database held
+   by another user's process.
 
 Either one firing counts as busy. Both are non-destructive: no row is read or
 written.

content/docs/deployment/environment-variables.mdx

@@ -49,6 +49,7 @@ read at startup unless noted otherwise. Boolean variables accept `true` / `false
 |:---|:---|:---|:---|
 | `OS_DATABASE_URL` | url | — | Database connection string (e.g. `file:./data.sqlite`, `postgres://…`, `mongodb://…`, `memory://`). `libsql://` (Turso) is not supported. |
 | `OS_DATABASE_DRIVER` | enum | inferred | Force a specific driver when the URL is ambiguous. `memory` \| `sqlite` \| `sqlite-wasm` \| `postgres` \| `mongodb`. |
+| `OS_DATABASE_SQLITE_JOURNAL_MODE` | enum | `wal` | Journal mode for **file-backed** SQLite. `wal` (default) lets a dev server and CLI commands share one file without blocking each other, and is what makes the `os migrate` occupancy check reliable. Set to `delete` for SQLite's rollback journal — required when the database lives on a **network filesystem** (NFS/SMB), where WAL cannot work. The setting is applied, not merely skipped: `delete` converts a database that already adopted WAL back. Ignored for `:memory:`, for the WASM SQLite driver, and for non-SQLite drivers. A per-datasource `sqliteJournalMode` in driver config outranks it. See [Journal mode](/docs/data-modeling/drivers#journal-mode-wal-and-cross-process-access). |
 | `OS_ALLOW_DRIVER_CONNECT_FAILURE` | boolean | `false` | Escape hatch for the driver-connect boot guard. By default a data driver that fails to connect at startup **refuses the boot** — a server that cannot reach its database must not report itself started and then fail every request. The same guard covers a **declared datasource** that objects bind to via `datasource: '…'`, or an `external` one with `validation.onMismatch: 'fail'`: those objects have no fallback datasource, so an unconnected one means they are all dead. Set to `1` to boot anyway, in an explicitly degraded state logged loudly at startup. There is **no reconnection**: whatever failed stays dead for the process lifetime and every query and schema sync routed to it fails. |
 | `OS_STORAGE_ROOT` | path | `./.objectstack/data/uploads` | Root directory for the local file storage adapter, relative to the process cwd (used by `os serve`'s default `storage` capability wiring). |
 | `OS_ARTIFACT_PATH` | path | — | Path or `http(s)://` URL to a compiled `objectstack.json` artifact to boot the kernel from. |

content/docs/releases/v17.mdx

@@ -990,6 +990,17 @@ most of its rows in silence.
   state, and requires each to be classified (`scope` / `closed` / `open` /
   `output`) with a rationale. Omission is the commonest authoring error a model
   makes; it must not also be the widest grant.
+- **File-backed SQLite runs in WAL mode** (#3941). SQLite's built-in rollback
+  journal makes a writer wait for every reader and leaves an idle connection
+  invisible to SQL — both wrong for the platform's normal shape, several
+  processes on one file (a dev server, `os migrate`, a test run). The driver now
+  switches such a database to WAL on connect, which is also what lets the
+  `os migrate` occupancy check *see* an idle server instead of inferring one from
+  file descriptors. Two things to know: `app.db-wal` / `app.db-shm` appear beside
+  the database while a connection is attached (so back up with
+  `sqlite3 … ".backup"`, never a bare file copy), and WAL cannot work on a
+  **network filesystem** — set `OS_DATABASE_SQLITE_JOURNAL_MODE=delete` there,
+  which converts an already-switched database back.
 - Metadata-plane FLS (per-caller masking) is proposed as ADR-0106.
 
 ## New in Console (Studio) — bundled objectui 17.0

packages/cli/src/commands/db/clean.ts

@@ -68,9 +68,20 @@ export default class DbClean extends Command {
 
     const { resolveSqliteDriver } = await import('@objectstack/service-datasource');
 
+    // Count the WAL sidecars as part of the database (#3941): a file-backed
+    // SQLite database runs in WAL mode, so bytes waiting in `-wal` are bytes on
+    // disk. Measuring `app.db` alone would compare a before that excludes them
+    // against an after that has absorbed them — and report a reclaim of zero (or
+    // less) for a VACUUM that genuinely shrank things.
+    const sizeOnDisk = (file: string): number =>
+      [file, `${file}-wal`, `${file}-shm`].reduce(
+        (total, path) => total + (existsSync(path) ? statSync(path).size : 0),
+        0,
+      );
+
     let failed = false;
     for (const file of targets) {
-      const before = statSync(file).size;
+      const before = sizeOnDisk(file);
       try {
         const resolved = await resolveSqliteDriver({
           filename: file,
@@ -86,7 +97,9 @@ export default class DbClean extends Command {
         await resolved.driver.execute('VACUUM');
         await resolved.driver.disconnect();
 
-        const after = statSync(file).size;
+        // Measured after `disconnect()`, which checkpoints and removes the
+        // sidecars — so this is the settled on-disk size either way.
+        const after = sizeOnDisk(file);
         const saved = before - after;
         const fmt = (n: number) => `${(n / 1024 / 1024).toFixed(2)} MB`;
         console.log(

packages/cli/src/utils/sqlite-occupancy.test.ts

@@ -165,12 +165,15 @@ describe('probeSqliteOccupancy', () => {
   });
 
   /**
-   * The dogfood regression. ObjectStack's sqlite driver runs
-   * `journal_mode = delete`, and an idle connection to a rollback-journal
-   * database holds NO lock — so the SQL probe reports idle and, before the
+   * The dogfood regression. An idle connection to a rollback-journal database
+   * holds NO lock — so the SQL probe reports idle and, before the
    * file-descriptor signal existed, `os migrate apply` ran unannounced against
    * a live `os serve`. Verified by hand against a real CRM project before this
    * test was written; the test is what keeps it fixed.
+   *
+   * Still reachable after #3941 made WAL the driver default: a database on a
+   * filesystem that cannot host WAL, or one explicitly opted back out with
+   * `OS_DATABASE_SQLITE_JOURNAL_MODE=delete`, lands here again.
    */
   it('reports busy for a ROLLBACK-JOURNAL database an idle connection holds open', async () => {
     const file = newDb('busy-journal-idle.db', 'delete');

packages/cli/src/utils/sqlite-occupancy.ts

@@ -15,35 +15,40 @@
  * ## Two independent signals, because neither covers the ground alone
  *
  * **1. Which processes hold the file open** (`/proc` on Linux, `lsof` on
- * macOS). This is the signal that actually fires in practice, and it took
- * dogfooding to learn why: ObjectStack's sqlite driver runs
- * `journal_mode = delete`, NOT WAL. A rollback-journal database keeps no
- * persistent record of who is attached — locks exist only for the duration of
- * a transaction — so an idle `os serve` holds no lock at all and no amount of
- * SQL can see it. An open file descriptor is the only thing it does leave, and
- * it names the culprit, which is what an operator needs anyway.
+ * macOS). The signal that survives every journal mode, and the only one that
+ * names the culprit — "stop pid 12367 (node)" is worth more to an operator than
+ * "something is using it". It is also the only one that fires on a
+ * rollback-journal database, where locks exist just for the duration of a
+ * transaction, so an idle `os serve` holds nothing any query could see. That is
+ * what dogfooding caught (#3940), back when every ObjectStack database ran
+ * `journal_mode = delete`.
  *
  * **2. A SQL lock probe** — `PRAGMA locking_mode = EXCLUSIVE` then
- * `BEGIN IMMEDIATE` under `busy_timeout = 0`. On WAL this catches an attached
- * connection even when signal 1 cannot run (a foreign platform, a hardened
- * container with no `/proc` visibility, a connection held by another *user's*
- * process); on a rollback journal it narrows to "someone is mid-write". Kept
- * because it is authoritative where it applies and costs a millisecond.
+ * `BEGIN IMMEDIATE` under `busy_timeout = 0`. Since #3941 the driver keeps
+ * file-backed databases in **WAL**, where this reports `SQLITE_BUSY` against an
+ * *attached* connection whether or not it is doing anything — so for a database
+ * ObjectStack created it is authoritative, and it reaches the cases signal 1
+ * cannot: a platform with no process inspection, a hardened container with no
+ * `/proc` visibility, a connection held by another *user's* process. On a
+ * database still (or deliberately) on a rollback journal it narrows back to
+ * "someone is mid-write", which is why signal 1 stays first.
  *
  * Either signal firing means busy. Three rejected alternatives, all measured
  * rather than assumed:
  *
- * - **`-wal` / `-shm` presence alone.** Correct while a connection is open, but
- *   it cannot tell a live server from a crashed one, and it does not exist at
- *   all on a rollback journal — which is what we actually ship. Sidecars are
- *   reported as supporting evidence, never as the verdict.
+ * - **`-wal` / `-shm` presence alone.** Correct while a connection is open — a
+ *   clean close checkpoints and removes them — but it cannot tell a live server
+ *   from a crashed one, and it says nothing at all about a database on a
+ *   rollback journal. Sidecars are reported as supporting evidence, never as
+ *   the verdict.
  * - **`PRAGMA wal_checkpoint(TRUNCATE)`.** Reports `busy = 1` only when a
  *   *writer* is mid-transaction; an attached-but-idle connection checkpoints
  *   cleanly. Misses precisely the common case.
  * - **The SQL probe on its own.** What shipped first, and what dogfooding
  *   caught: against a real `os serve` holding a real project database, it
  *   reported idle and the migration ran unannounced. Correct on WAL, blind on
- *   the mode the platform defaults to.
+ *   the mode the platform defaulted to then — and still blind on any database
+ *   opted back out to `delete`, so it does not get to be the only signal.
  *
  * Both probes are non-destructive: an empty transaction rolled back with
  * locking mode restored, and read-only filesystem inspection. No row is read

packages/plugins/driver-sql/src/index.ts

@@ -5,6 +5,7 @@ import { SqlDriver } from './sql-driver.js';
 export { SqlDriver };
 export type {
   SqlDriverConfig,
+  SqliteJournalMode,
   IntrospectedSchema,
   IntrospectedTable,
   IntrospectedColumn,