Skip to content

feat(driver-sql): run file-backed SQLite in WAL journal mode (#3941) - #4072

Merged
os-zhuang merged 3 commits into
mainfrom
claude/sqlite-journal-mode-delete-osacn0
Jul 30, 2026
Merged

feat(driver-sql): run file-backed SQLite in WAL journal mode (#3941)#4072
os-zhuang merged 3 commits into
mainfrom
claude/sqlite-journal-mode-delete-osacn0

Conversation

@os-zhuang

Copy link
Copy Markdown
Contributor

#3941 的 Ask 实现:SqlDriver.connect()文件型 SQLite 切到 WAL。issue 里点名的四条代价(网络文件系统、单向迁移、多出的 sidecar 文件、与 auto_vacuum 的交互)逐条处理,没有留给读者。

先把两边的事实测出来

在同一个文件上用 better-sqlite3 实测(不是照抄 SQLite 文档):

场景 rollback journal WAL
读方开着读事务时写方提交 SQLITE_BUSY —— 提交要拿 EXCLUSIVE,与任何读者互斥 通过
写方开着写事务时读方开新读事务 通过(阻塞窗口在 commit 那一刻,不是整个事务) 通过
空闲的已附着连接能否被 SQL 看到 看不到 —— 锁只活在事务期间 看到locking_mode = EXCLUSIVE + BEGIN IMMEDIATE 报 busy)
磁盘上的文件 app.db 连接期间多 app.db-wal / app.db-shm,干净关闭后自动消失

第三行就是 #3924 第一版占用检查失效、#3940 只能改去查文件描述符的原因。端到端复验(真 SqlDriver 持有文件、进程外探测):

probe    : busy signal=file_open
message  : …/app.db is in use — it is open in pid 14202 (node) (app.db-wal, app.db-shm present).
sql probe: journal=wal -> busy (SQLITE_BUSY)

最后一行是关键 —— 对方是空闲的。改之前这里报 idle,迁移就那么跑过去了。文件描述符信号继续保留并且仍然排第一:它能报出进程名,WAL 的锁探测报不出。顺带确认 knex 给 sqlite 的池是 min:1, max:1,所以一个空闲的 os serve 确实一直挂着一条连接 —— 这条通路是成立的,不是推测。

并发在任何模式下都还是串行(SQLite 只允许一个写者),文档里写清楚了,没有夸大。另外 better-sqlite3 自带 busy_timeout = 5000,所以不需要顺手加超时。

两种「不响」的失败,都不靠猜

  1. 拒绝是一个答案,不是一个异常。 PRAGMA journal_mode = X 返回的是当前真正生效的模式:memory:memory,撑不住 WAL 的文件系统回 delete,都不抛。所以回复一律读回来比对,绝不假定成功。

  2. 接受了,然后用不了。 WAL 需要数据库旁边的共享内存索引,网络文件系统(NFS/SMB)给不了 —— 模式切换可能成功,而第一次穿过 WAL 的读才失败,此时模式已经落进文件头。所以每次连到 WAL 库都用一次 SELECT count(*) FROM sqlite_master(任何读都会打开 shm,这是最便宜的)证明它能用,失败就回退 delete

    两个细节:验证不只在切换后跑,而是每次连接都跑 —— 否则被上一次启动搁在 WAL 里的库只有一次自救机会;而 SQLITE_BUSY 不算用不了(占用检查自己就会短暂持 EXCLUSIVE 锁),那种情况直接放过,不会把一个健康的库拽出 WAL。

逃生阀是「施加」而不是「跳过」

因为 journal mode 是文件的持久属性,跳过会把已经转过去的库永远留在 WAL 里。所以两种写法都会真的执行 PRAGMA journal_mode = delete,把库转回来:

export OS_DATABASE_SQLITE_JOURNAL_MODE=delete     # 每部署
new SqlDriver({, sqliteJournalMode: 'delete' }) // 每数据源,优先级高于环境变量

无法识别的值会报出来并继续用 wal,而不是当成静默 opt-out —— 一个拼写错误不该让库悄悄留在串行模式。全程不抛、不拦启动:留在 rollback journal 的库只是并发更差,不是坏了。synchronous 一个字没动,持久性与之前完全一致;auto_vacuum = INCREMENTAL 实测在 WAL 下照常回收(ADR-0057 的路径有测试盯着)。

wasm 驱动明确不进 WAL,理由改过一次

第一版我写的理由是「它靠导出字节镜像持久化,WAL 里的已提交事务会被丢掉 —— 数据丢失」。实测推翻了这条:sql.js 的 export() 会先关库再重开,关库即 checkpoint,行没丢(并且它会从镜像文件头继承 wal,因为 VFS 是内存的,压根不会拒绝这个 pragma)。所以理由改成真实的那个:它没有跨进程共享(活库在 WASM 堆里,别人读到的永远是导出的快照),pragma 对它零收益,却是往用户文件头里写一个持久标记,还让导出路径的正确性挂在 sql.js「关库即 checkpoint」这个实现细节上。声明式关掉,而不是等基类去发现。

顺带补掉这次改动新开的一个口子:原生驱动被 kill -9 后会留下 <db>-wal(干净关闭不会)。wasm 只读主镜像,读不到那段日志;更糟的是它下次 flush 会重写镜像,而把一个不匹配的日志留在旁边,之后真 SQLite 打开时会把不属于这个镜像的 frame 重放上去。所以把它改名挪开(字节留着可恢复),并在警告里说清丢了什么、怎么用真 sqlite3 捞回来。

os db clean 量体积时把 -wal/-shm 计进去,否则躺在日志里的字节会让一次真的瘦身报成 0。

验证

  • 全仓 turbo run test --force:132/132 任务通过,0 缓存(合并 main 之前);合并当前 main 后重建并重跑 driver-sql / driver-sqlite-wasm / cli,全绿(54 + 12 + 80 文件)。dogfood 那 71 个文件 / 410 个用例是真起服务器打真 SQLite 文件,是这次最有分量的集成信号。
  • 新增 14 个 driver-sql 用例 + 3 个 wasm 用例,覆盖:新库进 WAL、旧 rollback 库原地转换且行不丢、auto_vacuum 仍是 INCREMENTAL、:memory: 不碰、非 SQLite 方言一条 PRAGMA 都不发、三种 opt-out 路径(config / env / config 压 env)、非法值、静默拒绝、接受但读不通→回退、被搁在 WAL 的库下次启动自救、SQLITE_BUSY 不误判、原生 addon 加载失败时不追着发第二条 PRAGMA(better-sqlite3 → wasm auto-fallback doesn't cover the persistent-file / --artifact dev path (only the zero-config :memory: branch) #2229 的单条警告不能变两条)。
  • ESLint、check:doc-authoringcheck:nul-bytes 干净。文档改了 5 处(drivers / cli 占用检查 / 备份恢复 / 环境变量表 / v17 release notes),并修正了 sqlite-occupancy.ts 里现在已经过时的「驱动跑 journal_mode = delete」注释。

顺带发现(与本 PR 无因果)

跑第二遍全量测试时 @objectstack/runtime 必红,一开始怀疑是自己搞坏的,stash 掉改动后同样复现 —— InMemoryDriverpersistence 默认 'auto',Node 下就是落盘到 .objectstack/data/memory-driver.json,而 datasource-autoconnect.test.ts 用固定 id 播种并断言精确集合,于是每台机器只能跑绿第一次。CI 因为每个 job 都是全新 clone 才一直看不到。已开 #4065,未在本 PR 扩大范围。

Closes #3941


Generated by Claude Code

claude added 2 commits July 30, 2026 07:55
`connect()` set `auto_vacuum` and never touched `journal_mode`, so every
ObjectStack SQLite database ran SQLite's built-in rollback journal. That is
the worst mode for the shape this platform has — several processes on one
file (a dev server, `os migrate`, `os meta resync`, a test run). Measured on
one file: a writer must wait for every open reader (committing needs an
exclusive lock), and an idle attached connection leaves nothing any query can
see, which is why the occupancy check had to inspect file descriptors to
notice a live server at all (#3940). Under WAL neither holds: readers and
writers no longer block each other, and `locking_mode = EXCLUSIVE` +
`BEGIN IMMEDIATE` reports busy against an idle connection, so the SQL probe
becomes authoritative instead of blind.

Neither failure mode is loud, so neither is assumed:

- `PRAGMA journal_mode = X` answers with the mode actually in force rather
  than raising on refusal (`:memory:` answers `memory`; a filesystem that
  cannot host WAL answers `delete`), so the reply is always read back.
- A filesystem can accept WAL and then fail the first read *through* it, and
  the mode is already persisted by then. So a WAL database is proven with a
  read on every connect and reverted to `delete` when that fails — tolerating
  SQLITE_BUSY, which is contention rather than an unusable WAL.

Opt out per deployment with `OS_DATABASE_SQLITE_JOURNAL_MODE=delete`, or per
datasource with `sqliteJournalMode: 'delete'` (which outranks the env var).
Both *apply* `delete`, so a database that already adopted WAL converts back;
skipping would have stranded it. `synchronous` is untouched, so durability is
unchanged, and `auto_vacuum = INCREMENTAL` keeps reclaiming under WAL
(ADR-0057). Nothing here can fail a boot.

The WASM SQLite driver declines WAL: no other process reads its live
database, so a persistent header change buys it nothing — and sql.js accepts
the pragma rather than refusing, so the refusal the base class relies on for
`:memory:` never comes. It also parks a `-wal` an unclean native exit left
behind instead of loading the image beside it: wasm SQLite cannot read that
log, and leaving it next to a rewritten image would let a later real SQLite
replay frames that no longer belong to it.

`os db clean` counts `-wal`/`-shm` when measuring what a VACUUM reclaimed, so
bytes sitting in the log no longer read as a reclaim of zero.

Closes #3941

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SiCoKVAYsXvD7cfUZ8iDQ4
@vercel

vercel Bot commented Jul 30, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
objectstack Ignored Ignored Jul 30, 2026 8:10am

Request Review

@github-actions github-actions Bot added documentation Improvements or additions to documentation tests tooling labels Jul 30, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 3 package(s): @objectstack/cli, @objectstack/driver-sql, @objectstack/driver-sqlite-wasm.

24 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:

  • content/docs/ai/skills-reference.mdx (via packages/cli)
  • content/docs/api/client-sdk.mdx (via @objectstack/cli)
  • content/docs/api/data-flow.mdx (via @objectstack/cli)
  • content/docs/api/environment-routing.mdx (via @objectstack/cli)
  • content/docs/api/error-catalog.mdx (via @objectstack/cli)
  • content/docs/automation/hook-bodies.mdx (via packages/cli)
  • content/docs/data-modeling/drivers.mdx (via @objectstack/driver-sql, @objectstack/driver-sqlite-wasm)
  • content/docs/deployment/backup-restore.mdx (via @objectstack/cli)
  • content/docs/deployment/cli.mdx (via @objectstack/cli)
  • content/docs/deployment/self-hosting.mdx (via @objectstack/cli)
  • content/docs/getting-started/glossary.mdx (via @objectstack/driver-sql, @objectstack/driver-sqlite-wasm)
  • content/docs/getting-started/your-first-project.mdx (via @objectstack/cli)
  • content/docs/kernel/runtime-services/data-service.mdx (via packages/cli)
  • content/docs/kernel/runtime-services/index.mdx (via packages/cli)
  • content/docs/permissions/authentication.mdx (via @objectstack/cli)
  • content/docs/plugins/anatomy.mdx (via @objectstack/driver-sql)
  • content/docs/plugins/packages.mdx (via @objectstack/cli, @objectstack/driver-sql, @objectstack/driver-sqlite-wasm)
  • content/docs/protocol/kernel/index.mdx (via @objectstack/driver-sql)
  • content/docs/protocol/kernel/lifecycle.mdx (via @objectstack/driver-sql)
  • content/docs/protocol/kernel/plugin-spec.mdx (via @objectstack/cli)
  • content/docs/protocol/kernel/realtime-protocol.mdx (via @objectstack/cli)
  • content/docs/protocol/objectql/query-syntax.mdx (via @objectstack/driver-sql, @objectstack/driver-sqlite-wasm)
  • content/docs/releases/implementation-status.mdx (via @objectstack/cli, @objectstack/driver-sql, @objectstack/driver-sqlite-wasm)
  • content/docs/releases/v16.mdx (via @objectstack/cli)

Advisory only. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs origin/main → pass the list as args.docs.

The restore drill said "copy the SQLite file", which under WAL (#3941) is the
exact hazard the warning above it describes: a restored database sitting beside
a stale -wal makes a later open replay frames that do not belong to it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SiCoKVAYsXvD7cfUZ8iDQ4
@os-zhuang
os-zhuang marked this pull request as ready for review July 30, 2026 08:25
@os-zhuang
os-zhuang merged commit c53aa53 into main Jul 30, 2026
17 checks passed
@os-zhuang
os-zhuang deleted the claude/sqlite-journal-mode-delete-osacn0 branch July 30, 2026 08:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/xl tests tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SQLite runs journal_mode=delete, which serializes the dev server against every CLI tool sharing the file

2 participants