feat(driver-sql): run file-backed SQLite in WAL journal mode (#3941) - #4072
Merged
Conversation
`connect()` set `auto_vacuum` and never touched `journal_mode`, so every ObjectStack SQLite database ran SQLite's built-in rollback journal. That is the worst mode for the shape this platform has — several processes on one file (a dev server, `os migrate`, `os meta resync`, a test run). Measured on one file: a writer must wait for every open reader (committing needs an exclusive lock), and an idle attached connection leaves nothing any query can see, which is why the occupancy check had to inspect file descriptors to notice a live server at all (#3940). Under WAL neither holds: readers and writers no longer block each other, and `locking_mode = EXCLUSIVE` + `BEGIN IMMEDIATE` reports busy against an idle connection, so the SQL probe becomes authoritative instead of blind. Neither failure mode is loud, so neither is assumed: - `PRAGMA journal_mode = X` answers with the mode actually in force rather than raising on refusal (`:memory:` answers `memory`; a filesystem that cannot host WAL answers `delete`), so the reply is always read back. - A filesystem can accept WAL and then fail the first read *through* it, and the mode is already persisted by then. So a WAL database is proven with a read on every connect and reverted to `delete` when that fails — tolerating SQLITE_BUSY, which is contention rather than an unusable WAL. Opt out per deployment with `OS_DATABASE_SQLITE_JOURNAL_MODE=delete`, or per datasource with `sqliteJournalMode: 'delete'` (which outranks the env var). Both *apply* `delete`, so a database that already adopted WAL converts back; skipping would have stranded it. `synchronous` is untouched, so durability is unchanged, and `auto_vacuum = INCREMENTAL` keeps reclaiming under WAL (ADR-0057). Nothing here can fail a boot. The WASM SQLite driver declines WAL: no other process reads its live database, so a persistent header change buys it nothing — and sql.js accepts the pragma rather than refusing, so the refusal the base class relies on for `:memory:` never comes. It also parks a `-wal` an unclean native exit left behind instead of loading the image beside it: wasm SQLite cannot read that log, and leaving it next to a rewritten image would let a later real SQLite replay frames that no longer belong to it. `os db clean` counts `-wal`/`-shm` when measuring what a VACUUM reclaimed, so bytes sitting in the log no longer read as a reclaim of zero. Closes #3941 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SiCoKVAYsXvD7cfUZ8iDQ4
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Contributor
📓 Docs Drift CheckThis PR changes 3 package(s): 24 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:
|
The restore drill said "copy the SQLite file", which under WAL (#3941) is the exact hazard the warning above it describes: a restored database sitting beside a stale -wal makes a later open replay frames that do not belong to it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SiCoKVAYsXvD7cfUZ8iDQ4
os-zhuang
marked this pull request as ready for review
July 30, 2026 08:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
按 #3941 的 Ask 实现:
SqlDriver.connect()为文件型 SQLite 切到 WAL。issue 里点名的四条代价(网络文件系统、单向迁移、多出的 sidecar 文件、与auto_vacuum的交互)逐条处理,没有留给读者。先把两边的事实测出来
在同一个文件上用 better-sqlite3 实测(不是照抄 SQLite 文档):
SQLITE_BUSY—— 提交要拿 EXCLUSIVE,与任何读者互斥locking_mode = EXCLUSIVE+BEGIN IMMEDIATE报 busy)app.dbapp.db-wal/app.db-shm,干净关闭后自动消失第三行就是 #3924 第一版占用检查失效、#3940 只能改去查文件描述符的原因。端到端复验(真
SqlDriver持有文件、进程外探测):最后一行是关键 —— 对方是空闲的。改之前这里报 idle,迁移就那么跑过去了。文件描述符信号继续保留并且仍然排第一:它能报出进程名,WAL 的锁探测报不出。顺带确认 knex 给 sqlite 的池是
min:1, max:1,所以一个空闲的os serve确实一直挂着一条连接 —— 这条通路是成立的,不是推测。并发写在任何模式下都还是串行(SQLite 只允许一个写者),文档里写清楚了,没有夸大。另外 better-sqlite3 自带
busy_timeout = 5000,所以不需要顺手加超时。两种「不响」的失败,都不靠猜
拒绝是一个答案,不是一个异常。
PRAGMA journal_mode = X返回的是当前真正生效的模式::memory:回memory,撑不住 WAL 的文件系统回delete,都不抛。所以回复一律读回来比对,绝不假定成功。接受了,然后用不了。 WAL 需要数据库旁边的共享内存索引,网络文件系统(NFS/SMB)给不了 —— 模式切换可能成功,而第一次穿过 WAL 的读才失败,此时模式已经落进文件头。所以每次连到 WAL 库都用一次
SELECT count(*) FROM sqlite_master(任何读都会打开 shm,这是最便宜的)证明它能用,失败就回退delete。两个细节:验证不只在切换后跑,而是每次连接都跑 —— 否则被上一次启动搁在 WAL 里的库只有一次自救机会;而
SQLITE_BUSY不算用不了(占用检查自己就会短暂持 EXCLUSIVE 锁),那种情况直接放过,不会把一个健康的库拽出 WAL。逃生阀是「施加」而不是「跳过」
因为 journal mode 是文件的持久属性,跳过会把已经转过去的库永远留在 WAL 里。所以两种写法都会真的执行
PRAGMA journal_mode = delete,把库转回来:无法识别的值会报出来并继续用 wal,而不是当成静默 opt-out —— 一个拼写错误不该让库悄悄留在串行模式。全程不抛、不拦启动:留在 rollback journal 的库只是并发更差,不是坏了。
synchronous一个字没动,持久性与之前完全一致;auto_vacuum = INCREMENTAL实测在 WAL 下照常回收(ADR-0057 的路径有测试盯着)。wasm 驱动明确不进 WAL,理由改过一次
第一版我写的理由是「它靠导出字节镜像持久化,WAL 里的已提交事务会被丢掉 —— 数据丢失」。实测推翻了这条:sql.js 的
export()会先关库再重开,关库即 checkpoint,行没丢(并且它会从镜像文件头继承wal,因为 VFS 是内存的,压根不会拒绝这个 pragma)。所以理由改成真实的那个:它没有跨进程共享(活库在 WASM 堆里,别人读到的永远是导出的快照),pragma 对它零收益,却是往用户文件头里写一个持久标记,还让导出路径的正确性挂在 sql.js「关库即 checkpoint」这个实现细节上。声明式关掉,而不是等基类去发现。顺带补掉这次改动新开的一个口子:原生驱动被
kill -9后会留下<db>-wal(干净关闭不会)。wasm 只读主镜像,读不到那段日志;更糟的是它下次 flush 会重写镜像,而把一个不匹配的日志留在旁边,之后真 SQLite 打开时会把不属于这个镜像的 frame 重放上去。所以把它改名挪开(字节留着可恢复),并在警告里说清丢了什么、怎么用真sqlite3捞回来。os db clean量体积时把-wal/-shm计进去,否则躺在日志里的字节会让一次真的瘦身报成 0。验证
turbo run test --force:132/132 任务通过,0 缓存(合并 main 之前);合并当前 main 后重建并重跑 driver-sql / driver-sqlite-wasm / cli,全绿(54 + 12 + 80 文件)。dogfood 那 71 个文件 / 410 个用例是真起服务器打真 SQLite 文件,是这次最有分量的集成信号。auto_vacuum仍是 INCREMENTAL、:memory:不碰、非 SQLite 方言一条 PRAGMA 都不发、三种 opt-out 路径(config / env / config 压 env)、非法值、静默拒绝、接受但读不通→回退、被搁在 WAL 的库下次启动自救、SQLITE_BUSY不误判、原生 addon 加载失败时不追着发第二条 PRAGMA(better-sqlite3 → wasm auto-fallback doesn't cover the persistent-file / --artifact dev path (only the zero-config :memory: branch) #2229 的单条警告不能变两条)。check:doc-authoring、check:nul-bytes干净。文档改了 5 处(drivers / cli 占用检查 / 备份恢复 / 环境变量表 / v17 release notes),并修正了sqlite-occupancy.ts里现在已经过时的「驱动跑journal_mode = delete」注释。顺带发现(与本 PR 无因果)
跑第二遍全量测试时
@objectstack/runtime必红,一开始怀疑是自己搞坏的,stash 掉改动后同样复现 ——InMemoryDriver的persistence默认'auto',Node 下就是落盘到.objectstack/data/memory-driver.json,而datasource-autoconnect.test.ts用固定 id 播种并断言精确集合,于是每台机器只能跑绿第一次。CI 因为每个 job 都是全新 clone 才一直看不到。已开 #4065,未在本 PR 扩大范围。Closes #3941
Generated by Claude Code