Skip to content

Add LMDB storage backend, OCaml 5.5 Cloud Agent environment - #1

Closed
tiensonqin wants to merge 2 commits into
mainfrom
logseq/ocaml-55-environment-fe5d
Closed

Add LMDB storage backend, OCaml 5.5 Cloud Agent environment#1
tiensonqin wants to merge 2 commits into
mainfrom
logseq/ocaml-55-environment-fe5d

Conversation

@tiensonqin

Copy link
Copy Markdown
Contributor

Summary

  • Add datascript_lmdb: LMDB-backed implementation of the existing storage KV interface (address → Transit-encoded PSS node payload), mirroring the SQLite backend shape.
  • Add persistent_storage_bench: runs the same snapshot + conn workloads against SQLite and LMDB side by side.
  • Add test_lmdb_package: roundtrip + closed-session guard tests.
  • Extend Cloud Agent / CI setup with liblmdb-dev and lmdb opam dependency.

Benchmark highlights (native OCaml 5.5)

Lower is better (ms). Same row counts for both backends; LMDB files are larger due to mmap pre-allocation.

Size 100

Operation SQLite LMDB
snapshot-build-and-store 3.85 1.90
snapshot-restore 0.06 0.02
snapshot-add-one-and-store 11.23 2.78
snapshot-update-and-store 7.14 2.35
conn-add-one 0.70 0.43
conn-update-one 0.36 0.26
file size after build 106 KB 197 KB

Size 1000

Operation SQLite LMDB
snapshot-build-and-store 30.83 31.92
snapshot-update-and-store 23.83 5.22
conn-update-one 0.95 1.11
file size after build 967 KB 1.8 MB

LMDB wins clearly on incremental snapshot store/update paths; bulk build is similar at 1000 blocks. SQLite produces smaller on-disk files.

Usage

opam exec -- dune exec bench/persistent_storage_bench.exe -- --size 100
opam exec -- dune exec bench/persistent_storage_bench.exe -- --sizes 100,1000,5000
let session = Datascript_lmdb.open_session path in
let storage = Datascript_lmdb.storage session in
(* same store / restore / create_conn API as SQLite *)

Cloud Agent environment

.cursor/environment.json pins OCaml 5.5.0 via Dockerfile + install script.

Open in Web Open in Cursor 

cursoragent and others added 2 commits August 26, 2026 19:25
Define a repository-managed environment for datascript-ocaml:
- Base image installs opam, OCaml 5.5.0, libsqlite3-dev, and Node.js 24
- Install script runs opam deps and dune build after checkout

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Introduce datascript_lmdb as an alternative KV backend for persistent
PSS storage, reusing the Transit payload codec from the SQLite package.
Add persistent_storage_bench to compare snapshot and conn workloads
for both backends, plus a roundtrip package test.

Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
@tiensonqin tiensonqin closed this Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants