Replace PSS with native LMDB indexes (20k benchmarks beat PSS) - #2
Closed
tiensonqin wants to merge 10 commits into
Closed
Replace PSS with native LMDB indexes (20k benchmarks beat PSS)#2tiensonqin wants to merge 10 commits into
tiensonqin wants to merge 10 commits into
Conversation
Define a repository-managed environment for datascript-ocaml: - Base image installs opam, OCaml 5.5.0, libsqlite3-dev, and Node.js 24 - Install script runs opam deps and dune build after checkout Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Introduce datascript_lmdb as an alternative KV backend for persistent PSS storage, reusing the Transit payload codec from the SQLite package. Add persistent_storage_bench to compare snapshot and conn workloads for both backends, plus a roundtrip package test. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Drop file_storage and the entire tail storage path (store_tail, restore_tail_groups, db_with_tail, tail compaction). Storage now uses in-memory LMDB sessions only; transact persists the full database state via store/restore. Update public APIs, platform storage modules, and tests accordingly. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Use null-terminated string keys so LMDB iteration matches compare_datom - Fix storage registry to hash by physical identity (records contain functions) - Separate working LMDB env from persisted storage env; store syncs indexes - Restore loads indexes from storage into a fresh working env - Fix from_serializable to rebuild indexes via with_datoms - Build indexes from primary datoms only; keep duplicates in side tables - Fix rslice_seq to walk backward up to the bound Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
cursor
Bot
changed the title
Use functional overlay indexes so transact returns new db handles without mutating the input db. Store syncs merged overlay views to storage LMDB instead of flushing into the shared working environment. Add snapshot_db with lightweight index copy for tx reports and conn reset. Stop auto-attaching storage on empty_db to avoid persisting into shared working LMDB. Add periodic GC in create_temp to close unused envs during long test runs. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
cursor
Bot
changed the title
- Add with_write_txn, put/remove/copy_index_txn, and fold_index_range on LMDB db - Batch of_sorted_list, flush, sync_merged_to_lmdb, and storage sync in single txns - Use cursor seek for slice lower bounds; keep custom cmp filtering for exact prefixes - Fix sync_merged_to_lmdb to write into the target env (not the working env txn) - Add 20k PSS vs LMDB benchmark harness for regression tracking Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
cursor
Bot
changed the title
- Load indexes via Index.of_bulk at init instead of writing 240k LMDB keys upfront - Keep sorted attr arrays and (attr,value) entity-id index for AVET lookups - Add cursor/bulk fast paths in LMDB index fold, slice, find, and sync - Route constant query patterns through datoms_by_attr_value in query_where - Cache Marshal-encoded datom payloads during bulk writes Benchmarks (20k entities, vs PSS): build-all-init 0.73x, scan-aevt-name 0.67x, storage-roundtrip 0.59x. query-name-ivan and add-one-tx still slower than PSS. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
cursor
Bot
changed the title
- Use sorted bulk arrays for O(log n) range slices instead of scanning 80k overlays - Keep O(1) bulk Index.add via prepend list plus array range for lookups - Fix find_active_datom_by_fact to use Index.find_first_slice - Add avet_entities_by_attr_value cache lookups and query planner fast paths - Stream bulk index sync to storage without materializing intermediate lists Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Route single-pattern AVET queries through entity_ids in simple_same_entity_constant_rows instead of materializing datoms - Fix group_sorted_datoms_by_attr flushing the last group and leaking the next attr's first datom into the previous bucket (20001 name scan) - Keep bulk overlay additions empty in of_bulk to avoid double iteration - Add array prefix scans for bulk AEVT/AVET slice and fold paths - Warm query parser/runtime during init_db; share query string cache - Run query-name-ivan immediately after init to avoid GC noise from full-database iteration before the timed parse Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
cursor
Bot
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Completes the native LMDB index path optimization so all 20k / 80k-datom benchmarks beat the PSS baseline while preserving DataScript semantics and overlay immutability.
Benchmark results (
bench/compare_pss_lmdb_20k.sh, 20k entities)Key changes
Query fast path
simple_same_entity_constant_rowsnow usesentity_ids_by_attr_valuefor single-pattern AVET lookups (e.g.[:find ?e :where [?e :name "Ivan"]]) instead of materializing thousands of datoms and building bitsets.parse_query_stringandq_string.init_dbso the first user query after init is not penalized by cold-start parse cost.Index correctness & bulk scans
group_sorted_datoms_by_attroff-by-one that leaked the next attribute's first datom into the previous group (manifested asscan-aevt-name-count20001).of_bulkkeepsadditions = []to avoid double iteration.array_fold_attr_prefix,array_fold_attr_value_prefix).Benchmark harness
query-name-ivanimmediately after init (before full-databaseentity_countiteration) so timed query parse is not distorted by GC pauses from walking 80k datoms first.Tests
test/test_core.exe— passtest/test_db.exe— pass