Tx-filter history views and Datahike-style purge API - #3
Draft
tiensonqin wants to merge 9 commits into
Draft
Conversation
Introduce tx visibility filtering and public history/time-travel API matching dbval.core: basis_tx, as_of/as_of_t, since/since_t, history, temporal_view. - db fields: max_tx (basis), store_max_tx, as_of_tx, since_tx, history - tx_visibility module with datoms_filter matching dbval semantics - transact rejects temporal views with dbval-compatible error message - design doc for overlay removal and append-only migration Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Replace overlay merge model with append-only LMDB writes and dbval-style
tx visibility on all read paths.
- Simplify Index.t to { db, which }; remove additions/removals/bulk
- append_tx_data: single LMDB txn for EAVT/AEVT/AVET on transact
- init via of_eavt_datoms (one txn); snapshot_db is O(1) shared handle
- refresh_indexes_with_tx_data appends full tx_data (add + retract)
- apply_db_view on datoms/eavt/attr caches for datoms-filter + basis
- test_tx_history: as_of, since, history integration tests
Known regression: add-one-tx ~1.4ms vs ~0.01ms overlay (LMDB write cost).
Store still copies full index when session/storage envs differ.
Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Defer LMDB writes for incremental transacts on databases without attached storage into db.pending_datoms. Bulk init still writes session LMDB directly. Store flushes pending via flush_pending_datoms before syncing indexes. Read paths merge pending_overlay with LMDB cursors without forcing full list materialization when only duplicates are absent. Fix find_eavt/find_avet and exact-prefix/seek paths to include pending datoms. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
cursor
Bot
changed the title
Add sync_append_since_tx to copy only datoms with tx > stored meta max_tx when session and storage LMDB envs differ. Skip index copy when envs are shared (storage-attached dbs). Add test_storage multi-tx incremental store with as_of/history after restore. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Implement :db/purge, :db.purge/attribute, and :db.purge/entity transaction operations that physically remove datoms from current and history views, matching Datahike purge semantics. Purge searches the history stream, deletes keys from append-only LMDB indexes, and syncs removals to persistent storage. Also includes tx-filter history fixes (codec added flag, transact read ceiling, public temporal API) and test_purge regression coverage. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
cursor
Bot
changed the title
Cover basis_tx, as_of/since bounds, history retractions, entity retraction trails, temporal view transact guards, view immutability, index parity, and public API aliases. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Replace fixed storage_kind variants with extensible string labels (storage_kind_memory/lmdb/sqlite constants for built-in names) - Unify all backends behind storage_backend callbacks registered via Datascript_storage_protocol.register_backend - Default datascript-ocaml-native.storage package: memory only - Optional opam packages: datascript-ocaml-native-lmdb and datascript-ocaml-native-sqlite with plugin modules - Migrate storage/history/purge tests to Alcotest Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Since tx is exclusive (tx > since_tx), post-bootstrap age updates appear in since tx0 history, not since tx1 where age 30 was asserted. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Delete dead sync_merged_to_lmdb and stale lmdb root copies; storage lives under lmdb/native and storage/native only. - Route fold_datoms through datoms() on temporal views so tx-filter/history apply to full-index scans. - Replace melange LMDB db with in-memory Hashtbl backend matching native API; enable byte mode for jsoo via melange lmdb/storage libraries. - Add sqlite/datascript_sqlite_db.ml and tidy codec imports. - Update design doc and bench row_count after list_addresses removal. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Completes the tx-filter / append-only LMDB index migration: overlay model removed, temporal views (
history/as_of/since) are O(1) db handle copies with tx-visibility + datoms-filter applied on read.Storage architecture
storage_kindis an extensible string ("memory","lmdb","sqlite", or custom names like"pg").Datascript_storage_protocol.register_backend.datascript-ocaml-native— memory only.datascript-ocaml-native-lmdb,datascript-ocaml-native-sqlite.Tx-filter / history
sync_merged_to_lmdb.fold_datomson temporal views routes throughdatoms()so tx-filter applies to full-index scans.history,is_history,as_of,since,basis_tx,as_of_tx,since_tx.Platform parity
open Lmdb).Tests
All tx-filter gate tests pass:
test_core,test_db,test_tx_visibility,test_tx_history,test_storage,test_purge,test_lmdb_package,test_sqlite_package.Benchmark (20k entities)
LMDB is competitive or faster on query/scan/storage at 20k; init remains slower (expected for append-only history store).
Design doc
See
docs/design-tx-filter-history.md.