Datahike benchmark parity and predicate range scan fast path - #4
Draft
tiensonqin wants to merge 22 commits into
Draft
Datahike benchmark parity and predicate range scan fast path#4tiensonqin wants to merge 22 commits into
tiensonqin wants to merge 22 commits into
Conversation
- bench/datahike_compare.ml: shared-db query suite aligned with Datahike datascript-bench (15 queries, same timing protocol). - bench/compare_ocaml_datahike.sh + datahike_shared_bench.clj for side-by-side runs. - query_where: route [?e :attr ?v] + comparison predicates through avet index_range instead of rejecting with constant_patterns=[] (fixes qpred1/2/range scan path). - test/test_datahike_queries.ml: golden result-count parity tests for key queries. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Add docs/adr/query-planner.md and docs/query_planner_plan.md for compiled planner direction without new public APIs. - Route constant+value-var same-entity queries through the relation evaluator instead of the incomplete simple_same_entity fast path (fixes q2/q2-switch). - Fall back to hash-join when same-entity fusion is empty or missing value vars. - Add simple_avet_predicate_rows fast path with tighter AVET bounds and direct row collection for predicate/range queries. - Expand test_datahike_queries.ml to all 15 benchmark queries at size=2000. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
The AVET range path used fold_stored_bounded with an attr-only upper bound, which kept scanning until the next attribute and decoded every datom twice (stop check plus callback). slice_seq also materialized the full range into a list before iteration. Add fold_stored_avet_value_range to seek at the attr+value key prefix, stop on attr change or upper value using key parsing, and route AVET value-range slices through it. Expose fold_index_range for streaming query execution and remove the small-db AEVT array workaround. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Store empty LMDB values for AVET keys and decode keys without re-parsing attr - Reuse a long-lived read transaction for index cursor scans - Keep merged-index AVET range queries on the LMDB bounded fold path - Skip redundant in_range checks and post-filter branches on exact bounds - Label compare benchmark output with storage backend (memory vs LMDB temp) - Add count_avet and query_profile probes for predicate scan diagnostics Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Use benchmark_memory_storage for Datahike compare benches: shared in-memory LMDB with MDB_NOSYNC/NOMETASYNC/WRITEMAP, materialized attr indexes via refresh_db_indexes after bulk load, and AVET array-range fast paths when avet_by_attr is populated. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Route AEVT exact lookups through cached aevt_by_attr arrays instead of LMDB slices. Optimize simple_same_entity_constant_rows with entity-set intersection, lazy per-entity value lookup for small result sets, and materialized tables only for large multi-attr scans. Benchmark harness: default repeats=2, warmup/sample=200ms, configurable JIT warmup, BENCH_SIZE for Datahike side, and FULL=1 for publication timing. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Support --query / BENCH_QUERY so one query can be timed quickly without running the full suite. Applies to the OCaml bench, shared Datahike bench, and compare_ocaml_datahike.sh (second positional arg or env). Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Remove the guard that skipped the fast path whenever value variables were present, which forced q2-q4 through the generic query engine. Use Hashtbl entity-id intersection and fill value tables via datoms/ AEVT attr scans so attrs like last-name are not missed by the slice-based primary_attr_datoms helper. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Implement dedicated query shapes for cross-entity value joins, or-join with constant name branches, and not-join with a single constant clause. These avoid the generic binding engine and scan indexes directly. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
fold_slice and find_first_slice now recognize structurally equal attr-only prefix bounds instead of requiring physical (==) equality on bound datoms. When that fast path was missed, fold_stored_bounded used full datom compare against e=0 bounds and filtered out every real entity datom. Also route datascript primary_attr_datoms through Db.primary_attr_datoms (Index.fold_attr_prefix with pending overlay and view), remove the duplicate Index.slice implementation, and restore join fast paths to use the fixed attr cache directly. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Recognize the Datahike benchmark pattern (follow ?e1 ?e2) with a single non-recursive follows rule and enumerate follows edges via primary_attr_datoms instead of the generic rule engine. Parity: q-rule still returns 667 rows @ size=2000. Benchmark @2000: ~0.011 ms OCaml vs ~0.22 ms Datahike (~0.05x). Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Add query_implementation_comparison.md comparing Datahike's compiled entity-group / relation-union executor with OCaml's list-based interpreter. Apply a general bind_var fast path: physical equality (==) before query_results_equivalent, avoiding entity resolution on repeated binds. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Remove multi-strategy same-entity heuristics. Keep OR union at relation level (Datahike sum-rel style). In relation_of_same_entity_patterns, prefer constant-filtered candidate iteration before value-attr driver scan, matching Datahike entity-group lookup order. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
When a constant-filtered query binds one value attribute (q2, q2-switch), use indexed find_datom per candidate entity instead of scanning the full attribute into a max_e array. Multi-value shapes (q-5-merge) keep the attr-scan tables path. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Use constant slice plus per-entity index lookup instead of full attr scans and (max_e+1) value arrays. Single value var: candidate entities then find_datom per attr. Multiple value vars: driver attr scan filtered by constant candidates, then find_datom for remaining attrs. Add fast Aevt entity+attr point lookup via array_find_exact_prefix on aevt_by_attr, and fold_primary_attr_datoms to iterate cached attr arrays without list copies. Relation engine prefers driver scan when constants and two or more value patterns are present. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Replace bound_datom + Seq find_datom with find_entity_in_aevt_array binary search on cached aevt_by_attr arrays. simple_same_entity_constant_rows caches attr arrays once and looks up each candidate entity per value var. Parity 16/16 unchanged. Benchmark @ 2000: q2 ~0.01ms, q-5-merge ~0.25ms (vs Datahike ~0.18ms in full suite). Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Use read-session transactions for fold_index, meta_get, and get_index to avoid MDB_BAD_RSLOT after writes. - Require an upper AVET bound before using the attr-value range fast path so seek_datoms can continue across attributes. - Rehydrate seek/rseek values from in-memory attr caches to preserve Int vs Float after LMDB key decode. - Skip duplicate AVET pending merges when avet_by_attr cache already includes pending datoms. - Share decode_index_entry and encode_index_value in datascript_lmdb_codec so LMDB indexes and SQLite storage use the same index blob rules. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
- Drop logseq_sqlite_storage and dependents that required Persistent_sorted_set. - Use the write-path schema for AVET unique conflict lookups so mid-transaction schema updates do not spuriously require :db/index on the accumulating db. - Treat same-fact / retract lookups via visible EAVT datoms so reverse-ref overwrites are not skipped after prior cardinality-one retractions. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Leave functional runtest green without requiring lein or large timed planner/perf suites. Those can still be run manually by flipping enabled_if when tuning hot paths. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Temporal query paths were reading shared current-basis attr caches, so as_of/history missed or leaked facts. Skip those caches on temporal views, accept boolean history op flags, and cover all six benchmark categories with exact result-set assertions. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Decorrelate the seed=1 people generator so Ivan∩male is non-empty, replace query goldens with count+digest identity checks, expand rules/temporal assertions, and skip redundant AVET post-filters when input-bound Int thresholds already tighten the index range (qpred2). Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
Replace per-entity AEVT binary searches with one linear scan per attribute into entity-indexed value tables (q3/q4/q-5-merge). Extend the not-join fast path to plain (not ...) and use a bitset plus direct AEVT array walk instead of Seq + Hashtbl. Co-authored-by: Tienson Qin <tiensonqin@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Match Datahike shared-API behavior and benchmark parity without new public APIs.
Result identity
test/test_datahike_parity.ml) covers queries, writes, rules, aggregates, temporal, and joins with exact result setstest/test_datahike_queries.ml) lock count + MD5 for all 15 Datahike query shapes at size=2000 / seed=1Hot-path optimizations (this iteration)
q3/q4/q-5-merge): one linear AEVT scan per attr into entity-indexed value tables instead of per-entity binary searchq-not/q-not-join): plain(not …)uses the not-join fast path; bitset exclusion + direct AEVT array walkFixes carried on this branch
as_of/historyattr-cache isolationTest plan
dune exec test/test_datahike_queries.exedune exec test/test_datahike_parity.exe./bench/compare_ocaml_datahike.sh 20000(in progress)