Skip to content

perf(sqlite): reuse canonical proof encoding in retained reads - #4931

Draft
LIHUA919 wants to merge 2 commits into
loopx-project:mainfrom
LIHUA919:codex/sqlite-d2-read-hash
Draft

LIHUA919 wants to merge 2 commits into
loopx-project:mainfrom
LIHUA919:codex/sqlite-d2-read-hash

Conversation

@LIHUA919

Copy link
Copy Markdown
Contributor

Refs #4224. Design and the unmodified failed qualification evidence were recorded first in #4224 (comment).

Goal and delivered outcome

The formal 1 MiB, 10k/100k SQLite D2 run on measurement source e99a83b63c149685625ecc80e5065c31b2ad36d6 failed the existing read budgets: at 100k commits, readReceipt p95 was 269.03 ms (limit 50 ms) and scanCommitted(100) p95 was 801.81 ms (limit 250 ms). The 64 KiB, 100k/300k headroom run had no failed budget rows. Both reports remain historical evidence; this PR does not relabel either result.

SQLite v2 verifies every retained row from the covering checkpoint. With a stable 1 MiB projection, each row repeatedly canonicalized the same long JSON string for the exact state and commit SHA-256 proofs. A real SQLite 128-commit probe reproduced about 302 ms for a window-end receipt and 503 ms for a 100-row scan; raw indexed SQL was under 1 ms. CPU samples concentrated in canonical byte generation.

This PR retains the exact v0 proof bytes while reusing bounded encodings of repeated long strings within one verification or archive-audit call. A later empty delta reuses its verified predecessor state digest, still compares the row's declared digest and parent lineage, and still recomputes the full per-row commit proof. The first commit always derives its own state digest. No cache crosses a read call.

Scope and compatibility

  • Changed surfaces: SQLite retained receipt/scan/history verification and the internal canonical digest helper; focused TypeScript regressions.
  • Persisted SQLite schema, checkpoint interval, stored state/commit digests, original receipts, writer behavior, CLI/API response shapes, provider selection and RFC thresholds are unchanged.
  • Rollback: revert these read-path commits. SQLite v2 databases written before or after remain readable; no migration, re-encoding or history deletion is involved.
  • Entry points: the existing AuthorityStore read/scan methods and real Python CLI are affected. No Dashboard, Lark or packaged frontend contract changes.

Verification

  • Node 22.22.3: canonical digest parity across JSON key/order, sparse array, Unicode and cache eviction cases; a fixed 1 MiB payload's 100 distinct proofs reuse the encoding while matching the old digest.
  • Real SQLite: large unchanged projections retain independent scan rows; forged intermediate state-digest/parent lineage fails closed; an empty root remains readable.
  • SQLite TS suite: 292/296 passed in the sandbox; the four Python-consumer tests could not start their local Effect runtime there and passed 4/4 in an isolated real runtime. Migration/bounded-profile/capacity tests 19/19; real Python-to-TypeScript SQLite CLI tests 5/5; TypeScript typecheck and git diff --check passed.
  • Risk-based loopx canary premerge --from-git-diff --git-diff-base origin/main passed: 3 direct checks, 5 catalog checks, 8 risk-profile checks, zero failures and zero manual holds. Its temporary local dependency link was removed before push; the tracked diff contains only the five paths above.
  • Matched local 1 MiB / 128-commit probe: fixed-projection window-end receipt about 302→43 ms and scan 100 about 503→98 ms. A changing-projection receipt remains about 63 ms, above 50 ms. This probe is diagnostic, not formal D2 qualification.

A full reference-runtime matched-1m 10k/100k rerun on this exact head is still required. Retain the failed row if that rerun misses either threshold. Domain workload, stable RSS, large-history recovery, 24-hour consumer lag, restore/upgrade/rollback, OS/runtime coverage, the natural-time soak and provider promotion remain separate D2 holds. No default-provider change or live-Goal migration is requested.

Future-facing pass

The bounded encoding cache stays with the canonical codec owner and is consumed only by SQLite verification. It removes repeated proof work without introducing a second semantic owner or a speculative persisted proof format. Further proof-format work should follow measured real-workload gaps and a separate compatibility review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant