Skip to content

perf(memory): explore compact typed storage and lazy cell stringification for large results (#605) - #627

Merged
ZhuchkaTriplesix merged 1 commit into
devfrom
issue/605-compact-typed-storage
Aug 26, 2026
Merged

perf(memory): explore compact typed storage and lazy cell stringification for large results (#605)#627
ZhuchkaTriplesix merged 1 commit into
devfrom
issue/605-compact-typed-storage

Conversation

@ZhuchkaTriplesix

Copy link
Copy Markdown
Member

Summary

Introduces columnar compact typed dataset storage (CompactResultDataset) with lazy on-demand cell stringification for large query result sets.

Changes

  1. CompactResultDataset & Column Storage (lib/core/database/compact_result_dataset.dart):
    • Int64CompactColumn: Backed by Int64List and bitwise null-masking for 64-bit integer values (8 bytes per value vs 32+ bytes for full String object in heap).
    • Float64CompactColumn: Backed by Float64List and bitwise null-masking for 64-bit floating point numbers.
    • StringCompactColumn: Backed by deduplicated String references using StringInternPool.
    • GenericCompactColumn: Fallback column for composite objects and dynamic records.
  2. Lazy Row List Interface:
    • asLazyRowList(): Exposes a transparent List<List<String>> view formatting cells on-the-fly when indexed by viewport-based virtual grids, preventing upfront allocation of millions of String objects.
  3. Tests & Benchmarks:
    • Unit tests in test/core/database/compact_result_dataset_test.dart verifying typing, null bitmasks, lazy access, and exceptions.
    • 100,000-row x 5-column benchmark (500,000 cells) demonstrating <300ms packing and sub-millisecond visible viewport slicing.

Closes #605

@github-actions github-actions Bot added the performance Theme parser epic label: performance label Aug 26, 2026
@ZhuchkaTriplesix
ZhuchkaTriplesix merged commit 780f69b into dev Aug 26, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance Theme parser epic label: performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant