Skip to content

[Performance] Allocate AlignedTVList primitive arrays lazily#18276

Open
jt2594838 wants to merge 2 commits into
masterfrom
allocate_primitive_array_lazily_pr
Open

[Performance] Allocate AlignedTVList primitive arrays lazily#18276
jt2594838 wants to merge 2 commits into
masterfrom
allocate_primitive_array_lazily_pr

Conversation

@jt2594838

Copy link
Copy Markdown
Contributor

Description

Follow-up to #18264. This PR contains only the lazy primitive-array allocation changes on top of the merged dynamic-bitmap implementation.

  • Keep null placeholders for newly added or unwritten aligned columns and allocate each PrimitiveArray only on its first actual write.
  • Update TsFileProcessor memory accounting for lazy primitive-array allocation.
  • Cover clone/read/WAL/null-tablet paths with unit tests.
  • Add a 256 MiB integration test that writes 40,000 rows to one aligned measurement and then one row to 1,200 new measurements, preventing the eager-allocation memory failure.
  • Add a real 1C1D integration performance comparison against a master worktree, including both sparse/lazy-column and fully non-null aligned-tablet workloads, and generate a Markdown report.

Performance

Compared master 91476b0df7 with candidate b7f846ec98:

Workload master candidate Result
Sparse/lazy columns 30.598 ms 14.294 ms latency -53.29%, throughput +114.07%
Fully non-null aligned tablets 123.767 ms 123.359 ms +0.33%

Tests

  • AlignedTVList unit tests: 14/14 passed.
  • 256 MiB low-memory integration test passed (40,000 historical rows, then one row across 1,200 new columns).
  • 1C1D worktree comparison completed for master and candidate, including the no-null workload.

@Override
public synchronized RamInfo calculateRamSize() {
return new RamInfo(
timestamps.size(), alignedTvListArrayMemCost(), rowCount, new ArrayList<>(dataTypes));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

calculateRamSize() still multiplies the number of timestamp blocks by alignedTvListArrayMemCost(), which assumes that every column has a materialized primitive array in every block. That assumption is no longer valid after this change.

Comment on lines +1145 to 1150

if (acquireArray != 0) {
// memory of extending the TVList
memIncrements[0] +=
acquireArray * alignedMemChunk.getWorkingTVList().alignedTvListArrayMemCost();
for (TSDataType dataType : dataTypesInTVList) {
memIncrements[0] += acquireArray * AlignedTVList.valueListArrayMemCost(dataType);
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This still charges the dense per-block cost for every existing column whenever a new block is acquired, while AlignedTVList.expandValues() now only adds null placeholders and allocates primitive arrays on their first actual write.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants