Skip to content

[core] Split chain table batch scans by key range for read parallelism.#8831

Open
Stephen0421 wants to merge 1 commit into
apache:masterfrom
Stephen0421:chain-table-key-range-split
Open

[core] Split chain table batch scans by key range for read parallelism.#8831
Stephen0421 wants to merge 1 commit into
apache:masterfrom
Stephen0421:chain-table-key-range-split

Conversation

@Stephen0421

Copy link
Copy Markdown
Contributor

Purpose

Chain table batch reads currently produce one split per bucket, which limits read parallelism for large buckets. This PR adds key-range split generation for chain table batch scans:

  • Introduces chain-table.split.key-range-enabled (default: true); set to false to fall back to one split per bucket
  • Batch scans use IntervalPartition + BinPacking to split snapshot and delta files by key range; files with overlapping key ranges always stay in the same split so all versions of a key across branches are merged correctly
  • Streaming reads are unchanged and still use the original one-split-per-bucket behavior

Tests

  • Added ChainTableUtilsTest.testBuildChainSplitsWithKeyRangeSplitting: verifies key-disjoint sections are split into separate splits and overlapping snapshot/delta files are kept together
  • Added ChainTableUtilsTest.testBuildChainSplitsKeyRangeSplittingLargeTargetKeepsOneSplit: verifies a large targetSplitSize packs all sections into a single split
  • Added ChainTableUtilsTest.testBuildChainSplitsWithoutKeyRangeSplitting: verifies a null keyComparator preserves the original one-split-per-bucket behavior

@Stephen0421
Stephen0421 force-pushed the chain-table-key-range-split branch from 41bf23d to aad7514 Compare July 23, 2026 11:17
@JingsongLi

Copy link
Copy Markdown
Contributor

After adding chain-table.split.key-range-enabled, docs/generated/core_configuration.html was not updated.

Add chain-table.split.key-range-enabled to split snapshot and delta files
into multiple splits per bucket while keeping overlapping key ranges together.
@Stephen0421
Stephen0421 force-pushed the chain-table-key-range-split branch from aad7514 to e29dc37 Compare July 23, 2026 13:08
@Stephen0421

Copy link
Copy Markdown
Contributor Author

After adding chain-table.split.key-range-enabled, docs/generated/core_configuration.html was not updated.

OK, fix it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants