[core] Apply LIMIT during merge-file split read when safe.#8116
Open
wwj6591812 wants to merge 1 commit into
Open
[core] Apply LIMIT during merge-file split read when safe.#8116wwj6591812 wants to merge 1 commit into
wwj6591812 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
For primary-key tables with multiple overlapping level-0 files, batch queries often take the merge read path (
MergeFileSplitRead) instead of raw reads. Today,LIMITis mainly effective at the manifest/scan layer (no overlapping files) or Flink operator layer (per subtask). The first split can still run a full LSM merge and spill even when the query only needs a few rows.Why this PR
We want to stop merge I/O and CPU as early as the record reader, right after merge output is produced, for the common
SELECT … LIMIT ncase on multi-L0 buckets.What changes
LimitRecordReaderinpaimon-commonand use it fromMergeFileSplitRead(merge + no-merge readers).FormatTableRead(small refactor, no behavior change).KeyValueFileStoreScan#limitPushdownEnabled, plus non-PK filters): skip merge-read limit when filters cannot be fully applied on overlapping L0 sections, DV is enabled, merge-engine is partial-update/aggregation, orforceKeepDeleteis on.Stage optimized: per-split read phase (merge tree reader), after merge but before returning rows — not manifest planning.
Tests
MergeFileSplitReadTest#testWithLimit— multi-L0 bucket, direct read stops at limit.MergeFileSplitReadTest#testWithLimitDisabledByNonPrimaryKeyFilter— non-PK filter disables merge read limit.MergeFileSplitReadLimitITCase— Flink batch SQL on multi-L0 PK table (LIMIT+WHEREcorrectness).Limitations
LIMITstill relies on engine-level limiters).