Skip to content

[core] Apply LIMIT during merge-file split read when safe.#8116

Open
wwj6591812 wants to merge 1 commit into
apache:masterfrom
wwj6591812:add_merge_limit_0604
Open

[core] Apply LIMIT during merge-file split read when safe.#8116
wwj6591812 wants to merge 1 commit into
apache:masterfrom
wwj6591812:add_merge_limit_0604

Conversation

@wwj6591812
Copy link
Copy Markdown
Contributor

Background

For primary-key tables with multiple overlapping level-0 files, batch queries often take the merge read path (MergeFileSplitRead) instead of raw reads. Today, LIMIT is mainly effective at the manifest/scan layer (no overlapping files) or Flink operator layer (per subtask). The first split can still run a full LSM merge and spill even when the query only needs a few rows.

Why this PR

We want to stop merge I/O and CPU as early as the record reader, right after merge output is produced, for the common SELECT … LIMIT n case on multi-L0 buckets.

What changes

  • Add LimitRecordReader in paimon-common and use it from MergeFileSplitRead (merge + no-merge readers).
  • Reuse the same helper in FormatTableRead (small refactor, no behavior change).
  • Safety gate before applying limit (aligned with KeyValueFileStoreScan#limitPushdownEnabled, plus non-PK filters): skip merge-read limit when filters cannot be fully applied on overlapping L0 sections, DV is enabled, merge-engine is partial-update/aggregation, or forceKeepDelete is on.
  • Logging: one INFO log per reader when merge read limit is applied or disabled, with the reason.

Stage optimized: per-split read phase (merge tree reader), after merge but before returning rows — not manifest planning.

Tests

  • MergeFileSplitReadTest#testWithLimit — multi-L0 bucket, direct read stops at limit.
  • MergeFileSplitReadTest#testWithLimitDisabledByNonPrimaryKeyFilter — non-PK filter disables merge read limit.
  • MergeFileSplitReadLimitITCase — Flink batch SQL on multi-L0 PK table (LIMIT + WHERE correctness).

Limitations

  • Limit is per split reader, not global across splits (global LIMIT still relies on engine-level limiters).
  • Does not avoid merge work inside a section before the first row is emitted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant