feat(optimizer): coalesce peer first_value / last_value into a single struct aggregate#23681
Closed
tohuya6 wants to merge 2 commits into
Closed
feat(optimizer): coalesce peer first_value / last_value into a single struct aggregate#23681tohuya6 wants to merge 2 commits into
tohuya6 wants to merge 2 commits into
Conversation
… struct aggregate Add the CoalesceFirstLast logical optimizer rule, gated behind optimizer.enable_coalesce_first_last (default off). Peer first_value / last_value aggregates that share an ORDER BY key are rewritten into one struct-valued aggregate plus a get_field projection, so the input is scanned once instead of once per expression and one per-group state slot is kept instead of N. Skips DISTINCT / FILTER / IGNORE NULLS, expressions without an ORDER BY, and grouping-set aggregates, and no-ops when named_struct / get_field are not registered. Closes apache#23602. Part of apache#23600; the struct fast path depends on apache#23628.
…cs and tests Add the new config and rule to the generated artifacts that enumerate every setting and optimizer rule: configs.md, information_schema.slt, optimizer_rule_reference.md, and the EXPLAIN VERBOSE rule list in explain.slt.
tohuya6
force-pushed
the
coalesce-first-last-23602
branch
from
July 18, 2026 00:17
e41f217 to
d3f9311
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #23602.
Part of the epic #23600.
Rationale for this change
SELECT ..., first_value(a ORDER BY o), first_value(b ORDER BY o), ... GROUP BY pplans oneindependent
first_value/last_valueaggregate per selected column. When those peers allshare the same
ORDER BY, they pick the same winning row for every column, so the work isduplicated across N expressions even though a single pass already determines the winner for all.
Two costs on wide "one row per group" payloads:
first_valueruns its own argmax pass over the input, comparing theORDER BYkey row by row. Selecting N winner columns is N independent scans of the same rows,so cost scales linearly with the number of peer expressions.
per group — on wide payloads, N copies of the winner's columns retained for every group key.
What changes are included in this PR?
Adds a logical optimizer rule
CoalesceFirstLast(datafusion/optimizer/src/coalesce_first_last.rs)that runs bottom-up over
Aggregatenodes. Within a singleAggregate, it groups peerfirst_value/last_valueexpressions by(function, ORDER BY, null treatment)and, for anybucket with two or more members, rewrites them into one struct-valued aggregate whose argument is
a
named_structof the original values. AProjectionon top unpacks the struct back into theoriginal columns with
get_field, so the rewrite is transparent to everything above theAggregate:becomes
(function, ORDER BY, null treatment); buckets withfewer than two members are left unchanged.
DISTINCT/FILTER/IGNORE NULLSand on expressions without anORDER BY.named_struct/get_fieldare not in the function registry.optimizer.enable_coalesce_first_last, defaultfalse.configs.md,information_schema.slt,optimizer_rule_reference.md, and theEXPLAIN VERBOSElist inexplain.slt.Are these changes tested?
Yes.
coalesce_first_last.rsasserting the rewritten plan shape (peerfirst_valuecollapsed into one
named_structaggregate +get_fieldprojection; same forlast_value; anon-coalesced aggregate such as
countpreserved alongside), plus no-op cases: a singlefirst_value, mixedfirst_value+last_value,DISTINCT, flag disabled, missing registry.coalesce_first_last.slt) that runs the query with the flag off andon over multi-group data containing NULLs and asserts identical results, plus an
EXPLAINpinning the rewritten logical and physical plans.
cargo fmt,cargo clippy --all-targets --all-features -- -D warnings, the full sqllogictestsuite, and the config/rule doc-generation checks all pass.
Are there any user-facing changes?
Adds one new configuration option,
datafusion.optimizer.enable_coalesce_first_last, defaultingto
false. No SQL or API surface changes: with the flag off the plan is unchanged, and with it onthe rewrite is result-preserving, so existing queries and previously-passing tests behave identically.
Follow-ups
payload widths and group cardinalities.
Filter(row_number() = 1) → Aggregate(FIRST_VALUE(... ORDER BY)))composes with this rule: the peer
FIRST_VALUEaggregates it emits would be coalesced here.