Release DataFusion `54.1.0` (minor/patch) Release

[alamb]

Is your feature request related to a problem or challenge?

• Part of [EPIC] DataFusion Releases #19783
• Follow on to Release DataFusion 54.0.0 (Apr 2026 / May 2026) #21080

Let's plan a 54.0.1 or 54.1.0 patch release to fix any bugs / issues found in 54.0.0.

Steps:

• Update version and changelog
• Test with DataFusion Python
• Test with DataFusion Comet
• Test with delta.rs:
• Test vortex:
• Test with iceberg-rust:
• Test with LakeSail:
• Test with parquet viewer
• Test with datafusion-materialized-views
• Voting Thread:
• Publish to crates.io

Potential Content:

Bug Fixes

• logical JoinNode proto missing null_aware field (follow-up to #19635) #22065 (backport [branch-54] fix: preserve null_aware on logical JoinNode proto round-trip (backport #22104) #22785)
• fix: Avoid panicing when stats are not available for a file group split #23277 (backport [branch-54] fix: Avoid panicing when stats are not available for a file group split (backport #23277) #23340)
• bugfix: changed return type of spark's width_bucket to i64 #22811 (backport [branch-54]: backport #22811 (bugfix: changed return type of spark's width_bucket to i64) #23087)
• fix: array_compact handle edge case with NULLs #23192 (backport [branch-54] backport #23192 array_compact handle edge case with NULLs #23196)
• fix: handle IS TRUE correctly in EliminateOuterJoin #22444 (backport [branch-54] fix: handle IS TRUE correctly in EliminateOuterJoin (backport #22444) #23580)
• fix: preserve no-filter SMJ matches across pending outer batches #23049 (backport [branch-54] fix: preserve no-filter SMJ matches across pending outer batches (backport #23049) #23574)
• fix: NestedLoopJoinExec emits spurious unmatched-left rows with multiple probe partitions #22791 (backport [branch-54] fix: NestedLoopJoinExec emits spurious unmatched-left rows with multiple probe partitions (backport #22791) #23577)
• fix: regex simplification of anchored patterns produces wrong results #22727 (backport [branch-54] fix: regex simplification of anchored patterns produces wrong results (backport #22727) #23578)
• fix: approx_distinct over-counts for utf8view #22815 (backport [branch-54] fix: approx_distinct over-counts for utf8view (backport #22815, adapted) #23576)
• fix: isolate anonymous file statistics cache #22950 (backport [branch-54] fix: isolate anonymous file statistics cache (backport #22950, adapted) #23573)
• Fix:22477 any all schema error #22915 (backport [branch-54] fix: = ANY (SELECT ...) / <> ALL (SELECT ...) schema error (backport #22915) #23575)
• fix: Correctly compute nullability in recursive CTE schemas #22552 (backport [branch-54] fix: Correctly compute nullability in recursive CTE schemas (backport #22552) #23579)
• fix: don't duplicate volatile expressions when pushing projection into file scan #23395 (backport [branch-54] fix: don't duplicate volatile expressions when pushing projection into file scan (backport #23395, adapted) #23585)
• fix(lambda): only push referenced params into the merged batch #22853 (backport TODO)
• Handle nulls in type coercion of higher-order UDFs, map_extract, spark array_repeat #23071 (backport TODO)
• Resolve lost wakeup in SpillPoolReader with multiple concurrent SpillPoolWriters #23522 (backport TODO)

Performance

• Skip loading Parquet page index when row-group statistics already prove it cannot prune #22857 (backport [branch-54] backport #22857 (Skip loading Parquet page index when row-group statistics already prove it cannot prune) #23088)
• perf: avoid intermediate slice allocation in Spark slice function #23481 (backport [branch-54] perf: avoid intermediate slice allocation in Spark slice function (backport #23481) #23582)

Other

• [branch-54] chore: fix cargo audit #23607 (fix cargo audit directly on branch-54, no backport needed)

Minor Features:

• TBD

[mithuncy]

The backport for #22065 is up as #22785 (clean cherry-pick of #22104, targeting branch-54). Could this be added to the 54.1.0 Bug Fixes list? Thanks!

[alamb]

Note that @mbutrovich also saw some performance concerns when testing 54.0.0 with Comet:

• https://lists.apache.org/thread/1tc29mpx2j4yxto72p0vww41k7gdcs3z

[alamb]

The backport for #22065 is up as #22785 (clean cherry-pick of #22104, targeting branch-54). Could this be added to the 54.1.0 Bug Fixes list? Thanks!

Thanks @mithuncy -- added

[gstvg]

There are two lambda-related fixes which I believe qualify to backporting after they got merged into main

#22853 by @Adam-Alani
#23071

[mbutrovich]

I will open a backport for this today since it originated from Comet trying to upgrade to DF 54.0:

#22811

[mbutrovich]

I will also try to backport #22857 since Comet also encountered this while trying to upgrade to DF 54.0.

[mbutrovich]

Backport PRs:

#23088
#23087

[comphead]

I would like to backport #23192, the PR fixes correctness issue, no breaking changes

#23196 for backport