Skip to content

[branch-54] Handle nulls in type coercion of higher-order UDFs, map_extract, spark array_repeat (backport #23071)#23629

Draft
gstvg wants to merge 3 commits into
apache:branch-54from
gstvg:branch-54-null-arg-type-coercion
Draft

[branch-54] Handle nulls in type coercion of higher-order UDFs, map_extract, spark array_repeat (backport #23071)#23629
gstvg wants to merge 3 commits into
apache:branch-54from
gstvg:branch-54-null-arg-type-coercion

Conversation

@gstvg

@gstvg gstvg commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Note: Draft until the mainline PR get merged

Which issue does this PR close?

Rationale for this change

Unresolved lambda variables and unspecified placeholders report a DataType::Null data type, which causes an error when used as argument of a function that doesn't handle nulls during type coercion

What changes are included in this PR?

Handle null in coerce_single_list_arg for higher-order functions, map_extract and in spark array_repeat. All other functions (scalar, window, agg and higher-order) that uses user defined type coercion have been checked to currently handle null args.

Are these changes tested?

Test passing array_filter with an unresolved lambda variable to Dataframe::with_column (which indirectly calls value_fields_with_higher_order_udf_and_lambdas coercion via Projection::try_new, projection_schema, exprlist_to_fields, Expr::to_field ), and sqllogictests

Are there any user-facing changes?

No

@github-actions github-actions Bot added core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation spark labels Jul 16, 2026
@gstvg gstvg force-pushed the branch-54-null-arg-type-coercion branch from 77c39c3 to 89a2bf5 Compare July 16, 2026 08:56
@gstvg gstvg changed the title [branch-54] Handle nulls in type coercion of higher-order UDFs, map_extract, spark array_repeat [branch-54] Handle nulls in type coercion of higher-order UDFs, map_extract, spark array_repeat (backport #23071) Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate functions Changes to functions implementation spark sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant