Is your feature request related to a problem or challenge?
Since UDF can only return a single column as output, returning a StructArray is desired in many cases. However, accessing fields of the returned struct results in redundant calls to the UDF.
SELECT
example_struct(c)['a'],
example_struct(c)['b']
FROM (SELECT unnest(range(1,100)) as c)
Datafusion produces the following projection step for the query, resulting in calling the UDF with the same column twice.
ProjectionExec: expr=[get_field(example_struct(c@0), a) as example_struct(c)[a], get_field(example_struct(c@0), b) as example_struct(c)[b]]
Note that calling the UDF in a CTE or view does not change the execution plan.
Here is a minimal example https://github.com/liningpan/udf-get-field-optimization.
Describe the solution you'd like
@adriangb mentioned that CSE (Common Sub-expression Elimination) optimizer might be able to handle this.
Describe alternatives you've considered
No response
Additional context
No response
Is your feature request related to a problem or challenge?
Since UDF can only return a single column as output, returning a
StructArrayis desired in many cases. However, accessing fields of the returned struct results in redundant calls to the UDF.Datafusion produces the following projection step for the query, resulting in calling the UDF with the same column twice.
Note that calling the UDF in a CTE or view does not change the execution plan.
Here is a minimal example https://github.com/liningpan/udf-get-field-optimization.
Describe the solution you'd like
@adriangb mentioned that CSE (Common Sub-expression Elimination) optimizer might be able to handle this.
Describe alternatives you've considered
No response
Additional context
No response