Summary
Revisit whether the substrait.dataframe native DataFrame should evaluate eagerly rather than lazily, to enable eager error reporting (surfacing unknown-column / type-mismatch errors at the call site instead of only at to_plan()).
Raised in review of #204: #204 (comment)
Current design
The DataFrame is lazy: each verb builds up an UnboundPlan closure (Callable[[ExtensionRegistry], Plan]) and nothing is resolved — no schema inference, no function resolution — until to_plan(). So errors only surface at materialization, not where the offending call was made.
Two reasons it's lazy today:
- Inherited from the builder layer. The frame is a thin wrapper over the lazy
substrait.builders.plan closures, which defer because they take the ExtensionRegistry at resolve time. (The frame itself carries a registry from construction, so this reason doesn't strictly bind it.)
- Correlated subqueries. A subquery captures an inner frame whose
outer_references only resolve once the enclosing query's schema is pushed onto the outer_schemas stack — which the subquery builder does at resolve time, then pops. If the frame resolved eagerly at construction, the inner plan would be built before that outer schema exists, and the outer references would have nothing to bind to.
Idea
A hybrid: resolve/validate eagerly for the common path (the frame already has a registry), and stay lazy only for subquery bodies that need an enclosing-query context. That would give eager error reporting for the vast majority of pipelines without breaking correlated subqueries.
Needs a design pass — noting it here so it isn't lost.
Summary
Revisit whether the
substrait.dataframenative DataFrame should evaluate eagerly rather than lazily, to enable eager error reporting (surfacing unknown-column / type-mismatch errors at the call site instead of only atto_plan()).Raised in review of #204: #204 (comment)
Current design
The DataFrame is lazy: each verb builds up an
UnboundPlanclosure (Callable[[ExtensionRegistry], Plan]) and nothing is resolved — no schema inference, no function resolution — untilto_plan(). So errors only surface at materialization, not where the offending call was made.Two reasons it's lazy today:
substrait.builders.planclosures, which defer because they take theExtensionRegistryat resolve time. (The frame itself carries a registry from construction, so this reason doesn't strictly bind it.)outer_references only resolve once the enclosing query's schema is pushed onto theouter_schemasstack — which the subquery builder does at resolve time, then pops. If the frame resolved eagerly at construction, the inner plan would be built before that outer schema exists, and the outer references would have nothing to bind to.Idea
A hybrid: resolve/validate eagerly for the common path (the frame already has a registry), and stay lazy only for subquery bodies that need an enclosing-query context. That would give eager error reporting for the vast majority of pipelines without breaking correlated subqueries.
Needs a design pass — noting it here so it isn't lost.