Epic: consume Substrait spec test-case (.test) files — parse, map, and run conformance cases

[nielspardon]

Motivation

substrait-python authors and parses Substrait plans but does not execute them. The Substrait spec ships function conformance test cases (.test) describing, for each function invocation, the argument literals/types, options, and the expected result / return type. Both the grammar and the data already ship as dependencies:

• Parser — substrait_antlr/func_test_case/ (generated ANTLR lexer/parser/visitor/listener for the .test / FuncTestCase grammar), from substrait-antlr.
• Data — substrait_extensions/testcases/**/*.test (e.g. datetime/, aggregate_generic/, aggregate_approx/, …), from substrait-extensions.

Primary purpose — a verification framework for downstream users. substrait-python doesn't run plans, so the main value of consuming these files is to give its users — those who build their own execution/evaluation of Substrait plans on top of substrait-python — a framework to verify their implementation against the spec's canonical cases. A user plugs their evaluator into the framework and it checks each case's actual result against the expected result. substrait-python provides the parsing, the structured case model, function/type resolution, and the harness; it does not provide the compute engine.

Secondary — self-coverage for what substrait-python owns. The same cases can validate the parts substrait-python is responsible for — extension registry lookup and type derivation (type_inference / derivation_expression) — against the expected return types, turning the spec's canonical suite into real coverage (today validated only against hand-written cases) and catching conformance regressions on each spec bump.

Several .test-format spec features were closed as "pin-absorbed / not consumed" after the 0.96.0 bump (#204); this epic gives them a home:

• map & struct literals in test cases (v0.92.0, spec #1077) — Update to Substrait v0.92.0 #191
• function composition in test cases (v0.94.0, spec #1088) — Update to Substrait v0.94.0 #176
• user-defined type literals in test cases (v0.94.0, spec #1098) — Update to Substrait v0.94.0 #176
• enum-argument support in the FuncTestCase grammar (v0.87.0, spec #1010)

Scope

Add a first-class API for loading Substrait function test cases and a framework for verifying an execution implementation against them.

Tasks

• Parse .test files via the shipped substrait_antlr.func_test_case parser into a typed Python model: (function name, argument literals + types, function options, expected result / expected return type, error expectations).
• Discover / load the bundled cases under substrait_extensions/testcases/**, plus an API to load user-supplied files/directories.
• Map test-case literal & type syntax → substrait-python types and literals. This is where map/struct literals (#1077), user-defined-type literals (#1098), and enum arguments become meaningful.
• Conformance framework (primary): expose each case as structured data and let a user-supplied evaluator callback compute the actual result, so downstream execution engines can assert value conformance against the expected result. Provide pass/fail reporting and a maintained xfail mechanism.
• Self-check (secondary): with no user evaluator, resolve each case through the ExtensionRegistry and derive its return type, comparing against the expected type (not value). Wire into CI so spec-version bumps surface regressions.
• (stretch) authoring / emitting .test files from Python.

Non-goals

• Shipping a compute/execution engine in substrait-python. Value conformance is checked via a user-provided evaluator; substrait-python only provides the framework, parsing, and type/signature resolution. (Engine round-trips in substrait-python's own tests remain gated behind SUBSTRAIT_ENGINE_TESTS.)

Related

• Umbrella for the .test-only issues closed after the 0.96.0 bump (feat: ergonomic native DataFrame/Expr API (substrait.api) #204): Update to Substrait v0.94.0 #176, Update to Substrait v0.92.0 #191.
• Pairs with / depends on enum-argument calling (needed to bind the std_dev / variance cases): Update to Substrait v0.87.0 #161.