feat(vortex-geo): null-propagating scalar functions

[HarukiMoriarty]

What

Make the binary geo scalar functions (ST_Distance, ST_Intersects, ST_Contains) null-propagating. A nullable geometry operand is now allowed, and any row whose geometry input is null yields a NULL result.

Why

Spiral needs the geo predicates to accept nullable inputs to run SpatialBench.

How

A geo kernel decodes each operand into a geo_types geometry, and a null row has no geometry to decode, so it can't compute over every row and mask the nulls afterwards the way the numeric kernels do.

Instead, the shared execute_null_propagating dispatch drops the null rows, computes only the rows valid in both operands, and scatters the results back under the combined null mask. return_dtype is nullable iff an operand is, and validate_geometry_operands no longer rejects nullable operands.

[github-actions]

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	8fe6c0e		1	Explore Profiling Data

Previous Runs (8)

Status	Commit	Job	Attempt	Link
🟢 Done	eea991e		1	Explore Profiling Data
🟢 Done	cb2af19		1	Explore Profiling Data
🟢 Done	6ff78a8		1	Explore Profiling Data
🟢 Done	610a8c3		1	Explore Profiling Data
🟢 Done	9ad9376		1	Explore Profiling Data
🟢 Done	0716c2c		1	Explore Profiling Data
🟢 Done	9a50270		1	Explore Profiling Data
🟢 Done	90e4548		1	Explore Profiling Data

---

Powered by Polar Signals Cloud

[codspeed-hq]

Merging this PR will improve performance by 11.17%

⚡ 1 improved benchmark
✅ 1669 untouched benchmarks
⏩ 53 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	true_count_vortex_buffer[128]	580.6 ns	522.2 ns	+11.17%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing nemo/geo-null-propagation (8fe6c0e) with develop (e6eeaae)}

Footnotes

1. 53 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[github-actions]

Benchmarks: Vortex queries 📖

Verdict: Likely regression (medium confidence)
Attributed Vortex impact: +14.1%
Engines: DataFusion No clear signal (+5.8%, low confidence) · DuckDB Likely regression (+23.0%, high confidence)
Vortex (geomean): 1.081x ➖
Parquet (geomean): 1.012x ➖
Shifts: Parquet (control) +1.2% · Median polish +4.1%

How to read Verdict and Engines

• Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
• Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
• Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

---

datafusion / vortex-file-compressed (1.057x ➖, 0↑ 0↓)

name	PR 8fe6c0e (ns)	base e6eeaae (ns)	ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed	11949428	11446321	1.04
vortex_q01/datafusion:vortex-file-compressed	7969894	7449028	1.07

datafusion / parquet (0.999x ➖, 0↑ 0↓)

name	PR 8fe6c0e (ns)	base e6eeaae (ns)	ratio (PR/base)
vortex_q00/datafusion:parquet	23548429	23584739	1.00
vortex_q01/datafusion:parquet	5612178	5616200	1.00

duckdb / vortex-file-compressed (1.261x ❌, 0↑ 2↓)

name	PR 8fe6c0e (ns)	base e6eeaae (ns)	ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed 🚨	12986663	10387695	1.25
vortex_q01/duckdb:vortex-file-compressed 🚨	7490376	5885688	1.27

duckdb / parquet (1.025x ➖, 0↑ 0↓)

name	PR 8fe6c0e (ns)	base e6eeaae (ns)	ratio (PR/base)
vortex_q00/duckdb:parquet	25024200	24576114	1.02
vortex_q01/duckdb:parquet	10042084	9723548	1.03

---

No file size changes detected.

[connortsui20]

@claude review this PR, and review it also with the context of vortex-tensor, which has similar types and functions (extension types and null handling). Make sure things are consistent in the vortex-geo crate as well (so null semantics are handled correctly and tested well).

[github-actions]

Claude finished @connortsui20's task in 3m 58s —— View job

---

Claude PR Review — feat(vortex-geo): null-propagating scalar functions

Reviewed against origin/develop and cross-checked against vortex-tensor's binary scalar functions (InnerProduct, CosineSimilarity, L2Norm), which are the closest analog (extension types + null propagation).

• Read PR diff (vortex-geo)
• Study vortex-tensor null handling & extension types
• Assess null semantics correctness
• Assess test coverage
• Post review

Verdict: Solid, well-documented change. The shared execute_null_propagating dispatch is a clean abstraction and the "drop nulls → decode → scatter back" strategy is the right call for kernels that can't decode a null row. return_dtype nullability now matches vortex-tensor exactly (any operand nullable). My main feedback is a consistency gap with vortex-tensor around the vtable's null-related optimization hooks, plus a couple of test-coverage and minor cleanup notes. No correctness bugs found.

---

1. Missing vtable overrides that vortex-tensor provides (main consistency gap)

vortex-tensor's binary functions override three null-related vtable methods. After this PR the geo functions match on return_dtype but still inherit the conservative trait defaults for the other two.

vortex-tensor/src/scalar_fns/inner_product.rs (161–179):

fn validity(&self, _options, expression) -> VortexResult<Option<Expression>> {
    // The result is null if either input tensor is null.
    let lhs_validity = expression.child(0).validity()?;
    let rhs_validity = expression.child(1).validity()?;
    Ok(Some(and(lhs_validity, rhs_validity)))
}
fn is_null_sensitive(&self, _options) -> bool { false }

The geo functions define neither, so they fall back to vtable.rs defaults:

• validity() → Ok(None) — the planner can't derive output validity symbolically and must execute the whole kernel to get the mask.
• is_null_sensitive() → true (conservative) — but the geo kernels are not null-sensitive. They propagate nulls and never inspect a null value, so e(mask(a, m)) == mask(e(a), m) holds — exactly the property is_null_sensitive documents. This defaults to disabling mask-pushdown optimizations for geo predicates.

Since the whole point of this PR is null propagation, and vortex-tensor establishes the pattern for the identical shape of function, I'd add both to each of GeoDistance / GeoContains / GeoIntersects:

fn validity(&self, _: &Self::Options, expression: &Expression) -> VortexResult<Option<Expression>> {
    // Null iff either geometry operand is null.
    Ok(Some(and(expression.child(0).validity()?, expression.child(1).validity()?)))
}
fn is_null_sensitive(&self, _: &Self::Options) -> bool { false }

These are optimization/consistency improvements, not correctness fixes — behavior is already correct without them. Fix this →

Note: vortex-tensor also sets is_fallible = false. For geo I'd leave the true default — geometry decoding (geometries) genuinely can fail on malformed storage, unlike the tensor math kernels. Worth a conscious decision rather than copying blindly.

2. Test coverage: two-column path is only exercised by distance

eval_column_pair (the (None, None) two-nullable-columns branch) is only covered by distance_propagates_column_pair_nulls. contains and intersects only test a nullable column against a constant. Because ST_Contains is asymmetric (a contains b), a both-columns-nullable test there is the most valuable one to add — it's the case where a null on either side, in either role, must null the row. Fix this →

Also uncovered: an all-null column (the valid.all_false() early-return in eval_column / eval_column_pair). Only the all-null constant path is tested. A cheap addition worth having since it's a distinct code path.

3. Minor: dead AllOr::None arms in build_array

Both GeoOutput::build_array impls handle AllOr::None (all rows null), but every caller (eval_column, eval_column_pair, and the constant paths) already short-circuits valid.all_false() to all_null_array before calling build_array. So those arms are effectively unreachable defensive code. Fine to keep as a safety net, but a short comment noting "callers short-circuit the all-null case" would prevent a future reader from thinking it's a live path. Additionally BitBuffer::from_iter(vec![false; len]) allocates a throwaway Vec<bool>; if you keep the arm, prefer a direct all-false BitBuffer constructor.

4. Consistency nits vs. vortex-tensor

• Operand-equality check. vortex-tensor validates binary operands with eq_ignore_nullability so the two inputs must be the same underlying type. validate_geometry_operands only checks each operand is some native geometry, allowing e.g. Point vs Polygon — which is legitimate here (distance/contains/intersects across geometry types is meaningful), so this is a justified divergence, not a bug. Just flagging that the asymmetry with tensor is intentional.
• nullable_point_column placeholder coords. Nice touch storing 0.0 placeholders for null rows and relying on the filter-before-decode to never touch them — this genuinely tests that null rows are never decoded. It does assume the ExtensionArray's validity mirrors its storage StructArray validity; that assumption holds today but is worth a one-line comment since the whole null-propagation contract rests on column.validity() reflecting the null rows.

---

Correctness review (no issues found)

Walked the branches in execute_null_propagating:

• Null constant on either side → all_null_array with a Nullable dtype — consistent with return_dtype. ✅
• nullability derived once from operand dtypes; nulls can only originate from nullable dtypes, so a NonNullable output can never need a null slot. ✅
• AllOr::Some(rows) scatter zips valid row indices with filtered values in order — indices and values stay aligned because filter(valid) preserves order. ✅
• Uses bulk validity()?.execute_mask(len, ctx)? and filter, not per-element is_valid(i) in a loop — aligned with the repo's hot-loop guidance. ✅
• Length mismatch is guarded in the two-column path; constant/column paths inherit the column length. ✅

I ran no build/test commands (read-only review). CI (codspeed/benchmarks) is green on the PR.
· branch nemo/geo-null-propagation

[connortsui20]

@claude review again, again check for consistency within vortex-geo and between vortex-tensor, and generally look for null semantics bugs

[github-actions]

Claude finished @connortsui20's task in 3m 16s —— View job

---

Claude PR Review — round 2 (consistency + null semantics)

Re-reviewed the updated diff against origin/develop, cross-checked vortex-tensor's binary kernels (InnerProduct etc.), and traced the null paths end-to-end.

• Read updated mod.rs, distance.rs, contains.rs, intersects.rs, extension/mod.rs, test_harness.rs
• Cross-check vortex-tensor (validity, is_null_sensitive, is_fallible, dtype nullability)
• Audit null semantics (dispatch arms, scatter/gather alignment, all-null paths, ExtensionArray::validity)
• Post findings

Verdict: The round-1 feedback landed cleanly — validity() (via the shared binary_result_validity) and is_null_sensitive = false are now on all three functions, and they match vortex-tensor exactly. I found no correctness or null-semantics bugs. The remaining items are a test-coverage inconsistency across the three functions and a couple of confirmations.

---

Null-semantics audit (no bugs found)

Traced every path in execute_null_propagating:

• Operand order preserved in all dispatch arms. For the asymmetric ST_Contains, (Some(qa), None) → eval_column(b, |g| compute(&ga, g)) = a_const.contains(b_row), and (None, Some(qb)) → eval_column(a, |g| compute(g, &gb)) = a_row.contains(b_const). Container/contained roles stay correct. ✅
• Scatter alignment. column.filter(valid) preserves row order and valid.indices() is ascending, so rows.iter().zip(values) in build_array maps each computed value to the right row. ✅
• nullability vs return_dtype. Both derive from any operand nullable; a null scalar always carries a nullable dtype, so all_null_array (always nullable) can never contradict a NonNullable output. A NonNullable output only occurs when both operands are non-nullable, in which case the combined mask is all-true and Validity::from_mask(.., NonNullable) is never handed a partial mask. ✅
• validity() is exactly right here (not just conservative): distance/contains/intersects always return a non-null value for two valid geometries, so output-null ⟺ input-null, which is precisely and(child0.validity, child1.validity). (This would be wrong for a kernel that can produce null from valid inputs, e.g. division — but that's not the case here.) ✅
• The contract's foundation holds: ExtensionArray's ValidityChild delegates to storage_array() (vortex-array/src/arrays/extension/vtable/validity.rs:11), so column.validity()? reflects the storage StructArray's validity that nullable_point_column sets. The placeholder 0.0 coords in null rows are never decoded because the filter drops them first. ✅

1. Test-coverage inconsistency — intersects never exercises the two-column null path (main item)

The three functions have diverged in coverage of the very paths this PR adds. contains is now the model; the other two have gaps:

null path	distance	contains	intersects
col-vs-const null rows	✅	✅	✅
constant-null → all null	✅	✅	✅
two-col pair nulls (eval_column_pair)	✅	✅	❌
all-null column (valid.all_false() in eval_column)	❌	✅	❌
two-col all-null (valid.all_false() in eval_column_pair)	❌	✅	❌

ST_Intersects never runs eval_column_pair with nulls at all — the entire two-nullable-column path is untested for it. Since intersection is symmetric it's cheap to add, and it's the highest-value gap. Adding the all-false early-return tests to distance/intersects (mirroring contains_all_null_column_is_all_null / contains_column_pair_all_null) would bring all three to parity. Fix this →

2. is_fallible divergence from vortex-tensor is correct — leave as-is

vortex-tensor sets is_fallible = false; the geo functions keep the true default. This is the right call and should not be copied: geometry decoding (geometries → WKB/geoarrow decode) can genuinely fail on malformed storage, whereas the tensor math kernels cannot. Flagging only to confirm the asymmetry is intentional, not an oversight.

3. Minor — binary_result_validity and the dead AllOr::None arms

• The shared binary_result_validity helper is a nice consolidation, and @connortsui20's inline note (mod.rs:142) about promoting it to a general array-level helper is a reasonable follow-up — many vtable validity() impls compute exactly and(child validities). Out of scope for this PR, but worth a tracking issue.
• The unreachable! on the AllOr::None arms in both build_array impls is now clearly documented as guarded by the all_false() short-circuits — good. The earlier BitBuffer::from_iter(vec![false; len]) concern is gone since that arm no longer allocates. ✅

---

Consistency with vortex-tensor — summary

aspect	tensor	geo (this PR)	consistent?
return_dtype nullability	any operand nullable	any operand nullable	✅
validity()	and(child0, child1)	and(child0, child1) via helper	✅
is_null_sensitive	false	false	✅
is_fallible	false	true (default)	✅ justified (decode can fail)
operand type check	eq_ignore_nullability (same type)	each is some geometry	✅ justified (cross-type geo ops are valid)

I ran no build/test commands (read-only review); the coverage table was derived by reading the test modules. Nice, well-documented change overall.
· branch nemo/geo-null-propagation

[connortsui20]

Must fix

1. A null geometry literal now breaks zone pruning. Before this PR a nullable operand was rejected at type-check, so the prune rules never saw a null literal. Now ST_Intersects(geom, NULL) executes fine (all-null result, tested), but query_aabb (prune/mod.rs:84-89) calls single_geometry on the constant with no is_null check, hits "geo: null geometry is not supported", and the error propagates out of the stats rewrite driver (vortex-array/src/stats/rewrite.rs:148), failing the whole rewrite. Minimal fix: decline on scalar.is_null() (return Ok(None)); optimal: rewrite to lit(true) since an all-NULL predicate matches nothing — cf. how the null radius literal is already declined in prune/distance.rs:68-73. Needs a prune test either way.

2. Zero-length inputs with non-nullable operands produce the wrong dtype. Mask::AllTrue(0) has true_count() == 0, so all_false() is vacuously true, and eval_column / eval_column_pair check all_false() before all_true() (scalar_fn/mod.rs:220-223, 252-254). A len-0 non-nullable execution therefore returns the Nullable all_null_array while return_dtype declared NonNullable — tripping the debug-build result-dtype assertion (vortex-array/src/scalar_fn/typed.rs:149-158), silently mismatched in release. Untested; worth a test.

Suggested simplification (also fixes bug 2)

GeoOutput can drop from 3 methods to 2: make build_array handle AllOr::None by returning an all-invalid array instead of unreachable! (scalar_fn/mod.rs:87, 122). Then null_dtype(), all_null_array, and the two all_false() early-returns all become unnecessary — the const-null paths can route through T::build_array(len, &Mask::new_false(len), vec![], Nullable). That removes the panic and its cross-function invariant, and the len-0 case falls out correct for free (AllTrue(0) → indices() is All → correctly-typed empty array).

Also minor: the if q.scalar().is_null() { return all_null } block is repeated 3× in execute_null_propagating (mod.rs:170-191) — one loop over both operands before the match does it once.

Worth a note in this PR

Nullable columns now silently degrade AABB pruning: GeometryAabb::accumulate reads the unmasked coordinate buffers (aggregate_fn/aabb.rs:207-216), so null-row placeholder coords (e.g. (0,0)) get unioned into the zone box. Safe direction — only ever under-prunes — but a placeholder at the origin drags a distant chunk's box across the plane. Deserves a comment at accumulate, and there's no test covering a nullable column through the aggregate or pruning end-to-end.

[HarukiMoriarty]

1. Null literal prune, query_aabb declines on a null constant; prune tests added for both distance & intersects.
2. len-0 dtype, fixed by the simplification below, plus a regression test.
3. Simplification, GeoOutput down to 2 methods; build_array handles the empty mask, const-null routes through it, and the is_null checks are one loop.
4. AABB nulls went past the note: accumulate now skips null rows (matching min/max) + a test.

[connortsui20]

I remember (maybe incorrectly) that it used to be the case you didnt need this if statement, .filter should do this check for you. @joseph-isaacs is this still true? tracing through the code its hard for me to tell

[HarukiMoriarty]

Confirmed. .filter collapses an all-true mask: Filter's reduce rule (filter/rules.rs:43) returns the child, and ArrayRef::filter runs .optimize(). So the if all_true guard was redundant; removed it here.

[connortsui20]

so do any of the functions below need a precondition that it probably shouldnt take nulls? I also think that converting nulls to (0, 0) might be a footgun in general

[HarukiMoriarty]

Yes, both functions have an implicit "no nulls" precondition, now documented. I think we should never converts nulls to (0, 0).

[connortsui20]

Basically it would be good to add some comments here explaining that because nulls have been removed we are fine to use all the values.

But that brings up a separate question: did you look at the perf of computing this merge on very sparsely null data? for example, if there is only a single null value, it would be way faster to not execute the filter above and instead just merge if the value is not null.

[HarukiMoriarty]

Commented. I think it is fine, it's uniform across all geometry types, and the hot/common path (non-nullable) already costs nothing. The sparse-null compaction is an acceptable tradeoff for a write-time stat; if profiling ever flags nullable-geometry stat writes, revisit with a coordinate-validity-aware min/max.

[connortsui20]

I think this looks good, but I would like if someone else has eyes on this as well. @joseph-isaacs?