Add `UnionArray` compute functions by connortsui20 · Pull Request #8884 · vortex-data/vortex

connortsui20 · 2026-07-21T17:30:50Z

Rationale for this change

Tracking issue: #7882

This PR adds the straightforward compute support for canonical sparse Union arrays while keeping operations with unresolved nullability or placeholder semantics in focused follow-ups.

What changes are included in this PR?

Adds canonical filter execution and slice and mask reductions for UnionArray, following the same structural execution patterns as neighboring canonical dtypes.
Adds validity-mask execution while preserving row alignment across type IDs and sparse children.
Recursively compresses the type IDs and every sparse child.
Computes uncompressed size as the checked sum of the type IDs and sparse children.
Adds focused coverage for structural operations, outer null masking, and size accounting.

Deferred follow-ups

Take, including nullable-index outer-null propagation, and dictionary execution that depends on it.
Union casts and outer-nullability conformance coverage.
Constant Union canonicalization and inactive-child placeholder construction.
Chunked Union canonicalization, including a representation for empty chunked unions.

The source TODOs record the required semantics at each deferred dispatch point.

github-actions · 2026-07-21T17:33:30Z

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	`ec92eef`		1	Explore Profiling Data

Previous Runs (7)

Status	Commit	Attempt	Link
🟢 Done	`ab33ec0`	1	Explore Profiling Data
🟢 Done	`f2427ce`	1	Explore Profiling Data
🟢 Done	`e32db9e`	1	Explore Profiling Data
🟢 Done	`98f8f06`	1	Explore Profiling Data
🟢 Done	`1e867dd`	1	Explore Profiling Data
🟢 Done	`a6b7e6c`	1	Explore Profiling Data
🟢 Done	`6fe53ba`	1	Explore Profiling Data

Powered by Polar Signals Cloud

github-actions · 2026-07-21T17:34:24Z

Benchmarks: Vortex queries 📖

Verdict: No clear signal (low confidence)
Attributed Vortex impact: +1.2%
Engines: DataFusion No clear signal (+4.0%, low confidence) · DuckDB No clear signal (-1.5%, low confidence)
Vortex (geomean): 1.014x ➖
Parquet (geomean): 1.008x ➖
Shifts: Parquet (control) +0.8% · Median polish +0.8%

How to read Verdict and Engines

Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

datafusion / vortex-file-compressed (1.045x ➖, 0↑ 0↓)

name	PR `ec92eef` (ns)	base `57962f4` (ns)	ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed	8414693	8197567	1.03
vortex_q01/datafusion:vortex-file-compressed	5201551	4885078	1.06

datafusion / parquet (1.006x ➖, 0↑ 0↓)

name	PR `ec92eef` (ns)	base `57962f4` (ns)	ratio (PR/base)
vortex_q00/datafusion:parquet	20201445	20638806	0.98
vortex_q01/datafusion:parquet	4731033	4580085	1.03

duckdb / vortex-file-compressed (0.996x ➖, 0↑ 0↓)

name	PR `ec92eef` (ns)	base `57962f4` (ns)	ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed	9982079	10061599	0.99
vortex_q01/duckdb:vortex-file-compressed	6163313	6167184	1.00

duckdb / parquet (1.010x ➖, 0↑ 0↓)

name	PR `ec92eef` (ns)	base `57962f4` (ns)	ratio (PR/base)
vortex_q00/duckdb:parquet	23447372	23195009	1.01
vortex_q01/duckdb:parquet	9434025	9340266	1.01

No file size changes detected.

codspeed-hq · 2026-07-21T18:04:35Z

Merging this PR will not alter performance

✅ 1849 untouched benchmarks
⏩ 46 skipped benchmarks¹

_{Comparing ct/union-array-compute (c9d2f58) with develop (3048533)}

46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

codecov · 2026-07-21T18:36:20Z

Codecov Report

❌ Patch coverage is 74.09910% with 115 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.13%. Comparing base (cd92358) to head (98f8f06).

Files with missing lines	Patch %	Lines
...rtex-array/src/arrays/constant/vtable/canonical.rs	49.42%	44 Missing ⚠️
vortex-array/src/scalar_fn/fns/cast/mod.rs	0.00%	17 Missing ⚠️
vortex-array/src/arrays/union/tests.rs	93.59%	13 Missing ⚠️
...gregate_fn/fns/uncompressed_size_in_bytes/union.rs	0.00%	11 Missing ⚠️
vortex-array/src/arrays/masked/execute.rs	0.00%	10 Missing ⚠️
vortex-compressor/src/compressor/cascade.rs	0.00%	8 Missing ⚠️
vortex-array/src/arrays/filter/execute/mod.rs	0.00%	6 Missing ⚠️
vortex-array/src/arrays/dict/execute.rs	0.00%	3 Missing ⚠️
...aggregate_fn/fns/uncompressed_size_in_bytes/mod.rs	0.00%	1 Missing ⚠️
...ortex-array/src/arrays/chunked/vtable/canonical.rs	97.14%	1 Missing ⚠️
... and 1 more

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

## Rationale for this change Tracking issue: #8769 This is the array layer of the Union stack. The compute integration has been split into the stacked follow-up #8884. ## What changes are included in this PR? - Adds the canonical sparse `UnionArray` encoding with nullable u8 type IDs and row-aligned children. - Adds construction and invariant validation, typed accessors, scalar access, top-level validity, serialization, and session registration. - Registers UnionArray as the canonical representation for Union dtypes. - Leaves filter, take, slice, mask, cast, aggregate, constant/chunked canonicalization, and compression support to #8884. I think that this is a mostly straightforward implementation. The one implementation thing I will shout out is that we do not validate that the entire type IDs array has valid type IDs. If we come across an invalid type ID, we just panic, and that is similar behavior to `DictArray` validation. ## What APIs are changed? Are there any user-facing changes? Adds the public Union, UnionArray, UnionArrayExt, and UnionDataParts APIs. --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

connortsui20 changed the title ~~ct/union array compute~~ Add UnionArray compute functions Jul 21, 2026

connortsui20 mentioned this pull request Jul 21, 2026

Add UnionArray #8834

Merged

connortsui20 force-pushed the ct/union-array-compute branch from 6fe53ba to a6b7e6c Compare July 21, 2026 17:48

connortsui20 force-pushed the ct/union-array branch 2 times, most recently from 96923e7 to 3233233 Compare July 21, 2026 17:59

connortsui20 force-pushed the ct/union-array-compute branch from a6b7e6c to 1e867dd Compare July 21, 2026 17:59

connortsui20 force-pushed the ct/union-array-compute branch from 1e867dd to 98f8f06 Compare July 21, 2026 18:28

connortsui20 force-pushed the ct/union-array branch from 3233233 to cd92358 Compare July 21, 2026 18:28

connortsui20 force-pushed the ct/union-array branch from cd92358 to 9c9debf Compare July 21, 2026 18:45

connortsui20 force-pushed the ct/union-array-compute branch 2 times, most recently from e32db9e to f2427ce Compare July 21, 2026 20:31

connortsui20 force-pushed the ct/union-array branch 2 times, most recently from dd86118 to 772b738 Compare July 21, 2026 21:08

connortsui20 force-pushed the ct/union-array-compute branch from f2427ce to ab33ec0 Compare July 21, 2026 21:11

connortsui20 force-pushed the ct/union-array branch from 772b738 to f51f2dd Compare July 22, 2026 18:07

connortsui20 force-pushed the ct/union-array-compute branch from ab33ec0 to ec92eef Compare July 22, 2026 18:07

Base automatically changed from ct/union-array to develop July 23, 2026 01:34

connortsui20 force-pushed the ct/union-array-compute branch 4 times, most recently from 4ee0231 to 4616292 Compare July 23, 2026 16:22

Add UnionArray compute functions

c9d2f58

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

connortsui20 force-pushed the ct/union-array-compute branch from 4616292 to c9d2f58 Compare July 23, 2026 16:23

connortsui20 marked this pull request as ready for review July 23, 2026 16:26

connortsui20 added the changelog/feature A new feature label Jul 23, 2026

connortsui20 requested review from joseph-isaacs and robert3005 and removed request for robert3005 July 23, 2026 16:26

connortsui20 changed the title ~~Add UnionArray compute functions~~ Add UnionArray compute functions Jul 23, 2026

connortsui20 mentioned this pull request Jul 23, 2026

Tracking Issue: Add DType::Union #7882

Open

8 tasks

robert3005 approved these changes Jul 23, 2026

View reviewed changes

connortsui20 merged commit 640d802 into develop Jul 23, 2026
82 of 83 checks passed

connortsui20 deleted the ct/union-array-compute branch July 23, 2026 22:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add `UnionArray` compute functions#8884

Add `UnionArray` compute functions#8884
connortsui20 merged 1 commit into
developfrom
ct/union-array-compute

connortsui20 commented Jul 21, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 21, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 21, 2026 •

edited

Loading

Uh oh!

codspeed-hq Bot commented Jul 21, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jul 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

connortsui20 commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Deferred follow-ups

Uh oh!

github-actions Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Polar Signals Profiling Results

Latest Run

Uh oh!

github-actions Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks: Vortex queries 📖

Uh oh!

codspeed-hq Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

codecov Bot commented Jul 21, 2026

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

connortsui20 commented Jul 21, 2026 •

edited

Loading

github-actions Bot commented Jul 21, 2026 •

edited

Loading

github-actions Bot commented Jul 21, 2026 •

edited

Loading

codspeed-hq Bot commented Jul 21, 2026 •

edited

Loading