Add UnionValue and UnionScalar

[connortsui20]

Tracking issue: #8769

Depends on #8804 for the existing nested Scalar equality/hash contract fix.

Implements the scalar portion of #8769 on top of #8798.

• adds UnionValue and UnionScalar
• stores the full selected child Scalar, preserving its dtype and inner null state
• uses the Union dtype nullability for the independent outer null state
• preserves a selected null child instead of collapsing it into a null Union scalar
• treats selected null children as equal across variants while retaining variant identity for non-null values
• adds construction, downcasts, display, ordering, validation, recursive defaults, and constant scalar access
• updates scalar protobuf serialization for the full child scalar

[codspeed-hq]

Merging this PR will improve performance by 10.79%

⚡ 2 improved benchmarks
✅ 1668 untouched benchmarks
⏩ 52 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	compact[(4096, 90)]	867.8 ns	780.3 ns	+11.21%
⚡	Simulation	compact[(16384, 90)]	931.1 ns	843.6 ns	+10.37%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing agent/union-scalar-nullability (d5458d2) with develop (17f843a)²}

Footnotes

1. 52 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. No successful run was found on develop (407720c) during the generation of this report, so 17f843a was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[github-actions]

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	d5458d2		1	Explore Profiling Data

Previous Runs (2)

Status	Commit	Job	Attempt	Link
🟢 Done	3a22a55		1	Explore Profiling Data
🟢 Done	af088ad		1	Explore Profiling Data

---

Powered by Polar Signals Cloud

[github-actions]

Benchmarks: Vortex queries

Verdict: No clear signal (low confidence)
Attributed Vortex impact: +0.5%
Engines: DataFusion No clear signal (+0.2%, low confidence) · DuckDB No clear signal (+0.7%, low confidence)
Vortex (geomean): 1.005x ➖
Parquet (geomean): 1.003x ➖
Shifts: Parquet (control) +0.3% · Median polish +0.4%

How to read Verdict and Engines

• Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
• Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
• Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

---

datafusion / vortex-file-compressed (0.998x ➖, 0↑ 0↓)

name	PR d5458d2 (ns)	base 407720c (ns)	ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed	9808409	9824979	1.00
vortex_q01/datafusion:vortex-file-compressed	6348898	6363061	1.00

datafusion / parquet (0.996x ➖, 0↑ 0↓)

name	PR d5458d2 (ns)	base 407720c (ns)	ratio (PR/base)
vortex_q00/datafusion:parquet	20268628	20817537	0.97
vortex_q01/datafusion:parquet	4941477	4851302	1.02

duckdb / vortex-file-compressed (1.017x ➖, 0↑ 0↓)

name	PR d5458d2 (ns)	base 407720c (ns)	ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed	10228718	10100030	1.01
vortex_q01/duckdb:vortex-file-compressed	6439160	6304024	1.02

duckdb / parquet (1.010x ➖, 0↑ 0↓)

name	PR d5458d2 (ns)	base 407720c (ns)	ratio (PR/base)
vortex_q00/duckdb:parquet	23459510	23412652	1.00
vortex_q01/duckdb:parquet	9562752	9392853	1.02

---

No file size changes detected.