Add Mojo AOT-compiled SIMD take/filter kernels for primitive arrays

[joseph-isaacs]

Summary

Adds Mojo AOT-compiled SIMD gather kernels for primitive take and filter, with zero runtime dependency and graceful fallback when Mojo isn't installed.

CodSpeed CI Results

"Merging this PR will improve performance by 48.74%" — 11 improved, 0 regressed, 1111 untouched.

Benchmark	BASE	HEAD	Change
decode_primitives[u8] (5 variants)	53.5 µs	36.0 µs	+49%
bench_dict_mask (4 variants)	1.7 ms	1.5 ms	+10%
gather_u32_mojo[100K] vs gather_u32_avx2[100K]	N/A	699.8 vs 678.6 µs	within 3%

What's included

• kernels/take.mojo — 20 SIMD gather kernels (16 take + 4 filter), 4x unrolled, compiled with --mcpu skylake --mtune skylake for vpgatherqd
• build.rs — AOT compiles .mojo → .o → .a, detects Mojo via PATH + ~/.local/bin, passes --target-triple from Cargo's TARGET env, gracefully falls back
• mojo.rs — Rust FFI bridge with TakeImpl, dispatches by value byte-width
• slice.rs — Mojo SIMD filter for the sparse indices path (<80% selectivity)
• take_primitive_simd bench — divan 3-way comparison: scalar vs AVX2 vs Mojo
• CI — pip install --user mojo + MOJO_MCPU=skylake for codspeed shard Add CI #2

Key design decisions

• Pointers as Int: Mojo 0.26's UnsafePointer has origin/mut params incompatible with @export. Solved with type_of anchor pattern.
• Zero runtime dep: nm shows 0 undefined symbols. No Mojo runtime/GC.
• --mcpu skylake: Critical for vpgatherqd hardware gather. x86-64-v3 scalarizes the gather into 8 individual loads.
• 4x unroll: Saturates gather pipeline with independent ops.

⚠️ Known limitation

Mojo compiles for a single target CPU (no runtime dispatch). If the build machine has AVX-512 but the runtime machine only has AVX2, you'd get SIGILL. Currently mitigated by pinning MOJO_MCPU=skylake in CI. For production use, this needs runtime feature detection or multiple compiled objects — same pattern as the existing multiversion crate usage.

Test plan

• 203 take tests pass with Mojo kernel active
• 121 filter tests pass with Mojo kernel active
• Codspeed shard Add CI #2 builds and runs with Mojo installed
• CodSpeed: +49% u8 decode, +10% dict_mask, 0 regressions
• Mojo gather within 3% of hand-written AVX2 on u32

https://claude.ai/code/session_01EVcJZP4ZmfvWRRg2CsgvST

[codspeed-hq]

Merging this PR will improve performance by 81.52%

⚡ 34 improved benchmarks
✅ 1088 untouched benchmarks
⏩ 1455 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	decode_primitives[u8, (10000, 8)]	53.5 µs	36.3 µs	+47.41%
⚡	Simulation	decode_primitives[u8, (10000, 32)]	53.6 µs	36.5 µs	+46.99%
⚡	Simulation	decode_primitives[u8, (10000, 2)]	53.5 µs	36.3 µs	+47.3%
⚡	Simulation	decode_primitives[u8, (10000, 512)]	54 µs	36.6 µs	+47.27%
⚡	Simulation	decode_primitives[u8, (10000, 4)]	53.5 µs	36.7 µs	+45.75%
⚡	Simulation	bench_dict_mask[(0.01, 0.9)]	1.7 ms	1.5 ms	+10.3%
⚡	Simulation	bench_dict_mask[(0.5, 0.9)]	1.7 ms	1.5 ms	+10.3%
⚡	Simulation	bench_dict_mask[(0.1, 0.9)]	1.7 ms	1.5 ms	+10.3%
⚡	Simulation	bench_dict_mask[(0.9, 0.9)]	1.7 ms	1.6 ms	+10.11%
⚡	Simulation	varbinview_zip_fragmented_mask	7.1 ms	6.4 ms	+11.75%
⚡	Simulation	varbinview_zip_block_mask	3.7 ms	2.9 ms	+27.68%
⚡	Simulation	decompress[u32, (10000, 16)]	57.8 µs	44.8 µs	+29.15%
⚡	Simulation	decompress[u32, (100000, 4)]	1,030 µs	682.6 µs	+50.89%
⚡	Simulation	decompress[u32, (100000, 16)]	432.3 µs	303.6 µs	+42.39%
⚡	Simulation	decompress[u32, (10000, 4)]	117.1 µs	82.5 µs	+41.94%
⚡	Simulation	decompress[u64, (10000, 4)]	139.8 µs	93.9 µs	+48.92%
⚡	Simulation	decompress[u64, (1000, 4)]	28.6 µs	23.9 µs	+19.93%
⚡	Simulation	decompress[u64, (10000, 16)]	75.4 µs	64 µs	+17.86%
⚡	Simulation	decompress[u64, (100000, 16)]	609.9 µs	496.9 µs	+22.74%
⚡	Simulation	decompress[u16, (10000, 16)]	46.3 µs	33.8 µs	+36.96%
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

---

_{Comparing claude/plan-mojo-simd-kernels-IDywB (5f2a781) with develop (8d9052e)}

Footnotes

1. 1455 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[a10y]

Does Mojo handle runtime dispatch to choose the right kernel for architecture? Or does it just pick one you build the mojo kernels

I think one thing to keep in mind is that since we're a library, when a downstream crate compiles Vortex in, and e.g. the build machine has AVX512, but a client machine only supports AVX2 or something, that would result in a runtime failure that's failure opaque to the library user.

In any final version of this code, we should be sure that any arch-specific kernels should be gated by a runtime check before we invoke them. Similar to what we do for the existing AVX2 kernel.