Skip to content

perf(arrow/array): pack validity values in batches - #1185

Merged
zeroshade merged 1 commit into
apache:mainfrom
fallintoplace:perf/arrow-batch-validity-packing
Aug 14, 2026
Merged

perf(arrow/array): pack validity values in batches#1185
zeroshade merged 1 commit into
apache:mainfrom
fallintoplace:perf/arrow-batch-validity-packing

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Rationale for this change

Appending an explicit []bool validity slice currently updates the bitmap and null count one value at a time. This is a noticeable part of AppendValues for narrow types.

What changes are included in this PR?

  • handle the unaligned prefix and trailing values individually
  • pack aligned validity values eight at a time
  • count nulls once per packed byte
  • add coverage for all starting bit offsets, different lengths, and stale neighboring bits
  • add helper and builder benchmarks

Benchmarks append 65,536 values on an Apple M1 Pro. Values are medians from three runs.

Benchmark Before After Change
Int8 / all valid 68.9 us 19.6 us 3.51x
Int8 / 50% null 89.9 us 20.4 us 4.41x
Int64 / all valid 87.9 us 31.7 us 2.77x
Int64 / 50% null 103.0 us 33.6 us 3.06x
Boolean / all valid 212.0 us 162.0 us 1.31x
Boolean / 50% null 225.0 us 163.3 us 1.38x
String / all valid 538.1 us 471.4 us 1.14x
String / 50% null 553.9 us 476.0 us 1.16x

No allocations are added.

Are these changes tested?

Yes.

  • go test ./arrow/...
  • go test -race ./arrow/array
  • go vet ./arrow/array

Are there any user-facing changes?

No. This only changes how builder validity bitmaps are populated.

@fallintoplace
fallintoplace marked this pull request as draft August 12, 2026 12:30
@fallintoplace
fallintoplace force-pushed the perf/arrow-batch-validity-packing branch from c92dd6c to 6fcea67 Compare August 13, 2026 20:55
@fallintoplace
fallintoplace marked this pull request as ready for review August 13, 2026 20:56

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The batched packing logic checks out — prefix/tail bounds are right, the read-modify-write tail preserves neighboring bits, and null counting via popcount is correct. The exhaustive packValidityByte test and the poisoned-bitmap offset/length matrix give good confidence. Nice speedup.

Note: this will likely need a rebase against #1173/#1180 depending on merge order.


This review was drafted with the help of an AI-assisted tool and may contain mistakes; it was reviewed and confirmed by an Apache Arrow maintainer before posting. See the contributing guide for what the project considers a maintainer review.

@zeroshade
zeroshade merged commit 17ae330 into apache:main Aug 14, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants