Skip to content

perf(arrow/array): compare boolean values by valid runs - #1186

Draft
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/arrow-boolean-equality-valid-runs
Draft

perf(arrow/array): compare boolean values by valid runs#1186
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/arrow-boolean-equality-valid-runs

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What

  • compare all-valid Boolean value bitmaps in bulk
  • compare nullable values one valid run at a time
  • keep tiny runs and fragmented validity on the scalar path
  • ignore physical value bits under null slots

Boolean values are already bit-packed, but equality currently extracts the left and right value bit for every valid row.

The C++ equality path uses the same general shape: scalar comparison for tiny runs and bitmap comparison for larger runs.

https://github.com/apache/arrow/blob/485499fd02ea2b0c323d67871fbe96aae4232504/cpp/src/arrow/compare.cc#L279-L313

Benchmark

Apple M1 Pro, Go 1.26.3. Representative medians from five runs:

Case Before After Change
65K all valid, equal 401 us 219 us -45%
65K clustered 10% null, equal 395 us 225 us -43%
65K alternating null, equal 340 us 331 us -3%
1M all valid, equal 6.12 ms 3.56 ms -42%
1M clustered 10% null, equal 6.04 ms 3.66 ms -39%
1M unaligned slice, equal 6.15 ms 3.51 ms -43%

The fragmented-run probe keeps periodic and alternating null patterns near the existing scalar cost.

Tests

  • go test -race ./arrow/array -count=1
  • go vet ./arrow/array
  • go test ./...

@fallintoplace
fallintoplace marked this pull request as draft August 12, 2026 13:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant