Skip to content

perf(parquet/pqarrow): decode booleans directly to bitmaps - #1182

Draft
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-boolean-bitmap-read
Draft

perf(parquet/pqarrow): decode booleans directly to bitmaps#1182
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-boolean-bitmap-read

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What does this PR do?

  • decodes Parquet BOOLEAN values directly into a packed bitmap
  • supports both PLAIN and RLE boolean pages
  • expands nullable values inside the bitmap without creating a []bool
  • passes the record reader value bitmap directly to the Arrow Boolean array

Why?

The read path currently stores decoded booleans as one byte per value. It then allocates an Arrow bitmap and packs the values again in transferBool.

This finishes wiring the direct bitmap decoder API from #707 into the pqarrow read path. The record reader now keeps boolean values packed from decoding through Arrow array construction.

The change also fixes decoder state after a partial PLAIN bitmap decode, so the next call resumes at the correct source bit.

Benchmarks

Apple M1 Pro, 1M values, uncompressed Parquet, GOMAXPROCS=1:

case before after change before B/op after B/op change
dense 1.48 ms 0.15 ms -90% 1.61 MB 0.56 MB -65%
50% null 6.72 ms 4.63 ms -31% 4.11 MB 3.08 MB -25%
GOMAXPROCS=1 go test ./parquet/pqarrow -run '^$' -bench '^BenchmarkBooleanBitmapRead$' -benchmem -benchtime=500ms -count=5

Tests

  • added direct bitmap tests for consecutive PLAIN decode calls
  • added nullable bitmap tests for PLAIN and RLE with unaligned offsets
  • ran go test ./parquet/...
  • ran race tests for the touched encoding, file, and pqarrow packages
  • ran go vet for the touched packages

@fallintoplace
fallintoplace marked this pull request as draft August 12, 2026 12:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant