perf(arrow/array): bulk append fixed-width empty values - #1180
Merged
zeroshade merged 1 commit intoAug 14, 2026
Conversation
zeroshade
approved these changes
Aug 14, 2026
zeroshade
left a comment
zeroshade
left a comment
Member
There was a problem hiding this comment.
The bulk path ordering is correct (Reserve → zero range → unsafeSetValid), and both drive-by fixes — Advance zeroing per its documented contract and the BooleanBuilder.Resize truncation length — are genuine correctness improvements with test coverage. Benchmarks look convincing.
May need a rebase now that #1185 touched builder.go.
This review was drafted with the help of an AI-assisted tool and may contain mistakes; it was reviewed and confirmed by an Apache Arrow maintainer before posting. See the contributing guide for what the project considers a maintainer review.
zeroshade
pushed a commit
that referenced
this pull request
Aug 14, 2026
…#1183) ## What does this PR do? - reserves fixed-size list parent slots once in `AppendNulls` and `AppendEmptyValues` - updates the parent validity bitmap in bulk - appends all child null or empty values with one bulk builder call The child append count is `listSize * n`, so the parent and child lengths stay in sync. ## Why? Both methods currently loop over every parent value. Each parent append also loops over every child value. This removes the parent loop and lets child builders use their bulk append paths. The change is independent, but it also composes with #1173 and #1180. ## Benchmarks Apple M1 Pro, 65,536 `FixedSizeList<Int32>` rows, `GOMAXPROCS=1`. Standalone on `main`: | list width | nulls before | nulls after | change | empty before | empty after | change | | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | 4 | 1.10 ms | 0.86 ms | -22% | 1.55 ms | 1.07 ms | -31% | | 16 | 3.76 ms | 2.74 ms | -27% | 5.43 ms | 3.90 ms | -28% | | 64 | 13.03 ms | 11.07 ms | -15% | 19.35 ms | 14.86 ms | -23% | With #1173 and #1180 applied underneath: | list width | nulls before | nulls after | change | empty before | empty after | change | | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | 4 | 1.56 ms | 0.18 ms | -88% | 2.01 ms | 0.20 ms | -90% | | 16 | 4.69 ms | 0.33 ms | -93% | 6.89 ms | 0.42 ms | -94% | | 64 | 16.27 ms | 0.74 ms | -95% | 22.18 ms | 0.92 ms | -96% | ```text GOMAXPROCS=1 go test ./arrow/array -run '^$' -bench '^BenchmarkFixedSizeListBuilderBulkAppend$/(nulls|empty)/rows=65536/width=(4|16|64)$' -benchmem -benchtime=300ms -count=5 ``` ## Tests - added coverage for unaligned parent validity bits - verifies parent validity and null counts - verifies child length, null slots, and zero-valued empty slots - ran `go test ./arrow/...` - ran `go test -race ./arrow/array` - ran `go vet ./arrow/array`
fallintoplace
force-pushed
the
perf/arrow-fixed-width-empty-values
branch
from
August 14, 2026 16:54
ef7dbb0 to
e3418c8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
AppendEmptyValuesused to callAppendEmptyValuein a loop. For larger batches, each value repeated the reserve, bitmap, and storage work.This change makes fixed-width empty appends bulk operations:
The numeric generated source and template are updated together. The covered builders are numeric, temporal, interval, float16, decimal, fixed-size binary, and boolean.
Correctness
bufferBuilder.Advancenow zeroes skipped bytes as documented.BooleanBuilder.Resizenow preserves the requested logical length when truncating.Benchmark
AppendEmptyValues(1024)on an Apple M1 Pro. Three runs were used for each result.Small counts keep the single-value path, so the biggest improvement is for larger batches.
Tests
go test ./arrow/arraygo vet ./arrow/array