Skip to content

perf(parquet/pqarrow): build nullable fixed-size list children directly - #1195

Draft
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-pqarrow-fixed-size-list-direct
Draft

perf(parquet/pqarrow): build nullable fixed-size list children directly#1195
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-pqarrow-fixed-size-list-direct

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • keep the existing zero-null fixed-size-list path unchanged
  • validate nullable parent spans once
  • use the existing slice concatenation for a small number of validity runs
  • build one child array with a typed Take for highly fragmented validity
  • avoid one temporary Arrow array per validity run

Why

Parquet does not write child values for null fixed-size-list parents. Arrow still needs list_size child slots for every parent, so the reader has to add null child values.

The old nullable path creates one temporary array for each validity run and concatenates them. Alternating validity can therefore create one temporary array per parent row.

Benchmark

Apple M1 Pro, 65,536 FixedSizeList<int32, 4> parents:

validity old new old allocs new allocs
10% nulls ~4.4 ms ~1.3 ms 85,220 46
alternating nulls ~23–30 ms ~1.1 ms 458,770 46
one clustered null run ~0.3 ms ~0.3 ms 28 27

The low-run path is kept for the clustered case, where creating a single pair of slices is cheaper than building an index array.

Tests

  • go vet ./parquet/pqarrow
  • go test -race ./parquet/pqarrow -run 'TestBuildFixedSizeListArray|TestParquetArrowIO/TestFixedSizeList'
  • PARQUET_TEST_DATA=parquet-testing/data ARROW_TEST_DATA=arrow-testing/data go test ./...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant