Skip to content

perf(parquet): optimize numeric dictionary insertion - #1178

Merged
zeroshade merged 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-typed-dictionary-insertion
Aug 17, 2026
Merged

perf(parquet): optimize numeric dictionary insertion#1178
zeroshade merged 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-typed-dictionary-insertion

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

What changed

  • use typed memo insertion for int32, int64, float32, and float64 dictionary encoders
  • reserve the index buffer for the whole input batch and fill it directly
  • keep the existing byte-array and fixed-length byte-array paths unchanged
  • add correctness coverage for all four numeric types, including multiple Put calls
  • add a numeric dictionary encoding benchmark

Why

The numeric path was calling the generic dictionary insertion method for every value. That added interface and reflection overhead, and the index buffer could grow inside the hot loop.

The new path does the memo lookup with the concrete numeric type and sizes the index buffer once per batch. Dictionary order, duplicate handling, and encoded indices stay the same.

Benchmark

With 65,535 values and 100 unique values on an Apple M1 Pro:

  • int32: about 1.21 ms/op to 0.89 ms/op, 26 to 13 allocs/op
  • int64: about 1.21 ms/op to 0.92 ms/op, 26 to 13 allocs/op
  • float32: about 2.1 ms/op to 1.1 ms/op, 64,905 to 13 allocs/op
  • float64: about 2.2 ms/op to 1.2 to 1.4 ms/op, 64,906 to 13 allocs/op

Validation

  • go test ./parquet/internal/encoding -count=1
  • focused Parquet Arrow dictionary tests
  • go vet ./parquet/internal/encoding
  • git diff --check

@fallintoplace
fallintoplace marked this pull request as ready for review August 14, 2026 21:24
@zeroshade
zeroshade merged commit 382298a into apache:main Aug 17, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants