Skip to content

perf(internal/hashing): build binary memo offsets in one pass - #1196

Draft
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/binary-memo-offsets
Draft

perf(internal/hashing): build binary memo offsets in one pass#1196
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/binary-memo-offsets

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • Build binary offsets with one linear pass over the values.
  • Keep full and subset offsets relative to the requested start.
  • Apply the same change to int32 and int64 offsets.

Benchmarks

Added benchmarks for full and subset copies with int32 and int64 offsets. The local runs use 1k and 100k unique values, plus empty and null values.

The 100k-value cases were about 15% to 25% faster locally, with 0 allocs/op.

Tests

  • go test ./internal/hashing ./parquet/internal/encoding
  • go test -race ./internal/hashing
  • Full package test run passed with the local Parquet test data checkout, excluding the CSV example that needs a separate Arrow CSV fixture.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant