feat(parquet): wire in additional parquet writer settings#2887
Open
xanderbailey wants to merge 1 commit into
Open
feat(parquet): wire in additional parquet writer settings#2887xanderbailey wants to merge 1 commit into
xanderbailey wants to merge 1 commit into
Conversation
xanderbailey
force-pushed
the
xb/parquet_writer_settings
branch
from
July 24, 2026 12:44
abcfba9 to
ca7d5ba
Compare
xanderbailey
commented
Jul 24, 2026
| /// writer is built, not when properties are parsed. | ||
| pub const PROPERTY_PARQUET_COMPRESSION_CODEC: &str = "write.parquet.compression-codec"; | ||
| /// Default Parquet compression codec (matches Iceberg's default since 1.4.0). | ||
| pub const PROPERTY_PARQUET_COMPRESSION_CODEC_DEFAULT: &str = "zstd"; |
Contributor
Author
There was a problem hiding this comment.
Have called this out in the PR description but current default is actually uncompressed from what I can see.
This opens up a question generally as to if we should copy the Java defaults for these settings which might be more intuitive for user or keep the defaults as the arrow-rs defaults... I don't have a strong opinion here so interested to hear people's thoughts.
Contributor
There was a problem hiding this comment.
On the matter of inconsistency, there are other places where arrow-rs has different defaults to parquet-java, like the compression level for zstd too:
xanderbailey
commented
Jul 24, 2026
| assert_eq!(upper_bounds, HashMap::from([(0, Datum::int(i32::MAX))])); | ||
| } | ||
|
|
||
| // ----------------------------------------------------------------- |
Contributor
Author
There was a problem hiding this comment.
This looked very claude so I removed
xanderbailey
changed the title
mbutrovich
self-requested a review
July 24, 2026 17:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
What changes are included in this PR?
ParquetWriterBuilder::from_table_propertiespreviously only translated the content-defined-chunking settings; every other Parquet knob silently fell through to parquet-rs (arrow-rs) defaults, which diverge from the Java implementation. This PR maps the row-group / page / dictionary sizing and compression table properties into the parquet-rsWriterProperties:write.parquet.compression-codec(defaultzstd) andwrite.parquet.compression-levelwrite.parquet.row-group-size-bytes(default 128 MB)write.parquet.page-size-bytes(default 1 MB)write.parquet.page-row-limit(default 20000)write.parquet.dict-size-bytes(default 2 MB)How the new Iceberg defaults differ from the parquet-rs defaults that were previously in effect:
Are these changes tested?
Yes