feat(writer): Support ObjectStorageLocationGenerator#2871
Conversation
hsiang-c
changed the title
| field | ||
| .transform | ||
| .to_human_string(&field_types[i].field_type, value) | ||
| urlencoding::encode(field.name.as_str()).into_owned(), |
There was a problem hiding this comment.
My understanding is this won't affect the location generator logic as of now? If so, then let's remove this for now and fix it in #2875
There was a problem hiding this comment.
@CTTY Correct, I just found the issue in the unit test. I'll remove it for now. Sorry for my late reply.
|
@CTTY If you could take a look when you have time, thank you! |
CTTY
left a comment
CTTY
left a comment
There was a problem hiding this comment.
Just did a quick review, LGTM overall! Left some minor comments
| } | ||
|
|
||
| /// Strip a single trailing slash from a location, if present. | ||
| fn strip_trailing_slash(location: &str) -> &str { |
There was a problem hiding this comment.
in table_properties.rs, there is already one strip_trailing_slash, can we refactor and use that? We can create a new iceberg/src/util/location.rs to store such helpers
| const WRITE_DATA_LOCATION: &str = "write.data.path"; | ||
| const WRITE_FOLDER_STORAGE_LOCATION: &str = "write.folder-storage.path"; | ||
| const DEFAULT_DATA_DIR: &str = "/data"; | ||
|
|
||
| /// Deprecated object storage path property, kept as a fallback for compatibility. | ||
| const WRITE_OBJECT_STORAGE_LOCATION: &str = "write.object-storage.path"; | ||
| /// Property controlling whether partition values are included in object storage paths. | ||
| const WRITE_OBJECT_STORAGE_PARTITIONED_PATHS: &str = "write.object-storage.partitioned-paths"; | ||
| const WRITE_OBJECT_STORAGE_PARTITIONED_PATHS_DEFAULT: bool = true; | ||
|
|
||
| /// Number of trailing hash bits used to build the entropy directories. | ||
| const HASH_BINARY_STRING_BITS: usize = 20; | ||
| /// Length of each entropy directory. | ||
| const ENTROPY_DIR_LENGTH: usize = 4; | ||
| /// Number of entropy directories generated from the hash. | ||
| const ENTROPY_DIR_DEPTH: usize = 3; |
There was a problem hiding this comment.
We should move all of these to table_properties.rs
| let hash_code = murmur3::murmur3_32(&mut bytes, 0).unwrap(); | ||
|
|
||
| // Force the top bit so the binary string always has 32 characters (Rust, like Java's | ||
| // Integer.toBinaryString, drops leading zeros otherwise), then keep the trailing bits. |
There was a problem hiding this comment.
I think {:032b} already forces the binary to be 32 characters with zero paddings? In java we have to do | 0x8000_0000 because the hash to binary string conversion won't preserve the leading zeroes, but here looks like we don't actually need to set the first bit to 1?
The logic is ok, but I think it's worth clarifying in the comment that we are setting the first bit to 1 simply to imitate java's padding logic
2 participants
Which issue does this PR close?
What changes are included in this PR?
ObjectStorageLocationGenerator, another implementation toLocationGeneratorthat generates a deterministic hash for each stored file and appends the hash directly after thewrite.data.path. The equivalent class in Iceberg Java is ObjectStoreLocationProviderObjectStorageLocationGeneratorconfigurable yet. Plan to do it in the follow-up PR.Are these changes tested?
Unit tests