refactor(encoding): make block codecs fallible - #8040
Open
Xuanwo wants to merge 7 commits into
Open
Conversation
This was referenced Jul 28, 2026
Xuanwo
added a commit
that referenced
this pull request
Jul 30, 2026
Stack 1 of the generic block compression series. Mini-block codecs need page framing information to compare container-level choices without creating a parallel planner API. This change passes one explicit context through the existing mini-block codec tree and distinguishes ordinary mini-block pages from SparseLayout callers. It intentionally changes no codec selection or persisted bytes: ordinary mini-block and SparseLayout writers keep their existing behavior, while later stack layers can consume the context when evaluating generic offset containers. Validation covered the complete lance-encoding test suite and workspace clippy with warnings denied. <!-- generic-block-stack-navigation --> ## Stack navigation - Umbrella / integration reference: #8002 - Next: #8040
Base automatically changed from
xuanwo/generic-block-stack-1-miniblock-context
to
main
July 30, 2026 10:02
# Conflicts: # rust/lance-encoding/src/compression.rs # rust/lance-encoding/src/encodings/logical/primitive.rs # rust/lance-encoding/src/encodings/logical/primitive/miniblock.rs # rust/lance-encoding/src/encodings/logical/primitive/sparse/writer.rs # rust/lance-encoding/src/encodings/physical/binary.rs # rust/lance-encoding/src/encodings/physical/bitpacking.rs # rust/lance-encoding/src/encodings/physical/byte_stream_split.rs # rust/lance-encoding/src/encodings/physical/fsst.rs # rust/lance-encoding/src/encodings/physical/general.rs # rust/lance-encoding/src/encodings/physical/packed.rs # rust/lance-encoding/src/encodings/physical/rle.rs # rust/lance-encoding/src/encodings/physical/value.rs
Xuanwo
marked this pull request as ready for review
July 31, 2026 06:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 2 of the generic block compression series. Builds on merged #8038.
Generic block descriptors now enter one bounded, fallible decoder factory. Concrete codecs own their child codecs and framing, while
Option<LanceBuffer>distinguishes metadata-only encodings from payload-bearing empty buffers. Sparse structural metadata reuses the shared validator without changing its wire format.This layer intentionally adds no protobuf variants and does not change production block or mini-block selection. Lance 2.0–2.2 bytes and the existing variable-offset layout remain unchanged. After merging the version-free strategy refactor from
main, the stable wire fixtures exposed and now cover the all-zero inline-bitpacking boundary so 0-bit blocks remain byte-for-byte stable.Validation at
de91c0a07:cargo fmt --all -- --checkcargo check -p lance-encoding --no-default-features --testscargo test -p lance-encoding(542 passed, 5 ignored; integration and doctests passed)cargo test -p lance-file(130 passed; 4 doctests passed, including exact 2.0–2.2 wire fixtures)cargo clippy --all --tests --benches -- -D warningsStack navigation