feat(delete-vector): parse deletion-vector-v1 puffin blob - read support by Shekharrajak · Pull Request #2414 · apache/iceberg-rust

Shekharrajak · 2026-05-07T19:09:54Z

Which issue does this PR close?

What changes are included in this PR?

Implements DeleteVector::deserialize_blob to parse Iceberg V3 deletion-vector-v1 Puffin blobs into a RoaringTreemap of deleted row positions.

• Iceberg V3 spec — Deletion Vectors: https://iceberg.apache.org/spec/#deletion-vectors
• Iceberg V3 spec — Puffin file format: https://iceberg.apache.org/puffin-spec/
• Java reference implementation: org.apache.iceberg.deletes.PositionDeleteIndex and BaseDVFileWriter

Are these changes tested?

Five binary fixtures generated by the Java reference writer and validation using unit test.

Shekharrajak · 2026-05-09T19:10:43Z

    }
+
+    #[allow(dead_code)]
+    pub fn deserialize_blob(blob_bytes: &[u8], expected_cardinality: Option<u64>) -> Result<Self> {


Read path deserialize_blob parses a binary blob (length + magic + roaring bitmap + CRC32) into a DeleteVector.

Shekharrajak · 2026-05-09T19:19:06Z

+
+    #[allow(dead_code)]
+    pub fn deserialize_blob(blob_bytes: &[u8], expected_cardinality: Option<u64>) -> Result<Self> {
+        if blob_bytes.len() < DV_MIN_BLOB_SIZE {


Minimum size check

Shekharrajak · 2026-05-09T19:19:45Z

+        }
+
+        // 2GB cap matches Java's signed-i32 length field.
+        if blob_bytes.len() > i32::MAX as usize {


2GB cap - Java's Integer.MAX_VALUE is 2,147,483,647 — exactly i32::MAX - similar to java implementation

mbutrovich · 2026-05-13T16:53:02Z

Hi @Shekharrajak could you please update the PR description to make review easier?

Shekharrajak · 2026-05-13T17:01:41Z

 bimap = { workspace = true }
 bytes = { workspace = true }
 chrono = { workspace = true }
+crc32fast = { workspace = true }


crc32fast — fast CRC32 over the blob payload

Shekharrajak · 2026-05-14T04:22:13Z

Hi @mbutrovich , Added short PR decription. It is pretty much same logic as Iceberg java implementation. I can add more unit tests going forward when we start write support .

Shekharrajak · 2026-06-10T12:49:23Z

@c-thiel can you please review ?

mbutrovich

Thanks @Shekharrajak. I read this as the decode half of deletion-vector read support (Task 1 of #2792, under the v3 epic #2411).

The approach is sound. Leaning on roaring's RoaringTreemap::serialize_into / deserialize_from is the right call. In roaring 0.11.x that path is the official portable 64-bit format (8-byte little-endian bitmap count, ascending 4-byte little-endian key, standard 32-bit bitmap), so it matches Iceberg-Java and iceberg-go without hand-rolling the framing. The length, magic, and CRC handling line up with BitmapPositionDeleteIndex, hoisting the sizes and magic into named constants is good, and the error-path tests are thorough.

The most important change is the golden fixtures. They are generated by the crate itself, so they do not validate against Java, and they never exercise run-optimized bitmaps, which is the case most likely to break a cross-engine read. Detail is inline on the regenerate test.

The rest, all inline: verify the CRC before parsing the payload, add a doc comment on deserialize_blob that pins down the byte layout and the expected_cardinality cross-check, drop the dead_code allows once the reader wiring lands, and settle the method name now so it does not get renamed across #2678 and #2681 when they land together.

mbutrovich · 2026-07-09T21:20:57Z

+    /// Regenerate the committed golden fixtures (see testdata README).
+    #[test]
+    #[ignore]
+    fn dv_blob_regenerate_golden_fixtures() {


This regenerates the fixtures with serialize_blob_for_test, which encodes through roaring's serialize_into. So the committed fixtures are produced by this crate, not by Iceberg-Java, and the dv_blob_golden_* tests validate the codec against its own encoder rather than against a reference. The PR description says the fixtures come from the Java reference writer, but this test overwrites them with roaring-encoded bytes.

It also leaves a real gap. Java runs runLengthEncode (runOptimize) on each 32-bit bitmap before serializing, so production DVs use RUN containers. roaring's serialize_into does not run-optimize, so none of these fixtures decode a RUN container, which is the case most likely to break cross-engine reads.

Commit at least one fixture produced by Iceberg-Java, ideally a run-heavy one like the dense range, document the exact command that generated it in the README, and either drop this regenerate test or gate it so it cannot silently replace the reference bytes. #2678 already committed Java-verified DV fixtures under testdata/puffin that can be reused here.

mbutrovich · 2026-07-09T21:21:13Z

+
+        let roaring_end = blob_bytes.len() - DV_CRC_SIZE;
+        let mut cursor = Cursor::new(&blob_bytes[magic_end..roaring_end]);
+        let inner = RoaringTreemap::deserialize_from(&mut cursor).map_err(|e| {


Check the CRC before deserializing the roaring payload. Right now the payload is parsed here and the CRC is verified at line 141, so a corrupted blob fails with an opaque roaring parse error instead of a clear CRC mismatch, and dv_blob_rejects_bad_crc having to accept either error is the symptom. The CRC is a single cheap pass over blob_bytes[4..len-4], so verifying it first gives a deterministic error and keeps corrupt bytes out of the parser.

mbutrovich · 2026-07-09T21:21:27Z

    }
+
+    #[allow(dead_code)]
+    pub fn deserialize_blob(blob_bytes: &[u8], expected_cardinality: Option<u64>) -> Result<Self> {


Add a doc comment on deserialize_blob, matching the style of insert_positions above it. State the layout it expects ([length][magic][roaring][crc32]), document that expected_cardinality is an optional cross-check against the manifest record_count and is skipped when None, and list the error conditions, with a link to the Puffin spec deletion-vector-v1 section. That keeps the byte-format contract next to the code and settles where the cardinality check lives.

mbutrovich · 2026-07-09T21:21:43Z


 use crate::{Error, ErrorKind, Result};

+#[allow(dead_code)]


These constants and deserialize_blob are all #[allow(dead_code)] because nothing calls them in this PR. Add a reference to the read tracking issue #2792 next to them, and remove the allows when the loader wiring lands, so they do not become permanent.

mbutrovich · 2026-07-09T21:22:05Z

    }
+
+    #[allow(dead_code)]
+    pub fn deserialize_blob(blob_bytes: &[u8], expected_cardinality: Option<u64>) -> Result<Self> {


deserialize_blob reads a full deletion-vector-v1 blob. #2678 and #2681 use from_puffin_blob / from_serialized_bytes for the same operation, and Iceberg-Java names it deserialize on the index type. Pick one name now so it does not get renamed when these land together.

raghav-reglobe · 2026-07-11T10:45:40Z

Since the fixture gap came up: we have real deletion-vector-v1 blobs written by both Iceberg-Java and DuckDB's iceberg extension (the cross-engine writers we exercise the read path against on #2681), including run-optimized bitmaps. Happy to contribute them as testdata/deletes/ reference bytes — @Shekharrajak if that's useful, say the word and I'll open a small PR against your branch.

On naming: no attachment to from_puffin_blob / from_serialized_bytes on our side — happy to converge on whatever gets settled here.

mbutrovich · 2026-07-21T17:52:34Z

Closing in favor of #2866. Thank you for the initial PR, @Shekharrajak!

Shekharrajak force-pushed the feat/dv-blob-parser-2411 branch 2 times, most recently from 7ce78ba to f625093 Compare May 8, 2026 18:27

Shekharrajak changed the title ~~[WIP] feat(delete-vector): parse deletion-vector-v1 puffin blob - read support~~ feat(delete-vector): parse deletion-vector-v1 puffin blob - read support May 9, 2026

Shekharrajak commented May 9, 2026

View reviewed changes

Shekharrajak commented May 13, 2026

View reviewed changes

feat(delete-vector): parse deletion-vector-v1 puffin blob

5c928e2

Shekharrajak force-pushed the feat/dv-blob-parser-2411 branch from f625093 to 5c928e2 Compare May 13, 2026 17:05

Shekharrajak mentioned this pull request May 29, 2026

Tracking issues of Iceberg Rust 0.10.0 Release #2527

Closed

22 tasks

u70b3 mentioned this pull request Jul 1, 2026

feat: add core COW rewrite primitive #2752

Open

mbutrovich mentioned this pull request Jul 9, 2026

[EPIC] Deletion vector read support (deletion-vector-v1 Puffin blobs) #2792

Open

3 tasks

mbutrovich suggested changes Jul 9, 2026

View reviewed changes

mbutrovich mentioned this pull request Jul 10, 2026

feat: [DO NOT MERGE] Iceberg table format V3: native deletion vector reads, fall back for other V3 features apache/datafusion-comet#4887

Draft

mbutrovich mentioned this pull request Jul 21, 2026

feat(delete-vector): [1/N] decode deletion-vector-v1 puffin blobs #2866

Open

mbutrovich closed this Jul 21, 2026

Uh oh!

Conversation

Shekharrajak commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

What changes are included in this PR?

Are these changes tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mbutrovich commented May 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Shekharrajak commented May 14, 2026

Uh oh!

Shekharrajak commented Jun 10, 2026

Uh oh!

mbutrovich left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

raghav-reglobe commented Jul 11, 2026

Uh oh!

mbutrovich commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Shekharrajak commented May 7, 2026 •

edited

Loading