Core: Add all puffin files metadata table

[yangshangqing95]

Summary

This PR adds a new all_puffin_files metadata table that exposes Puffin files associated with all snapshots currently tracked in Iceberg table metadata.

The metadata table reports Puffin files referenced through two metadata sources:

• Registered statistics files from TableMetadata#statisticsFiles()
• Deletion vector files referenced by live delete-manifest entries

The implementation does not open Puffin files, read Puffin footers, or parse blob payload bytes. All reported information is derived from Iceberg table metadata, snapshot manifest lists, and delete manifests.

Each row represents a Puffin file associated with one reference snapshot through one metadata source. A physical Puffin file may appear in multiple rows when it is referenced by multiple snapshots or through multiple sources.

The table contains the following columns:

Column	Description
reference_snapshot_id	ID of the snapshot that references the Puffin file
file_path	Fully qualified location of the Puffin file
source	Metadata source: statistics or deletion_vector
file_size_in_bytes	Total size of the Puffin file in bytes
referenced_blob_count	Number of referenced Puffin blobs represented by the row
referenced_blob_types	Distinct types of the referenced blobs
referenced_fields	Distinct referenced fields, represented as field_id and current_field_name pairs

current_field_name is resolved against the current table schema and may be null when a referenced field has been dropped.

Example

SELECT *
FROM catalog.db.table.all_puffin_files;

Example output:

reference_snapshot_id	file_path	source	file_size_in_bytes	referenced_blob_count	referenced_blob_types	referenced_fields
1234567890	.../1234567890-table.stats	statistics	2144	2	["apache-datasketches-theta-v1"]	[{"field_id":1,"current_field_name":"id"},{"field_id":2,"current_field_name":"data"}]
1234567890	.../00000-4-deletes.puffin	deletion_vector	525	1	["deletion-vector-v1"]	[{"field_id":<ROW_POSITION_FIELD_ID>,"current_field_name":"_pos"}]
1234567891	.../00000-4-deletes.puffin	deletion_vector	525	1	["deletion-vector-v1"]	[{"field_id":<ROW_POSITION_FIELD_ID>,"current_field_name":"_pos"}]

The same deletion-vector Puffin file appears more than once in this example because it is referenced by multiple snapshots.

Snapshot history semantics

The all_puffin_files metadata table scans all snapshots currently retained in table metadata.

Deletion vector files are reconstructed independently for each snapshot from that snapshot's live delete-manifest entries. As a result, the same physical Puffin file may appear once for every retained snapshot that references one or more blobs from the file.

Statistics files are obtained from the statistics files currently registered in table metadata and associated with their registered snapshot IDs.

Statistics registration can be updated independently of data snapshots. Therefore, the table does not reconstruct historical statistics registration state. Replaced or removed statistics files are not reported merely because their associated snapshots are still retained.

The table reports rows by snapshot ID rather than by branch or tag name. If multiple references point to the same snapshot, the snapshot's Puffin files are reported once for that snapshot.

Filter planning

The implementation performs conservative task pruning for predicates over:

• reference_snapshot_id
• source

Source predicates can avoid preparing or scanning the unrelated metadata source entirely.

Snapshot predicates can prevent tasks from being created for snapshots that cannot match the filter.

A three-state evaluator is used during planning:

• CANNOT_MATCH
• MIGHT_MATCH
• MUST_MATCH

Predicates over unsupported fields or values that are not known during the current planning phase return MIGHT_MATCH.

This is especially important for negated predicates, because negating an unknown predicate must remain unknown and must not incorrectly prune a task.

Deletion-vector tasks are only planned for format version 3 or later. Snapshot summaries are used conservatively to skip snapshots that explicitly report zero delete files.

Motivation

Iceberg stores multiple types of metadata in Puffin files, including table statistics and deletion vectors.

Users currently need to query individual snapshots or inspect metadata JSON files, manifest lists, and delete manifests separately to understand how Puffin file references evolve over time.

Exposing this information through an all-snapshots metadata table makes it easier to:

• Inspect statistics files registered for retained snapshots
• Identify Puffin files containing deletion vectors
• Determine which snapshots reference a physical Puffin file
• Find Puffin files shared across multiple snapshots
• Determine how many distinct blobs in a Puffin file are referenced by each snapshot
• Inspect the blob types and fields associated with those references
• Compare statistics and deletion-vector references across snapshot history
• Debug stale, replaced, duplicated, or unexpected Puffin metadata
• Inspect Puffin usage without opening Puffin files or reading blob payloads

Tests

The added tests cover:

• Metadata table schema and field documentation
• Tables without Puffin references
• Registered statistics files
• Statistics blob count, type, and field aggregation
• Statistics replacement and removal
• Field rename and dropped-field name resolution
• Deletion vector files in format version 3 tables
• Ignoring equality deletes and other non-Puffin delete files
• other behaviors