Skip to content

Core: Add pruneColumnStats to RewriteManifests#17258

Open
JoeryH wants to merge 1 commit into
apache:mainfrom
JoeryH:prune-column-stats-rewrite-manifests
Open

Core: Add pruneColumnStats to RewriteManifests#17258
JoeryH wants to merge 1 commit into
apache:mainfrom
JoeryH:prune-column-stats-rewrite-manifests

Conversation

@JoeryH

@JoeryH JoeryH commented Jul 16, 2026

Copy link
Copy Markdown

Closes #17257.

Adds a pruneColumnStats() option to RewriteManifests. When enabled, column
stats that exceed the table's current write.metadata.metrics.* configuration
are dropped as manifests are rewritten: columns configured none lose all
stats, counts columns lose their bounds, and truncate(n)/full columns are
left unchanged (existing bounds are never re-truncated).

Iceberg infers metrics for up to 100 columns by default, which bloats manifests
and slows planning on wide tables. Tightening the metrics config doesn't shrink
existing manifests today because RewriteManifests copies each data file's
stats verbatim. This option lets users reclaim that space during a manifest
rewrite without rewriting the data files.

Pruning applies only to manifests that are actually rewritten. With no
clusterBy(...) set, enabling it triggers a full rewrite so all data files are
regrouped. The new default interface method throws
UnsupportedOperationException, keeping existing implementors compatible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add option to prune column stats when rewriting manifests

1 participant