Skip to content

[python] Fix Parquet VARIANT reads with multi-chunk nested data#8848

Merged
JingsongLi merged 3 commits into
apache:masterfrom
XiaoHongbo-Hope:fix/variant-parquet-bounded-batches
Jul 26, 2026
Merged

[python] Fix Parquet VARIANT reads with multi-chunk nested data#8848
JingsongLi merged 3 commits into
apache:masterfrom
XiaoHongbo-Hope:fix/variant-parquet-bounded-batches

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Purpose

PyArrow can raise Nested data conversions not implemented for chunked array outputs when projected VARIANT data is decoded into multiple chunks, including within one row group or inside ROW, ARRAY, and MAP columns.

Use bounded ParquetFile.iter_batches() reads while preserving projection, predicates, filter-only columns, and row-group pruning. Exact post-read projection avoids dotted top-level name collisions, and nested shredded VARIANT values are assembled recursively.

Tests

  • Single-row-group multi-chunk reproducers for top-level and nested VARIANT.
  • Dotted top-level and nested-path projection regressions.
  • Plain and shredded VARIANT in ROW, ARRAY, and MAP, including NULL and overflow values.
  • PyArrow 19: 169 passed, 3 subtests passed.
  • PyArrow 6 targeted compatibility: 13 passed, 3 skipped.

PyArrow dataset scans can fail when a nested VARIANT child produces multiple chunks. Read projected Parquet VARIANT columns with ParquetFile.iter_batches while preserving row-group pruning, predicates, and projection.
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as draft July 25, 2026 14:39
@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Read VARIANT Parquet in bounded batches [python] Fix Parquet VARIANT reads with multi-chunk nested data Jul 25, 2026
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review July 25, 2026 22:52
@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit 2037746 into apache:master Jul 26, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants