Skip to content

[python] Render min/max_partition_stats in the $manifests system table#8842

Open
wombatu-kun wants to merge 1 commit into
apache:masterfrom
wombatu-kun:python/manifests-partition-stats
Open

[python] Render min/max_partition_stats in the $manifests system table#8842
wombatu-kun wants to merge 1 commit into
apache:masterfrom
wombatu-kun:python/manifests-partition-stats

Conversation

@wombatu-kun

Copy link
Copy Markdown
Contributor

Purpose

This closes a TODO in pypaimon/table/system/manifests_table.py:

# TODO: render min/max_partition_stats by casting partition
# rows to their string form. pypaimon
# has SimpleStats but no shared partition-row-to-string
# helper yet; emit NULL to preserve the column shape.
min_partition_stats.append(None)
max_partition_stats.append(None)

So $manifests returns NULL for min_partition_stats and max_partition_stats in pypaimon while Java renders them, and inspecting which manifests cover which partitions gives nothing on the Python side. The data is already parsed: ManifestListManager builds SimpleStats from the manifest list, only the rendering was missing.

This adds cast_row_to_string, the helper the TODO asks for: a port of RowToStringCastRule and the per type *ToStringCastRule rules of paimon-common, with fields joined by ", " inside braces, the literal null for a null field, and {} for an unpartitioned table. The types that need more than str() are boolean (lower case), binary (decoded as UTF-8), date, time, timestamp and float. Timestamps follow DateTimeUtils.formatTimestamp, which separates date and time with a space and strips trailing zeros of the fraction down to the declared precision, so datetime.isoformat() can not be used. Floats print the shortest text that round trips as float32, otherwise a float32 0.1 read into a Python float prints as 0.10000000149011612.

One point for a maintainer opinion, deliberately not changed here: pypaimon renders the partition column of $files and $buckets as pt=v/pt2=v2, while Java renders those two with the brace form and keeps pt=v only for $partitions. Aligning them would change values these tables already return, so it is out of scope for this PR.

Tests

New pypaimon/tests/row_to_string_test.py covers the rendering per type, including a null field, an empty row, the timestamp fraction at precision 0/3/6 and the float32 shortest form.

pypaimon/tests/system/manifests_table_test.py replaces the assertions that pinned the NULL placeholder: an unpartitioned table now yields {}, and a table partitioned by INT and STRING yields {1, 2024-01-01} and {2, 2024-01-02}.

Cross checked against Java on a single warehouse: a table written by pypaimon and partitioned by INT, STRING, DATE, BOOLEAN and DECIMAL renders identically on both sides.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant