[core] chain table support special partition expire.#7643
Open
Stephen0421 wants to merge 1 commit intoapache:masterfrom
Open
[core] chain table support special partition expire.#7643Stephen0421 wants to merge 1 commit intoapache:masterfrom
Stephen0421 wants to merge 1 commit intoapache:masterfrom
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
This PR implements partition expiration for chain tables. Chain tables store data across snapshot
and delta branches, where delta partitions depend on their nearest earlier snapshot partition as an
anchor for merge-on-read. Standard partition expiration cannot be applied directly because dropping
a snapshot partition without considering its dependent deltas would break the chain integrity.
Changes
New: Segment-based partition expiration (
ChainTablePartitionExpire)Introduces a segment-based expiration algorithm that preserves chain integrity. A segment consists
of one snapshot partition and all delta partitions whose time falls between that snapshot and the
next snapshot. The segment is the atomic unit of expiration.
Algorithm per group:
now - partition.expiration-time).kept as anchor for its dependent deltas).
expirable segments together with their associated delta partitions.
Refactored:
PartitionExpireinterface extractionExtracted
PartitionExpirefrom a concrete class into an interface with three methods:expire(long),isValueExpiration(), andisValueAllExpired(Collection<BinaryRow>).The original implementation is preserved as
NormalPartitionExpire. All existing consumers(Flink/Spark procedures, actions,
TableCommitImpl,ConflictDetection) use only the interfacemethods — no compatibility impact.
Fixed:
ChainTableCommitPreCallbackgroup partition awarenessThe commit pre-callback that validates snapshot partition drops was not group-partition aware.
It used full partition comparators and triangular predicates, which could match partitions across
different groups and produce incorrect pre/next snapshot lookups. Refactored to:
ChainPartitionProjectorto extract group and chain dimensions.Tests
maxExpireNumlimits number of expired segmentsisValueAllExpired: anchor partitions not reported as expiredisValueAllExpired: groups with < 2 snapshots retain allisValueAllExpired: cross-group mixed scenariosisValueAllExpired: partitions after cutoff not expired