feat: migrate Paimon C++ 0.3 development history - #161

Merged

SteNicholas merged 138 commits into

apache:mainfrom

lxy-9602:migrate-0.3

Aug 1, 2026

Commits on Jul 31, 2026

feat: Add metadata system tables

suxiaogang223
authored and
lxy-9602
committed
fix: add break for CHAR_END_SUBTYPE

lszskye
authored and
lxy-9602
committed
fix: Fix MemorySegment::Compare to use big-endian byte-order comparison semantics
lxy-9602
committed
chore(license): refine build support notices

zjw1111
authored and
lxy-9602
committed
fix: Fix undefined behavior in BinarySection bitmask constants caused by signed left shift
lxy-9602
committed
refactor: ColumnarArray: change constructor parameter to raw pointer to clarify non-owning semantics
lxy-9602
committed
feat: Support macOS builds with AppleClang and Homebrew Clang

suxiaogang223
authored and
lxy-9602
committed
fix: fix undefined behavior in bitwise left-shift operations across codebase
lxy-9602
committed
feat: optimize parquet reads with page-level filtering

liangjie3138
authored and
lxy-9602
committed
fix: add ScopeGuard to wait async tasks before early return in OrphanFilesCleanerImpl::Clean()

zjw1111
authored and
lxy-9602
committed
feat(lumina): support null values in vector column during index building
lxy-9602
committed
chore: modify comment in RecordBatch & add check in GenericRow

lszskye
authored and
lxy-9602
committed
fix: fix clang-tidy issues

zjw1111
authored and
lxy-9602
committed
chore: add check scale in BinaryRowWriter

lszskye
authored and
lxy-9602
committed
fix: fix ub in Blob

lszskye
authored and
lxy-9602
committed
refact: Refactor Literal operator== to delegate to CompareTo for consistent equality semantics
lxy-9602
committed
refactor: Reuse RowGroupPageIndexReader across columns to improve page-level predicate pushdown performance

zhf999
authored and
lxy-9602
committed
feat(blob): Support blob-descriptor-field for inline blob descriptor storage
lxy-9602
committed
fix: Like::TestString to align with Java LIKE semantics
lxy-9602
committed
chore: update LICENSE with additional PyTorch copyright information

zjw1111
authored and
lxy-9602
committed
fix: date validation reject invalid dates instead of silent normalization

dalingmeng
authored and
lxy-9602
committed
fix: fix thread-safe problem for file store path factory

lucasfang
authored and
lxy-9602
committed
feat: Improve PK MOR read performance with batch queue and cached column metadata
lxy-9602
committed
fix: avoid signed integer overflow UB in BloomFilter and DeltaVarintCompressor
lxy-9602
committed
feat: optimize count(*) for pk/append table

authored and
lxy-9602
committed
feat(blob): support blob-view-field for cross-table blob reference re…

lszskye
authored and
lxy-9602
committed
fix: fix unstable prefetch reader case

lucasfang
authored and
lxy-9602
committed
feat: remove support for FLOAT/DOUBLE type partition field

lszskye
authored and
lxy-9602
committed
fix: skip object store check when UseRESTCatalogCommit is enabled

liujiayi771
authored and
lxy-9602
committed
feat: add Bucket() virtual interface to DataSplit base class

fourier307
authored and
lxy-9602
committed
fix(fs): normalize local filesystem paths and some refactoring

zjw1111
authored and
lxy-9602
committed
feat(benchmark): add append/pk table benchmark

lucasfang
authored and
lxy-9602
committed
Allow FileStoreCommit for PK tables with postpone bucket mode

authored and
lxy-9602
committed
fix: Avoid signed overflow UB in BSI index and null-deref in bloom filter index
lxy-9602
committed
fix: LZ4 block compressor/decompressor safety issues
lxy-9602
committed
fix(avro): format avro types in decoder errors

wgtmac
authored and
lxy-9602
committed
fix: override CachedInputStream::Advance to avoid real I/O for skipped pages

zhf999
authored and
lxy-9602
committed
fix: replace arrow type DECIMAL to DECIMAL128

lucasfang
authored and
lxy-9602
committed
fix: remove trailing semicolons from statement-like macro definitions

lucasfang
authored and
lxy-9602
committed
fix: change FileStorePathFactory::Create() to return shared_ptr

lucasfang
authored and
lxy-9602
committed
refactor(blob): refactor BlobFileContext to honor the actual schema when classifying BLOB fields
lxy-9602
committed
fix: address ubsan findings

zjw1111
authored and
lxy-9602
committed
fix(build): make package config relocatable

zjw1111
authored and
lxy-9602
committed
fix(parquet): fix incorrect chunk_end calculation in ComputePageRanges when dictionary page is present

zhf999
authored and
lxy-9602
committed
feat(fs): support int64 file IO sizes

zjw1111
authored and
lxy-9602
committed
fix: small fixes for parquet reader including RowRanges, PreBuffer, ColumnIndexFilter and code cleanup

zhf999
authored and
lxy-9602
committed
feat(blob view): support multi-thread reading upstream table & add inte test

lszskye
authored and
lxy-9602
committed
test: add blob compatible test
lxy-9602
committed
refractor(parquet): simplify FileReaderWrapper and PageFilteredRowGroupReader with unified TargetRowGroup structure

zhf999
authored and
lxy-9602
committed
fix(scan): fix dangling reference and unsigned overflow in FileStoreScan
lxy-9602
committed
feat(shredding): Add infrastructure for shared-shredding MAP storage layout
lxy-9602
committed
fix: make byte hashing independent of char signedness

wgtmac
authored and
lxy-9602
committed
feat(shared-shredding): support nested field projection in ORC format reader

lszskye
authored and
lxy-9602
committed
fix: prevent zero-thread DefaultExecutor and validate FileSystem/SchemeMap conflict

zjw1111
authored and
lxy-9602
committed
fix: small fixes for AI reviewing feedback

zhf999
authored and
lxy-9602
committed
feat: Optimize PK MOR read performance by reduce per-row object allocation and improve merge read performance.

dalingmeng
authored and
lxy-9602
committed
feat: Add manifests and files system tables

suxiaogang223
authored and
lxy-9602
committed
fix: canonicalize NaN for Hive bucket hash
lxy-9602
committed
feat(cache): support manifest file cache

gripleaf
authored and
lxy-9602
committed
feat(schema): support pk table schema evolution

gripleaf
authored and
lxy-9602
committed
fix: fix min/max row id serialize bug

lucasfang
authored and
lxy-9602
committed
feat: update lumina lib to v0.3.0-rc1
lxy-9602
committed
feat(shredding): support shared-shredding map write
lxy-9602
committed
fix(parquet): fallback nested field reading to RowGroup reading (walkaround)

zhf999
authored and
lxy-9602
committed
feat(tantivy): add Tantivy full-text global index via Rust FFI

spaces-X
authored and
lxy-9602
committed
feat(parquet): support parquet metadata cache

gripleaf
authored and
lxy-9602
committed
fix(format): reset avro reader on schema change

zjw1111
authored and
lxy-9602
committed
feat(shredding): support shared shredding file reader

lszskye
authored and
lxy-9602
committed
fix: address avro blob and executor review issues

zjw1111
authored and
lxy-9602
committed
feat(shredding): improve shared-shredding adaptive width restore & refactor write
lxy-9602
committed
feat: support read for nested type sub column

lucasfang
authored and
lxy-9602
committed
fix: close input/output streams when remote lookup file read/write failed
lxy-9602
committed
fix(testing): address review issues

zjw1111
authored and
lxy-9602
committed
feat(shared_shredding): add read inte test for shared_shredding

lszskye
authored and
lxy-9602
committed
feat: add CreateReader api with field_name & index_type

lszskye
authored and
lxy-9602
committed
feat(shredding): add shared-shredding map placement policies

zjw1111
authored and
lxy-9602
committed
feat(types): support CHAR/VARCHAR/BINARY/VARBINARY in data type json parser

SteNicholas
authored and
lxy-9602
committed
fix: align duplicate-key replacement semantics with Java Paimon

lucasfang
authored and
lxy-9602
committed
fix: handle non-contiguous RowRanges when resolving global row IDs

zhf999
authored and
lxy-9602
committed
test: add tests for executor
lxy-9602
committed
fix: release global index writer before reader to avoid mem issue when write index failed
lxy-9602
committed
fix: fix IsThreadSafe() in UnionGlobalIndexReader

lszskye
authored and
lxy-9602
committed
test: add map shared-shredding tests for compaction and alter table and predicates
lxy-9602
committed
feat(blob): Remove blob external storage mode
lxy-9602
committed
feat(schema): scan context support set table schema

gripleaf
authored and
lxy-9602
committed
feat(manifest): support snapshot live manifest cache

gripleaf
authored and
lxy-9602
committed
feat(parquet): support pushing bitmaps down to page-level filtering (except for nested columns)

zhf999
authored and
lxy-9602
committed
chore: adjust set read_ranges_freshed_ in SetReadRanges

lszskye
authored and
lxy-9602
committed
docs: add coding agent guidelines

zjw1111
authored and
lxy-9602
committed
fix: compile with correct flags with AppleClang

smdsbz
authored and
lxy-9602
committed
feat(shared-shredding): Expose shared-shredding utilities

dalingmeng
authored and
lxy-9602
committed
feat(blob view): support blob-view.resolve.enabled to preserve blob view references at read time

SteNicholas
authored and
lxy-9602
committed
fix(build): disable glog unwind without libunwind

mrdrivingduck
authored and
lxy-9602
committed
fix(avro): include headers used by file batch reader

wgtmac
authored and
lxy-9602
committed
fix(orc): avoid removed Arrow bitmap append API

wgtmac
authored and
lxy-9602
committed
feat(commit): align functionality with java paimon commit

lucasfang
authored and
lxy-9602
committed
feat: add read-optimized system table

gripleaf
authored and
lxy-9602
committed
feat(blob): support blob.split-by-file-size to weigh blob files in scan splitting

SteNicholas
authored and
lxy-9602
committed
feat(blob): support blob-write-null-on-missing-file and blob-write-null-on-fetch-failure options

SteNicholas
authored and
lxy-9602
committed
fix(commit): fix changelogRecordCount nullopt and add ut

lucasfang
authored and
lxy-9602
committed
fix(fs): create parent directories before opening Jindo writers

Zouxxyy
authored and
lxy-9602
committed
fix(jindo): make async reads concurrency-safe

zjw1111
authored and
lxy-9602
committed
fix: enable TSAN and resolve detected data races

zjw1111
authored and
lxy-9602
committed
feat(parquet): support configurable bitmap row-range refining strategies (coalesce and trim)

zhf999
authored and
lxy-9602
committed
feat(commit): support TruncateTable/Abort/RollbackToAsLatest and add ut

lucasfang
authored and
lxy-9602
committed
build: fetch Lumina from release package

zjw1111
authored and
lxy-9602
committed
fix(blob): identify a missing blob file with FileSystem::Exists

SteNicholas
authored and
lxy-9602
committed
fix(btree): preserve empty keys in index metadata & disable global bitmap index
lxy-9602
committed
feat(variant): support variant data type

SteNicholas
authored and
lxy-9602
committed
build: fetch Boost from OSS and define network test macro globally

zjw1111
authored and
lxy-9602
committed
feat: add metrics storage-read-bytes for parquet and fix clean inte test

lucasfang
authored and
lxy-9602
committed
feat(lumina): support Lumina tag filtering
lxy-9602
committed
fix(reader): ensure batch readers return zero-offset Arrow arrays
lxy-9602
committed
feat: add global system tables framework under `sys` database

suxiaogang223
authored and
lxy-9602
committed
refactor(format): remove Lance support

zjw1111
authored and
lxy-9602
committed
refactor(tantivy): move FFI crate under crates

zjw1111
authored and
lxy-9602
committed
chore: clean up attribution and test data whitespace

zjw1111
authored and
lxy-9602
committed
fix(blob): support blob-only writes and reject nested blob fields
lxy-9602
committed
fix: nested list schema evolution

XiaoHongbo-Hope
authored and
lxy-9602
committed
fix(abi): dynamic_cast error on Predicate across shared libraries

zhangweilst
authored and
lxy-9602
committed
feat(fs): add object store and S3 file systems

mrdrivingduck
authored and
lxy-9602
committed
feat(parquet): support page-level (page index) filtering for nested column types (patch arrow)

zhf999
authored and
lxy-9602
committed
test: add test for test coverage

lucasfang
authored and
lxy-9602
committed
test: add ut for release-0.3

lszskye
authored and
lxy-9602
committed
chore: remove Git LFS and refresh documentation

zjw1111
authored and
lxy-9602
committed
fix(blob): allow blob files across schema IDs
lxy-9602
committed
fix(core): support fallback keys for ignore-delete option

SteNicholas
authored and
lxy-9602
committed
feat(shredding): support adaptive schemas across rolling files
lxy-9602
committed
fix(lucene): normalize prefix and wildcard queries

lszskye
authored and
lxy-9602
committed
chore: update version to 0.3.0

lszskye
authored and
lxy-9602
committed
feat(blob): support placeholder fallback for partial updates

SteNicholas
authored and
lxy-9602
committed
feat(parquet): support configuring Arrow pre-buffer hole-size-limit for range coalescing

zhf999
authored and
lxy-9602
committed
fix: align floating-point semantics with Java
lxy-9602
committed
chore: remove target-only migration leftover
lxy-9602
committed
fix(release): align documentation version
lxy-9602
committed
ci: add consolidated build and test workflow
lxy-9602
committed
fix: align source archive and integration tests
lxy-9602
committed
ci: disable clang-tidy
lxy-9602
committed