Skip to content

[Pipe] Use buffered input for TsFile scan parser#18254

Open
Caideyipi wants to merge 3 commits into
apache:masterfrom
Caideyipi:pipe-buffered-tsfile-input
Open

[Pipe] Use buffered input for TsFile scan parser#18254
Caideyipi wants to merge 3 commits into
apache:masterfrom
Caideyipi:pipe-buffered-tsfile-input

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Description

Use buffered input for Pipe TsFile scanning

  • Upgrade TsFile to 2.3.2-260721-SNAPSHOT, which provides BufferedTsFileInput and the new TsFileInput-based TsFileSequenceReader API.
  • Use BufferedTsFileInput for the no-modification path of TsFileInsertionEventScanParser, where the reader performs small sequential reads.
  • Keep the existing String-based reader path when modifications are present so metadata loading and device metadata caching remain enabled there.

Keep aligned TVList memory accounting accurate

TsFile 260721 changed BitMap to delegate storage to a BitMapImpl. The previous calculation inferred bitmap memory from the shallow BitMap class layout and its byte array, so it omitted the new implementation object.

  • Use BitMap.ramBytesUsed() to include the wrapper, implementation, and backing storage.
  • Keep the reference held by the bitmap list in the aligned TVList block cost.
  • Add a focused regression test and update the exact TsFileProcessorTest RAM expectations.

Validation

  • mvn -nsu spotless:apply -pl iotdb-core/datanode
  • Targeted regression tests: 3 tests passed.
  • Full TsFileProcessorTest and AlignedTVListTest: 25 tests passed.
  • Checkstyle, Spotless check, compilation, and git diff --check passed.

This PR has:

  • been self-reviewed.
  • added or updated version, license, or notice information
  • added comments explaining the why and the intent of the code wherever it would not be obvious for an unfamiliar reader.

Key changed/added classes (or packages if there are too many classes) in this PR
  • TsFileInsertionEventScanParser
  • AlignedTVList
  • TsFileProcessorTest
  • AlignedTVListTest
  • Root pom.xml

@Caideyipi
Caideyipi force-pushed the pipe-buffered-tsfile-input branch from 439a295 to 0672b08 Compare July 22, 2026 04:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant