Skip to content

[Pipe] Resume TsFile parsing after retryable failures#18261

Open
Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:fix/pipe-parser-progress-resume
Open

[Pipe] Resume TsFile parsing after retryable failures#18261
Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:fix/pipe-parser-progress-resume

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

This is the parser progress recovery part of the Pipe resilience work. It is independent of #18260 and uses the existing parser memory reservation API on master.

Preserve parsing progress

PipeTsFileInsertionEvent now retains:

  • the tablet iterator created by the TsFile parser;
  • the currently parsed but not successfully consumed tablet;
  • the parsed tablet count and count-operation progress.

If tablet consumption fails, the event yields its active parser memory reservation but keeps the parser, iterator, and pending tablet. The next subtask retry reacquires parser memory and resumes from the exact same tablet object instead of reopening and reparsing the TsFile from the beginning.

Parser lifecycle

Call sites that split Pipe TsFile events now close the parser only after successful completion. Processing exceptions stop at the failed tablet and propagate immediately. Non-Pipe TsFile implementations retain their original finally-close behavior.

Explicit event close and resource finalization still release the pending tablet and parser resources.

OOM retry backoff

Local parser OOM retries use bounded exponential backoff derived from the existing Pipe memory retry interval and maximum retry count. When the retry policy decides to yield, later subtask retries resume from saved progress.

Tests

TsFileInsertionEventParserTest verifies that an OOM does not close the parser and that the next retry receives the identical failed tablet object. The full parser test class passes.


This PR has:

  • been self-reviewed.
    • concurrent read and write
  • added comments explaining the progress and resource-lifecycle decisions.
  • added or updated unit tests for the retry path.

Key changed/added classes
  • PipeTsFileInsertionEvent
  • PipeProcessorSubtask
  • PipeEventCollector
  • AggregateProcessor
  • DownSamplingProcessor
  • WebSocketSink
  • TsFileInsertionEventParserTest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant