Skip to content

[Pipe] Improve timeout diagnostics and DROP PIPE observability#18297

Open
Caideyipi wants to merge 2 commits into
apache:masterfrom
Caideyipi:fix/pipe-timeout-diagnostics
Open

[Pipe] Improve timeout diagnostics and DROP PIPE observability#18297
Caideyipi wants to merge 2 commits into
apache:masterfrom
Caideyipi:fix/pipe-timeout-diagnostics

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Description

This PR improves the observability of Pipe control procedures, especially when the client times out while the Procedure continues running in the background.

Improve timeout diagnostics for Pipe control statements

When CREATE PIPE, ALTER PIPE, START PIPE, STOP PIPE, or DROP PIPE times out while waiting for its Procedure, the returned diagnostic message now includes:

  • the Pipe operation and Procedure ID
  • the current execution or rollback stage
  • a stage-specific possible cause, or the latest retry/failure exception when available
  • an explicit reminder that the Procedure is still running in the background

The diagnostic state is volatile and non-persistent, so it does not affect Procedure execution or recovery. Internal consensus Pipe procedures retain their optimistic timeout behavior.

Make timed-out DROP PIPE observable through SHOW PIPES

The new DROP PIPE flow is:

VALIDATE_TASK
  -> CALCULATE_INFO_FOR_TASK
  -> PRE_DELETE
  -> OPERATE_ON_DATA_NODES
  -> WRITE_CONFIG_NODE_CONSENSUS

At PRE_DELETE, ConfigNode persists the Pipe runtime status as PRE_DELETE. Therefore:

  • while the Drop Procedure is unfinished, SHOW PIPES still returns the Pipe with state PRE_DELETE
  • after DataNode cleanup completes, the final ConfigNode consensus write removes the Pipe metadata
  • heartbeat parsing, metadata synchronization, failure recording, and auto-restart cannot overwrite PRE_DELETE
  • START PIPE, STOP PIPE, and ALTER PIPE reject a Pipe in PRE_DELETE; DROP PIPE can still be retried
  • task agents treat PRE_DELETE as a coordinator-only marker and wait for the explicit DROPPED metadata

Compatibility

The implementation reuses DropPipeProcedureV2 while preserving persisted data compatibility:

  • OperatePipeTaskState.PRE_DELETE is appended at ordinal 4; existing ordinals 0-3 are unchanged
  • PipeStatus.PRE_DELETE uses byte code 3; existing codes 0-2 are unchanged
  • persisted state history distinguishes the new flow from recovered legacy Procedures
  • legacy Procedures already at WRITE_CONFIG_NODE_CONSENSUS or OPERATE_ON_DATA_NODES continue with the old ordering
  • legacy Procedures that did not persist pipeMetaToDrop restore it from PipeTaskInfo before entering PRE_DELETE

Tests

Added coverage for:

  • timeout diagnostics and execution stages
  • new and legacy Drop Procedure state ordering
  • Procedure state ordinal and Pipe status byte compatibility
  • legacy Procedure recovery without persisted pipeMetaToDrop
  • PRE_DELETE consensus-plan and runtime-metadata serialization
  • SHOW PIPES output
  • heartbeat protection and auto-restart protection

Local verification:

  • Spotless passed for node-commons and confignode
  • ConfigNode and Node Commons Checkstyle passed
  • PipeMetaDeSerTest: 3 tests passed
  • focused state-machine tests: 7 tests passed
  • focused Drop serialization/recovery/auto-restart tests passed
  • all modified ConfigNode main sources and focused tests passed isolated compilation
  • git diff --check passed

The complete ConfigNode Maven test invocation is currently blocked locally by pre-existing generated Thrift/cache API mismatches in this checkout.


This PR has:

  • been self-reviewed
    • concurrent read and write
  • preserved persisted Procedure and Pipe status compatibility
  • added comments explaining the compatibility and lifecycle decisions
  • added unit tests for the new behavior and recovery paths

Key changed/added classes
  • AbstractOperatePipeProcedureV2
  • DropPipeProcedureV2
  • OperatePipeTaskState
  • StateMachineProcedure
  • PipeStatus
  • PipeTaskAgent
  • PipeTaskInfo
  • PipeHeartbeatParser
  • PipeMetaSyncProcedure
  • ProcedureManager

@Caideyipi Caideyipi changed the title [Pipe] Improve timeout diagnostics for Pipe control procedures [Pipe] Improve timeout diagnostics and DROP PIPE observability Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant