Skip to content

[Subscription] Bound consensus prefetch memory usage#18298

Open
Caideyipi wants to merge 2 commits into
apache:masterfrom
Caideyipi:fix/consensus-subscription-memory-backpressure
Open

[Subscription] Bound consensus prefetch memory usage#18298
Caideyipi wants to merge 2 commits into
apache:masterfrom
Caideyipi:fix/consensus-subscription-memory-backpressure

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Bound consensus subscription materialization memory

  • Add a dedicated subscription share to query memory. It defaults to about 20% when subscription is enabled, is forced to 0 when disabled, and remains compatible with the legacy eight-part query-memory configuration.
  • Enforce one DataNode-wide byte budget across PreparedEntry, realtimeEntriesByWriter, lingerBatch, prefetchingQueue, and inFlightEvents.
  • Drain already materialized backlog before converting more WAL entries, and leave blocked data in WAL when no budget is available.

Close lifecycle and ownership races

  • Release retained bytes on ACK, seek, close, leader deactivation, fatal-cleanup reconciliation, and recycled-event late ACK paths.
  • Prevent a late ACK from an old consumer from stealing an event already claimed by a new consumer.
  • Separate initialization and runtime-activation monitors to avoid lock-order inversion, and reject outdated commit contexts after seek/reboot changes.

Regression coverage

  • Add wide-table high-throughput tests with a paused consumer and assert materialized memory remains bounded.
  • Add coverage for ACK release, deactivation cleanup, recycled-event ownership, oversized entries, and orphaned reservation reconciliation.

This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added comments explaining the why and intent of non-obvious lifecycle code.
  • added unit tests for new code paths.

Tested with:

./mvnw -nsu -pl iotdb-core/datanode -am \
  -Dtest=DataNodeMemoryConfigTest,SubscriptionMemoryManagerTest,ConsensusPrefetchingQueueTest \
  test -DskipITs -Dsurefire.failIfNoSpecifiedTests=false

Result: 28 reactor modules succeeded; 29 tests passed; Checkstyle and Spotless passed.


Key changed/added classes
  • DataNodeMemoryConfig
  • SubscriptionMemoryManager
  • ConsensusPrefetchingQueue
  • QueryEngineMemoryMetrics

@Caideyipi
Caideyipi force-pushed the fix/consensus-subscription-memory-backpressure branch from 6680310 to 010635c Compare July 24, 2026 08:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant