feat: asynchronous message publishing with batching and pre-commit delivery guarantees (Enterprise) - #684
Merged
Conversation
…a TCP_NODELAY default
…eue-full backpressure
…mmand before commit
…en by command handler batch publishing tests with delay and ttl across providers
…cycle - catch \RdKafka\Exception instead of the undefined KafkaException so producer queue-full backpressure actually polls and retries rather than failing the publish on the first full-queue signal - require AsyncPublishingRegistry in the SQS and Enqueue outbound adapters, matching DBAL and Redis, removing the nullable dereference - drop the register_shutdown_function auto-flush; unresolved deliveries are now flushed via explicit flushUnawaitedDeliveries and the backlog limit
…nd error channel routing, shutdown flush of unawaited deliveries
…t package testing passes
…or loaded ci runners
- capture amqp confirm epoch pre-publish and fail all records on reconnect so stale delivery tags can never settle against a fresh channel - redis batch publishing reports exactly which entries were not pushed so retries do not duplicate already delivered ones - dead letter each failed batch delivery as separate message in send retries exhaustion path - rename AsyncPublishingFailedException to PublishingFailedException as it serves sync and async publishing - make BatchMessage immutable with fromEntries bulk construction - raise sqs publishing timeout default to 25s as pre-batching sends had no http timeout
…age count with generous time budget
…atch handling BatchMessage publishing is now an explicit channel capability instead of a transparent interceptor fallback: - In-memory queue channels split batches into individual messages under enterprise licence and throw LicensingException without one, so tests mirror production behaviour - Broker outbound adapters reject BatchMessage when async publishing is not enabled, closing the direct MessagePublisher bypass - Collector combines into a batch only for batch-supporting channels and flattens already-batched payloads, keeping per-message flow and channel spies intact everywhere else - SendingInterceptorAdapter no longer splits batches - Kafka message channel licence test covers the batch-enabled channel variant
…nd give reactive quickstart consumer realistic time budget
dgafka
commented
Aug 3, 2026
dgafka
commented
Aug 3, 2026
dgafka
commented
Aug 3, 2026
dgafka
commented
Aug 3, 2026
dgafka
commented
Aug 3, 2026
…tching existing in-file pattern
This was referenced Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why is this change proposed?
Publishing to a message broker blocks the handler on one full round-trip per message — a
handler emitting many messages pays N sequential broker round-trips, which caps publishing
throughput as systems scale.
Ecotone now provides an Enterprise feature that makes publishing scale, while still keeping
all the delivery guarantees end users rely on: every message is still confirmed by the
broker, a failed delivery still fails the business operation or is routed to the error
channel, and every failure points to the exact message that failed — nothing is ever lost
silently.
This is achieved by firing messages to the broker immediately, batching them natively per
provider, and deferring confirmation awaiting to the last responsible moment — messages
travel to the broker while the handler keeps working, and all confirmations are collected
before the operation completes.
Benchmarks
Publishing 1,000 messages, relative to each provider's previous message-by-message
synchronous publishing (= 100%). Mode of 10 iterations, all delivery confirmations awaited:
Batching + async sending publishes the workload as provider-native batches with
confirmation awaiting deferred; async sending fires individual messages without waiting.
DBAL and Redis write synchronously, so batching is their applicable feature.
Resulting flow
graph TD A[Handler emits messages] --> B[Collector gathers them] B -->|one BatchMessage| C[Adapter: provider-native batch publish, no wait] C --> D[Registry: PendingDelivery per publish] D --> E[Waiter interceptor: await all confirmations] E -->|all confirmed| F[Operation completes] E -->|some failed| G[Only failed messages to error channel, or operation fails]Enabling
Example
Pull Request Contribution Terms