Skip to content

feat: asynchronous message publishing with batching and pre-commit delivery guarantees (Enterprise) - #684

Merged
dgafka merged 38 commits into
mainfrom
async-publishing
Aug 4, 2026
Merged

feat: asynchronous message publishing with batching and pre-commit delivery guarantees (Enterprise)#684
dgafka merged 38 commits into
mainfrom
async-publishing

Conversation

@dgafka

@dgafka dgafka commented Jul 18, 2026

Copy link
Copy Markdown
Member

Why is this change proposed?

Publishing to a message broker blocks the handler on one full round-trip per message — a
handler emitting many messages pays N sequential broker round-trips, which caps publishing
throughput as systems scale.

Ecotone now provides an Enterprise feature that makes publishing scale, while still keeping
all the delivery guarantees end users rely on: every message is still confirmed by the
broker, a failed delivery still fails the business operation or is routed to the error
channel, and every failure points to the exact message that failed — nothing is ever lost
silently.

This is achieved by firing messages to the broker immediately, batching them natively per
provider, and deferring confirmation awaiting to the last responsible moment — messages
travel to the broker while the handler keeps working, and all confirmations are collected
before the operation completes.

Benchmarks

Publishing 1,000 messages, relative to each provider's previous message-by-message
synchronous publishing (= 100%). Mode of 10 iterations, all delivery confirmations awaited:

Provider Feature used Time vs previous Speed-up
Kafka Batching + async sending 9% ~11×
SQS Batching + async sending 9% ~11×
DBAL Batching 10% ~10×
AMQP Batching + async sending 32% ~3×
Redis Batching 36% ~3×
Kafka Async sending (single messages) 36% ~3×
SQS Async sending (single messages) 65% ~1.5×

Batching + async sending publishes the workload as provider-native batches with
confirmation awaiting deferred; async sending fires individual messages without waiting.
DBAL and Redis write synchronously, so batching is their applicable feature.

Resulting flow

graph TD
    A[Handler emits messages] --> B[Collector gathers them]
    B -->|one BatchMessage| C[Adapter: provider-native batch publish, no wait]
    C --> D[Registry: PendingDelivery per publish]
    D --> E[Waiter interceptor: await all confirmations]
    E -->|all confirmed| F[Operation completes]
    E -->|some failed| G[Only failed messages to error channel, or operation fails]
Loading

Enabling

ServiceConfiguration::createWithDefaults()
    ->withExtensionObjects([
        // Message Channel: messages published from handlers are gathered,
        // sent as one batch and confirmed before the operation completes
        AmqpBackedMessageChannelBuilder::create('orders')
            ->withAsyncPublishing(),
        KafkaMessageChannelBuilder::create('orders')
            ->withAsyncPublishing(timeoutInMilliseconds: 12000),

        // Message Publisher: enables MessagePublisher::asyncPublish()
        AmqpMessagePublisherConfiguration::create()
            ->withAsyncPublishing(),
    ]);

Example

#[CommandHandler]
public function place(PlaceOrder $command, EventBus $eventBus): void
{
    // events are batched to the broker and their confirmations
    // awaited automatically before the operation completes
    $eventBus->publish(new OrderWasPlaced($command->orderId));
}

// explicit async publishing with a Future
$future = $messagePublisher->asyncPublish(
    BatchMessage::constructEmpty()
        ->append($firstOrder)
        ->append($secondOrder)
);

$future->resolve(); // throws AsyncPublishingFailedException listing only the messages that failed

Pull Request Contribution Terms

  • I have read and agree to the contribution terms outlined in CONTRIBUTING.

dgafka added 30 commits July 3, 2026 07:00
…en by command handler batch publishing tests with delay and ttl across providers
…cycle

- catch \RdKafka\Exception instead of the undefined KafkaException so
  producer queue-full backpressure actually polls and retries rather than
  failing the publish on the first full-queue signal
- require AsyncPublishingRegistry in the SQS and Enqueue outbound adapters,
  matching DBAL and Redis, removing the nullable dereference
- drop the register_shutdown_function auto-flush; unresolved deliveries are
  now flushed via explicit flushUnawaitedDeliveries and the backlog limit
dgafka added 7 commits July 18, 2026 08:00
…nd error channel routing, shutdown flush of unawaited deliveries
- capture amqp confirm epoch pre-publish and fail all records on reconnect so stale delivery tags can never settle against a fresh channel
- redis batch publishing reports exactly which entries were not pushed so retries do not duplicate already delivered ones
- dead letter each failed batch delivery as separate message in send retries exhaustion path
- rename AsyncPublishingFailedException to PublishingFailedException as it serves sync and async publishing
- make BatchMessage immutable with fromEntries bulk construction
- raise sqs publishing timeout default to 25s as pre-batching sends had no http timeout
…atch handling

BatchMessage publishing is now an explicit channel capability instead of a
transparent interceptor fallback:

- In-memory queue channels split batches into individual messages under
  enterprise licence and throw LicensingException without one, so tests
  mirror production behaviour
- Broker outbound adapters reject BatchMessage when async publishing is not
  enabled, closing the direct MessagePublisher bypass
- Collector combines into a batch only for batch-supporting channels and
  flattens already-batched payloads, keeping per-message flow and channel
  spies intact everywhere else
- SendingInterceptorAdapter no longer splits batches
- Kafka message channel licence test covers the batch-enabled channel variant
…nd give reactive quickstart consumer realistic time budget
Comment thread packages/Sqs/src/SqsOutboundChannelAdapter.php
Comment thread packages/Amqp/src/AmqpOutboundChannelAdapter.php Outdated
Comment thread packages/Ecotone/src/Messaging/BatchMessage.php
Comment thread packages/Redis/src/RedisOutboundChannelAdapter.php Outdated
@dgafka
dgafka merged commit eed173a into main Aug 4, 2026
8 checks passed
@dgafka
dgafka deleted the async-publishing branch August 4, 2026 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant