bug: Transactional topic example exhausts producer_id limit with random IDs

[krasnovdm]

Bug Report

YDB Python SDK version:

Environment

Environment is irrelevant for reproduction - issue is in code example

Current behavior:

In the transactional topic write example (topic_transactions_async_example.py):

1. Each call to tx_writer without explicit producer_id generates a new random producer_id
2. Transaction retries create additional producer_ids
3. This rapidly consumes the producer_id limit (100K per 14 days)
4. The example encourages production-unsafe usage patterns

Expected behavior:

Examples should demonstrate best practices:

1. Explicit producer_id should be used to avoid random ID generation
2. Example should be resilient to transaction retries
3. Should show either fixed producer_id or partition_id usage
4. Should not risk exhausting producer_id limits in production scenarios

Steps to reproduce:

1. Use example from topic_transactions_async_example.py
2. Run write_with_tx_example method in a loop or with frequent retries
3. Observe rapid growth of producer_ids in the topic

Related code:
https://github.com/ydb-platform/ydb-python-sdk/blob/main/examples/topic/topic_transactions_async_example.py#L28

async def write_with_tx_example(driver: ydb.aio.Driver, topic: str, message_count: int = 10):
    async with ydb.aio.QuerySessionPool(driver) as session_pool:

        async def callee(tx: ydb.aio.QueryTxContext):
            tx_writer: ydb.TopicTxWriterAsyncIO = driver.topic_client.tx_writer(tx, topic)

            for i in range(message_count):
                async with await tx.execute(query=f"select {i} as res;") as result_stream:
                    async for result_set in result_stream:
                        message = str(result_set.rows[0]["res"])
                        await tx_writer.write(ydb.TopicWriterMessage(message))
                        print(f"Message {result_set.rows[0]['res']} was written with tx.")

        await session_pool.retry_tx_async(callee)

Other information:

Current example can exhaust 100K producer_id limit under moderate load

Possible Solutions
Fixed producer_id
Explicit partition selection

[robot-vibe-db]

Analysis of Issue #700: Transactional Topic Example Exhausts producer_id Limit

Severity: HIGH

This is a significant documentation/example bug that can lead users into production-unsafe usage patterns. The issue affects both sync and async transactional topic writer examples.

---

Root Cause

In ydb/_topic_writer/topic_writer.py:46-48, the PublicWriterSettings.__post_init__ method auto-generates a random producer_id when none is provided:

def __post_init__(self):
    if self.producer_id is None:
        self.producer_id = uuid.uuid4().hex

The transactional examples call tx_writer(tx, topic) without specifying producer_id:

• Async (topic_transactions_async_example.py:28):

  tx_writer = driver.topic_client.tx_writer(tx, topic)  # No producer_id

• Sync (topic_transactions_example.py:27):

  tx_writer = driver.topic_client.tx_writer(tx, topic)  # No producer_id

Since the callee function is inside retry_tx_async/retry_tx_sync, every call (including retries) creates a new tx_writer with a fresh random UUID as producer_id. This rapidly exhausts the 100K producer_id limit (per 14 days).

Inconsistency with Other Examples

The regular writer examples all correctly use explicit producer_id:

• writer_example.py: producer_id="producer-id" (every call)
• writer_async_example.py: producer_id="producer-id" (every call)

Only the transactional examples omit this parameter.

Both Sync and Async Are Affected

The issue exists in both:

• examples/topic/topic_transactions_async_example.py (async version)
• examples/topic/topic_transactions_example.py (sync version)

Both must be fixed.

---

Proposed Solutions

Solution 1 — Fixed producer_id (Recommended)

Define producer_id once outside the retry callback so it remains stable across retries:

# Async version
async def write_with_tx_example(driver: ydb.aio.Driver, topic: str, message_count: int = 10):
    async with ydb.aio.QuerySessionPool(driver) as session_pool:

        async def callee(tx: ydb.aio.QueryTxContext):
            tx_writer = driver.topic_client.tx_writer(
                tx, topic,
                producer_id="example-producer",
            )

            for i in range(message_count):
                async with await tx.execute(query=f"select {i} as res;") as result_stream:
                    async for result_set in result_stream:
                        message = str(result_set.rows[0]["res"])
                        await tx_writer.write(ydb.TopicWriterMessage(message))
                        print(f"Message {result_set.rows[0]['res']} was written with tx.")

        await session_pool.retry_tx_async(callee)

The same fix should be applied to the sync version in topic_transactions_example.py.

Solution 2 — Use partition_id Instead

If targeting a specific partition:

tx_writer = driver.topic_client.tx_writer(tx, topic, partition_id=0)

This avoids producer_id-based partitioning entirely.

Solution 3 (Additional) — SDK-Level Warning

Consider adding a log warning in TopicClient.tx_writer() when producer_id is not provided, since auto-generated random IDs are almost always wrong for transactional writers called inside retry callbacks. This would help catch the issue proactively in user code.

---

Why This Is High Severity

1. Silent failure mode: No warning or error until the limit is hit; users only discover the problem under production load.
2. Examples are the primary learning path: Many users copy examples directly into production code.
3. Retry amplification: The retry_tx_async/retry_tx_sync pattern (correctly used for fault tolerance) makes the issue worse — every retry generates yet another producer_id.
4. Hard limit: The 100K producer_id limit per 14 days is a server-side constraint that cannot be changed by client configuration.
5. Both API variants affected: Sync and async examples both exhibit the problem.

Recommendation

The minimal fix is updating both example files to pass an explicit producer_id. A stronger fix would also add a log-level warning in the SDK when tx_writer() auto-generates a producer_id.