Add opt-in connection pool with lease semantics to `AWSCRTHTTPClient`

[bilardi]

Summary

AWSCRTHTTPClient._get_connection() (packages/smithy-http/src/smithy_http/aio/crt.py:224) keeps a single shared HTTP connection per (scheme, host, port). The existing TODO at that location (# TODO: Use CRT connection pooling instead of this basic kind) acknowledges the limitation but is currently untracked.

In addition, the deepcopy(self._config) performed by every smithy-generated operation invokes AWSCRTHTTPClient.__deepcopy__, which returns a fresh instance with an empty _connections cache (the risk was flagged in PR #355: "I'm a bit concerned about not sharing the connection pool"). The effect is measurable: every bidirectional operation pays a full TLS + ALPN + HTTP/2 cold start, adding ~60 ms to median latency vs an implementation that reuses connections across operations.

This issue proposes an opt-in connection pool with lease semantics, attached to _AWSCRTEventLoop so it survives the deepcopy. The pool solves the per-connection stream limit for bidirectional streaming (Polly, Transcribe, Bedrock Nova Sonic) without breaking any existing API.

Empirical impact

Measured on a realtime fan-out workload (1 speaker -> 2 concurrent listener languages, Polly bidirectional streaming, us-west-2 region):

Implementation	P50	Sample size
aws-sdk-polly 0.6.0 vanilla	350 ms	30 (3 run)
aws-sdk-polly 0.6.0 + this proposal	290 ms	40 (4 run)

The PR recovers ~60 ms of median latency by enabling warm connection reuse across operations. Full historical comparison across 4 benchmarks (including the reference hand-rolled package) and reproduction commands are in PR_TO_SMITHY.md § Full evolution table.

Proposed design

Opt-in via a new ConnectionPoolConfig dataclass on AWSCRTHTTPClientConfig. Default None preserves current behavior; setting the field activates a bounded pool with lease semantics per (scheme, host, port).

from smithy_http.aio.crt import AWSCRTHTTPClient, AWSCRTHTTPClientConfig, ConnectionPoolConfig

# Default: unchanged behavior, no pool
transport = AWSCRTHTTPClient()

# Opt-in with default cap (8 connections per host)
transport = AWSCRTHTTPClient(
    client_config=AWSCRTHTTPClientConfig(
        connection_pool=ConnectionPoolConfig()
    )
)

The ConnectionPool is attached lazily to _AWSCRTEventLoop.pool on first send when connection_pool is set. The eventloop is already shared by __deepcopy__ of AWSCRTHTTPClient, so the pool survives the deepcopy. Lease ownership transfers to AWSCRTHTTPResponse and is released in chunks()'s finally block, tying lease lifetime to stream lifetime. Full mechanism analysis: PR_TO_SMITHY.md § Detailed mechanism.

Implementation

Branch: bilardi/smithy-python @ add-awscrt-connection-pool

Single file modified: packages/smithy-http/src/smithy_http/aio/crt.py, +248 / -24 lines.

Quality checks: make check-py clean (ruff + ruff format + pyright strict mode, 0 errors 0 warnings, Python 3.12 target).

Backwards compatible: connection_pool default None, existing _send_via_cache path untouched semantically. New public symbols: ConnectionPoolConfig and ConnectionPool (the _Lease helper is private).

Use cases affected

• Polly bidirectional fan-out (1 speaker -> N target languages in multilingual broadcast)
• Transcribe streaming multi-channel (concurrent input streams)
• Bedrock Runtime Nova Sonic with concurrent sessions (per await_output() hangs forever on HTTP 429 (ThrottlingException) for bidirectional streams #657)
• Generally any AWS service over smithy-http[awscrt] that uses bidirectional streaming with per-connection stream limits, or HTTP/2 fan-out

Open questions before sending the PR

1. Default max_connections_per_host: chosen 8 based on the amazon-polly-streaming reference implementation validated in production with multilingual fan-out workloads. Open to a different default.

2. Stale connection detection beyond is_open(): 2 of 4 measurement runs showed a mid-run latency spike. Three candidate causes: Polly server-side jitter, stale-connection in the pool, network spike. Plan to instrument connection_age_ms to disambiguate; can address before merge or as a separate issue.

3. Tests in upstream style: packages/smithy-http/tests/unit/aio/ doesn't currently include CRT tests. Pure-unit tests for ConnectionPool logic can be written without CRT by mocking the connection factory. Prefer unit, integration, or both?

4. Naming and module structure: file already exports ConnectionPoolKey / ConnectionPoolDict aliases. Adding ConnectionPoolConfig / ConnectionPool / _Lease to the same module is consistent. Open to extract to a separate _pool.py module if preferred.

References

• TODO at packages/smithy-http/src/smithy_http/aio/crt.py:224 (as of develop 2026-05-22)
• Related: Safely copy crt clients #355 (Safely copy crt clients) — introduced the __deepcopy__ that creates the warm-reuse loss
• Related: await_output() hangs forever on HTTP 429 (ThrottlingException) for bidirectional streams #657 (await_output() hangs forever on HTTP 429) — same use case (Nova Sonic bidirectional fan-out)
• Full operational documentation (branches, benchmarks, mechanism analysis): PR_TO_SMITHY.md

License: contribution under Apache-2.0, consistent with the repository.

[jonathan343]

Hey @bilardi,

We really appreciate the time and effort you've put into this issue and POC!

As you've discovered through our TODO, this is known gap and something we're interested in adding. The performance impact demonstrated by your POC is pretty significant which is very promising.

We want to make sure this is done right the first time, so it will require dedicated investigation time on our end to help facilitate properly. We'll get back as soon as someone is able to spend time on it.

Thanks again!