Skip to content

fix : idle and max connection duration timeouts to Envoy HTTP listeners - #2403

Merged
davidbirdsong merged 1 commit into
developfrom
davidbirdsong/envoy-idle-conn-tuning
Aug 27, 2026
Merged

fix : idle and max connection duration timeouts to Envoy HTTP listeners#2403
davidbirdsong merged 1 commit into
developfrom
davidbirdsong/envoy-idle-conn-tuning

Conversation

@davidbirdsong

Copy link
Copy Markdown
Contributor

What kind of change does this PR introduce?

Bug fix / reliability fix — adds connection-reclaim timeouts to Envoy's HTTP listeners.

What is the current behavior?

Envoy's HTTP connection manager has no idle_timeout or max_connection_duration configured, so downstream connections accumulate indefinitely once accepted — nothing ever proactively closes an idle or long-lived connection.

The only enforcement on connection count is envoy.resource_monitors.global_downstream_max_connections, which is self-enforcing (it doesn't require a paired overload action): once the process-wide active connection count reaches the configured cap, Envoy rejects/closes new connections at accept time, before the filter chain — and therefore before the TLS handshake — ever runs. From the client's (Cloudflare's) perspective this surfaces as a 525 (SSL handshake failed), with no HTTP response and no access log entry, since the only configured access log lives inside the HTTP connection manager and is never reached.

With no reclaim mechanism, connection count only trends upward over time. Any sustained increase in concurrency — more edge/colo connections, more traffic, or requests simply taking longer to complete — directly erodes headroom under whatever cap is configured, with no way to recover it without repeatedly raising the cap.

What is the new behavior?

Adds common_http_protocol_options to the shared HTTP connection manager filter chain in both lds.yaml and lds.supabase.yaml:

common_http_protocol_options:
  idle_timeout: 300s
  max_connection_duration: 600s
  • idle_timeout: 300s — closes a connection once it has zero active streams for 5 minutes. Only reclaims genuinely idle connections; never touches one with an in-flight request.
  • max_connection_duration: 600s — force-cycles every connection at 10 minutes regardless of activity, so even a connection that's used just often enough to never go idle still gets recycled on a bounded schedule.

Both settings trigger Envoy's normal graceful drain (an HTTP/1.1 Connection: close on the next response, or an HTTP/2 GOAWAY) rather than an abrupt reset. This is intentional — the goal is for Envoy to finish returning an HTTP response wherever possible and only close the TCP connection afterward, not sever a connection out from under in-flight work.

Additional context

This was hard-won during inc-752-latency-issues-and-525s-08-27-major. Mid-incident, the on-box connection count was observed climbing to and pressing against the (then) 30,000-connection cap under sustained load, confirmed live via the Envoy admin stats endpoint while the cap was raised in steps (30k → 40k → 50k) as a stopgap. With no idle/duration reclaim in place, that climb had no ceiling of its own — raising the cap only bought time rather than addressing the underlying growth.

The values here (300s idle / 600s max duration) are intentionally moderately conservative for a first rollout of a previously-untested mechanism at this scale: tight enough to meaningfully bound connection growth, not so tight that we risk cycling connections faster than is comfortable under normal traffic. Tighter values can be considered once we've observed this config's behavior in production for longer.

@davidbirdsong

Copy link
Copy Markdown
Contributor Author

This change is part of the following stack:

Change managed by git-spice.

@davidbirdsong
davidbirdsong requested review from a team as code owners August 27, 2026 20:09
Connections had no reclaim mechanism, so the downstream pool grew unbounded and eroded headroom under the global connection cap, contributing to load-related handshake failures. Bounds connections to a 5min idle timeout and 10min max duration to force periodic turnover.
@davidbirdsong
davidbirdsong force-pushed the davidbirdsong/envoy-idle-conn-tuning branch from 054dffc to 7631402 Compare August 27, 2026 21:57
@davidbirdsong
davidbirdsong added this pull request to the merge queue Aug 27, 2026
Merged via the queue into develop with commit 7dd761d Aug 27, 2026
63 of 73 checks passed
@davidbirdsong
davidbirdsong deleted the davidbirdsong/envoy-idle-conn-tuning branch August 27, 2026 23:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants