fix : idle and max connection duration timeouts to Envoy HTTP listeners - #2403
Merged
Merged
Conversation
Contributor
Author
|
This change is part of the following stack: Change managed by git-spice. |
Connections had no reclaim mechanism, so the downstream pool grew unbounded and eroded headroom under the global connection cap, contributing to load-related handshake failures. Bounds connections to a 5min idle timeout and 10min max duration to force periodic turnover.
davidbirdsong
force-pushed
the
davidbirdsong/envoy-idle-conn-tuning
branch
from
August 27, 2026 21:57
054dffc to
7631402
Compare
sprsquish
approved these changes
Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What kind of change does this PR introduce?
Bug fix / reliability fix — adds connection-reclaim timeouts to Envoy's HTTP listeners.
What is the current behavior?
Envoy's HTTP connection manager has no
idle_timeoutormax_connection_durationconfigured, so downstream connections accumulate indefinitely once accepted — nothing ever proactively closes an idle or long-lived connection.The only enforcement on connection count is
envoy.resource_monitors.global_downstream_max_connections, which is self-enforcing (it doesn't require a paired overload action): once the process-wide active connection count reaches the configured cap, Envoy rejects/closes new connections at accept time, before the filter chain — and therefore before the TLS handshake — ever runs. From the client's (Cloudflare's) perspective this surfaces as a 525 (SSL handshake failed), with no HTTP response and no access log entry, since the only configured access log lives inside the HTTP connection manager and is never reached.With no reclaim mechanism, connection count only trends upward over time. Any sustained increase in concurrency — more edge/colo connections, more traffic, or requests simply taking longer to complete — directly erodes headroom under whatever cap is configured, with no way to recover it without repeatedly raising the cap.
What is the new behavior?
Adds
common_http_protocol_optionsto the shared HTTP connection manager filter chain in bothlds.yamlandlds.supabase.yaml:idle_timeout: 300s— closes a connection once it has zero active streams for 5 minutes. Only reclaims genuinely idle connections; never touches one with an in-flight request.max_connection_duration: 600s— force-cycles every connection at 10 minutes regardless of activity, so even a connection that's used just often enough to never go idle still gets recycled on a bounded schedule.Both settings trigger Envoy's normal graceful drain (an HTTP/1.1
Connection: closeon the next response, or an HTTP/2GOAWAY) rather than an abrupt reset. This is intentional — the goal is for Envoy to finish returning an HTTP response wherever possible and only close the TCP connection afterward, not sever a connection out from under in-flight work.Additional context
This was hard-won during
inc-752-latency-issues-and-525s-08-27-major. Mid-incident, the on-box connection count was observed climbing to and pressing against the (then) 30,000-connection cap under sustained load, confirmed live via the Envoy admin stats endpoint while the cap was raised in steps (30k → 40k → 50k) as a stopgap. With no idle/duration reclaim in place, that climb had no ceiling of its own — raising the cap only bought time rather than addressing the underlying growth.The values here (
300sidle /600smax duration) are intentionally moderately conservative for a first rollout of a previously-untested mechanism at this scale: tight enough to meaningfully bound connection growth, not so tight that we risk cycling connections faster than is comfortable under normal traffic. Tighter values can be considered once we've observed this config's behavior in production for longer.