fix(#4060): extend backoff gate to remote MCP HTTP errors - #4074
Open
aheritier wants to merge 2 commits into
Open
fix(#4060): extend backoff gate to remote MCP HTTP errors#4074aheritier wants to merge 2 commits into
aheritier wants to merge 2 commits into
Conversation
Remote MCP servers returning 503/429/5xx during the initialize handshake previously triggered a new connect attempt on every agent turn. enrichConnectError now wraps the HTTP status captured by the oauthTransport in modelerrors.WrapHTTPError, so retryable responses surface as *StatusError and arm the StartableToolSet backoff gate exactly as RAG embedding 429s do. 4xx client-error responses (400/401/403) are also wrapped in *StatusError for structured access but are classified non-retryable, so bad-config and auth failures still fail promptly without pacing. Local stdio MCP failures (missing binary, connection refused) never reach enrichConnectError and are unaffected by this change. Classifier policy: startBackoffRetryable arms only on *StatusError with a retryable HTTP status. Deliberately excluded (still fast-retry or fail-promptly): lifecycle.ErrServerUnavailable (missing binary), lifecycle.ErrTransport (connection refused / no such host), lifecycle.ErrAuthRequired / ErrCapabilityMissing, lifecycle.ErrInitTimeout, lifecycle.ErrSessionMissing. Note: ErrServerCrashed is NOT currently surfaced by supervisor.Start(); LSP crash-loop pacing is deferred until that propagation path is wired. Refs #4060 (partial — A2A pacing deferred: agent-card resolver does not expose HTTP status cleanly; LSP crash-loop pacing also deferred)
aheritier
marked this pull request as ready for review
August 28, 2026 21:24
Three review findings on PR #4074, all addressed in this commit: 1. (must-fix) enrichConnectError previously gated the *modelerrors.StatusError wrap on the extracted server message being non-empty. Many load-balancer and rate-limit responses carry an empty body, so a bare 429/503 with no payload silently skipped the wrap and the backoff gate never armed — defeating the whole point of this PR for exactly the responses it exists to pace. Now wraps on status code alone; the enrichment text degrades gracefully to '(server responded %d)' when no message is available. 2. (should-fix) Retry-After was discarded: WrapHTTPError was always called with resp=nil. oauthTransport now also captures the raw Retry-After header value alongside the status/body it already tracks, and enrichConnectError builds a minimal *http.Response carrying that header so WrapHTTPError parses it onto the StatusError — matching the handling already in place for model-provider adapters. Status, message and Retry-After are read together as a single lastServerErrorSnapshot() under one lock (not three separately-locked accessors), so a caller can never pair a status from one response with a Retry-After header captured from a different concurrent response on the same transport (this transport's RoundTrip can run concurrently for a single logical connect attempt, e.g. a standalone SSE probe alongside the initialize call). 3. (should-fix) Added an end-to-end regression test that drives a real *mcp.Toolset (built via NewRemoteToolset, exactly as production wiring does) through tools.StartableToolSet.TryStart against a mock 503/403 server, proving the whole chain (enrichConnectError -> Toolset.Start -> supervisor.Start -> the backoff gate) stays intact end to end, not just the enrichConnectError unit boundary.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on PR #4062. Base retargets to
mainonce that merges.Refs #4060 (partial — A2A pacing and LSP crash-loop pacing deferred; see below)
What
Remote MCP servers responding with 503/429/5xx during the initialize handshake previously triggered a fresh connect attempt on every agent turn (the #4060 burst pattern). This PR fixes that by wrapping the HTTP status from the remote server in a
*modelerrors.StatusErrorso theStartableToolSetbackoff gate arming logic can pace retries.Design
Wrap point —
enrichConnectErrorinpkg/tools/mcp/remote.go. TheoauthTransportalready records the last HTTP error status vialogErrorResponse(for any>= 400response).enrichConnectErrorpreviously embedded that status only as text in the error message. It now also callsmodelerrors.WrapHTTPError(status, nil, enriched), surfacing the status as a*StatusErrorin the chain.What arms the gate.
startBackoffRetryablechecks for a*modelerrors.StatusErrorwith a retryable HTTP status (429/408/5xx) viaerrors.As— exactly as it already does for RAG embedding failures. No regex heuristics; no new classification logic in the gate itself.What does NOT arm (unchanged policy):
enrichConnectError.*StatusErrorfor structured access butRetryableHTTPStatusreturns false → fail promptly.oauthDeclined,authorizationRequired) — handled by their own early-return paths before the status branch; unaffected.lifecycle.ErrServerUnavailable,ErrTransport,ErrAuthRequired,ErrInitTimeout,ErrSessionMissing— the gate classifier explicitly excludes all of these.Deferred:
lifecycle.ErrServerCrashedis produced only insidelspSession.Wait()which flows to the supervisor's internal watcher, not tosupervisor.Start(). The gate never sees it via the current error propagation path; deferred.Changed files
pkg/tools/mcp/remote.goenrichConnectError: wrap HTTP-status branch withmodelerrors.WrapHTTPErrorpkg/tools/startable_backoff.goErrInitTimeout,ErrSessionMissing), note deferred LSP crash-loop pathpkg/tools/mcp/remote_test.go*StatusError; 403→non-retryable*StatusError; network failure→no*StatusErrorpkg/tools/startable_backoff_test.godocs/tools/mcp/index.mddocs/tools/lsp/index.md