Skip to content

fix(#4060): extend backoff gate to remote MCP HTTP errors - #4074

Open
aheritier wants to merge 2 commits into
fix/startable-toolset-backofffrom
fix/startable-toolset-backoff-mcp-lsp
Open

fix(#4060): extend backoff gate to remote MCP HTTP errors#4074
aheritier wants to merge 2 commits into
fix/startable-toolset-backofffrom
fix/startable-toolset-backoff-mcp-lsp

Conversation

@aheritier

Copy link
Copy Markdown
Collaborator

🤖 Automated implementer agentthis comment was posted by the implementer bot from Docker Agentic Platform, not by a human developer

Stacked on PR #4062. Base retargets to main once that merges.

Refs #4060 (partial — A2A pacing and LSP crash-loop pacing deferred; see below)

What

Remote MCP servers responding with 503/429/5xx during the initialize handshake previously triggered a fresh connect attempt on every agent turn (the #4060 burst pattern). This PR fixes that by wrapping the HTTP status from the remote server in a *modelerrors.StatusError so the StartableToolSet backoff gate arming logic can pace retries.

Design

Wrap point — enrichConnectError in pkg/tools/mcp/remote.go. The oauthTransport already records the last HTTP error status via logErrorResponse (for any >= 400 response). enrichConnectError previously embedded that status only as text in the error message. It now also calls modelerrors.WrapHTTPError(status, nil, enriched), surfacing the status as a *StatusError in the chain.

What arms the gate. startBackoffRetryable checks for a *modelerrors.StatusError with a retryable HTTP status (429/408/5xx) via errors.As — exactly as it already does for RAG embedding failures. No regex heuristics; no new classification logic in the gate itself.

What does NOT arm (unchanged policy):

  • Local stdio MCP spawn failures (missing binary, connection refused) — never reach enrichConnectError.
  • 4xx client errors (400/401/403) — wrapped in *StatusError for structured access but RetryableHTTPStatus returns false → fail promptly.
  • Auth paths (oauthDeclined, authorizationRequired) — handled by their own early-return paths before the status branch; unaffected.
  • lifecycle.ErrServerUnavailable, ErrTransport, ErrAuthRequired, ErrInitTimeout, ErrSessionMissing — the gate classifier explicitly excludes all of these.

Deferred:

  • A2A pacing: the agent-card resolver does not cleanly expose HTTP status in its error chain; deferred to a follow-up.
  • LSP crash-loop pacing: lifecycle.ErrServerCrashed is produced only inside lspSession.Wait() which flows to the supervisor's internal watcher, not to supervisor.Start(). The gate never sees it via the current error propagation path; deferred.

Changed files

File What changed
pkg/tools/mcp/remote.go enrichConnectError: wrap HTTP-status branch with modelerrors.WrapHTTPError
pkg/tools/startable_backoff.go Updated classifier doc comment: complete excluded-sentinel list (adds ErrInitTimeout, ErrSessionMissing), note deferred LSP crash-loop path
pkg/tools/mcp/remote_test.go 3 new tests: 503→retryable *StatusError; 403→non-retryable *StatusError; network failure→no *StatusError
pkg/tools/startable_backoff_test.go 4 new tests: lifecycle sentinels (ErrServerUnavailable, ErrTransport, ErrAuthRequired) do NOT arm gate; 4xx StatusError does NOT arm gate
docs/tools/mcp/index.md Updated startup-failure note: remote MCP 5xx/429 now paced
docs/tools/lsp/index.md Updated startup-failure note: crash-loop pacing not yet in place; supervisor policy is the throttle

Remote MCP servers returning 503/429/5xx during the initialize handshake
previously triggered a new connect attempt on every agent turn. enrichConnectError
now wraps the HTTP status captured by the oauthTransport in
modelerrors.WrapHTTPError, so retryable responses surface as *StatusError and
arm the StartableToolSet backoff gate exactly as RAG embedding 429s do.

4xx client-error responses (400/401/403) are also wrapped in *StatusError for
structured access but are classified non-retryable, so bad-config and auth
failures still fail promptly without pacing.

Local stdio MCP failures (missing binary, connection refused) never reach
enrichConnectError and are unaffected by this change.

Classifier policy: startBackoffRetryable arms only on *StatusError with a
retryable HTTP status. Deliberately excluded (still fast-retry or fail-promptly):
lifecycle.ErrServerUnavailable (missing binary), lifecycle.ErrTransport
(connection refused / no such host), lifecycle.ErrAuthRequired /
ErrCapabilityMissing, lifecycle.ErrInitTimeout, lifecycle.ErrSessionMissing.
Note: ErrServerCrashed is NOT currently surfaced by supervisor.Start(); LSP
crash-loop pacing is deferred until that propagation path is wired.

Refs #4060 (partial — A2A pacing deferred: agent-card resolver does not expose
HTTP status cleanly; LSP crash-loop pacing also deferred)
@aheritier aheritier changed the title fix: extend backoff gate to remote MCP HTTP errors (#4060 follow-up) fix(#4060): extend backoff gate to remote MCP HTTP errors Aug 28, 2026
@aheritier
aheritier requested a review from docker-agent August 28, 2026 21:24
@aheritier
aheritier marked this pull request as ready for review August 28, 2026 21:24
@aheritier
aheritier requested a review from a team as a code owner August 28, 2026 21:24
@aheritier aheritier added area/docs Documentation changes area/tools For features/issues/fixes related to the usage of built-in and MCP tools status/needs-triage For issues that need to be triaged kind/fix PR fixes a bug (maps to fix:). Use on PRs only. and removed status/needs-triage For issues that need to be triaged labels Aug 28, 2026
Three review findings on PR #4074, all addressed in this commit:

1. (must-fix) enrichConnectError previously gated the *modelerrors.StatusError
   wrap on the extracted server message being non-empty. Many load-balancer
   and rate-limit responses carry an empty body, so a bare 429/503 with no
   payload silently skipped the wrap and the backoff gate never armed —
   defeating the whole point of this PR for exactly the responses it exists
   to pace. Now wraps on status code alone; the enrichment text degrades
   gracefully to '(server responded %d)' when no message is available.

2. (should-fix) Retry-After was discarded: WrapHTTPError was always called
   with resp=nil. oauthTransport now also captures the raw Retry-After
   header value alongside the status/body it already tracks, and
   enrichConnectError builds a minimal *http.Response carrying that header
   so WrapHTTPError parses it onto the StatusError — matching the handling
   already in place for model-provider adapters. Status, message and
   Retry-After are read together as a single lastServerErrorSnapshot() under
   one lock (not three separately-locked accessors), so a caller can never
   pair a status from one response with a Retry-After header captured from
   a different concurrent response on the same transport (this transport's
   RoundTrip can run concurrently for a single logical connect attempt,
   e.g. a standalone SSE probe alongside the initialize call).

3. (should-fix) Added an end-to-end regression test that drives a real
   *mcp.Toolset (built via NewRemoteToolset, exactly as production wiring
   does) through tools.StartableToolSet.TryStart against a mock 503/403
   server, proving the whole chain (enrichConnectError -> Toolset.Start ->
   supervisor.Start -> the backoff gate) stays intact end to end, not just
   the enrichConnectError unit boundary.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/docs Documentation changes area/tools For features/issues/fixes related to the usage of built-in and MCP tools kind/fix PR fixes a bug (maps to fix:). Use on PRs only.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant