[Bug]: Upstream reconnect has no backoff — supervisor re-dials failed servers every 30s forever, hammering dead remotes (~9k req/day)

[antongulin]

What happened?

The supervisor reconnects failed upstreams every 30 seconds forever, with no backoff, no retry cap, and no distinction for auth failures. Each attempt fires ~3 HTTP requests at the upstream (plain initialize, a retry, then the OAuth-strategy initialize), so a single dead remote server generates ~8,600 requests/day indefinitely.

This is not theoretical — it burned a real vendor relationship twice:

• Late July 2026: a remote upstream (pressranger.com/mcp) started returning 401 after a static bearer token expired. mcpproxy hammered it (~95k req/day across our agents' instances) until the vendor's server alarms tripped and they paused our account, emailing us to ask what was wrong.
• Aug 6–20, 2026: same upstream, token revoked by the vendor — the loop ran again at ~8–9k req/day for two weeks before we noticed. A failed upstream is completely silent from the operator's side (agents just don't see its tools); the entire cost of the retry loop lands on the remote server.

The ironic part: on every failed attempt mcpproxy correctly diagnoses the situation and logs OAuth authentication required for X: login available via Web UI... — a state that cannot be fixed by retrying — and then retries 30 seconds later anyway.

Where

Confirmed present on main @ 52146a4 (post-v0.59.0), same behavior observed on v0.51.0:

• internal/runtime/supervisor/supervisor.go — reconciliationLoop runs reconcile on a fixed time.NewTicker(30 * time.Second).
• computeReconcilePlan plans ActionConnect for any server with Enabled && !actuallyConnected (unless quarantined or user-logged-out). It never consults the managed client's RetryCount, last error time, or error class — so an upstream in Error state is re-dialed on every tick, forever.
• The per-client retry_count already tracked in ConnectionInfo/stateview is only used for display, never for pacing.

Backoff logic already exists elsewhere in the codebase (tray API client, registry httpclient, OAuth refresh manager) — the upstream reconnect path is the one place it's missing, and it's the path that hits third-party servers.

Expected behavior

1. Exponential backoff per server on consecutive connect failures: e.g. 30s → 1m → 2m → ... capped at 15–30min, reset on success, config change, or explicit user action (enable/restart/login). The existing retry_count in ConnectionInfo is already most of the state needed; computeReconcilePlan could skip ActionConnect while now < lastFailure + backoff(retryCount).
2. Auth-required failures (401 + OAuth-required classification) should park the server in an "awaiting login" state rather than retry at all — retrying cannot succeed until a human completes the OAuth flow, and mcpproxy already emits exactly that log line. reconnect_on_use / manual login can still trigger a fresh attempt.
3. Honor Retry-After when the upstream answers 429 (the OAuth resource-detect path already parses hints; connect retries should too).

Steps to reproduce

1. Add a streamable-http upstream that answers 401 to initialize (any endpoint with a revoked/expired bearer token in headers), "enabled": true, no valid OAuth token stored.
2. Run mcpproxy serve headless and tail logs/server-<name>.log.
3. Observe the identical attempt cycle every 30s with no growth in interval: Starting connection attempt → 2× MCP initialize JSON-RPC call failed ... authorization required → OAuth strategy failure → Connection failed → repeat. tcpdump/upstream access logs show ~3 requests per cycle, ~8.6k/day.

MCPProxy version

v0.51.0 linux/amd64 (incident); bug confirmed by code inspection on main @ 52146a4

Operating system

Linux

Relevant logs

2026-08-20T18:15:52.759Z | INFO  | managed/client.go:239 | Starting connection attempt | {"server": "press-ranger", "transport": "http", "url": "https://pressranger.com/mcp", "protocol": "http"}
2026-08-20T18:15:53.809Z | ERROR | core/connection_http.go:191 | MCP initialize JSON-RPC call failed | {"server": "press-ranger", "error": "transport error: authorization required"}
2026-08-20T18:15:54.523Z | ERROR | core/connection_http.go:219 | MCP initialize JSON-RPC call failed | {"server": "press-ranger", "error": "transport error: authorization required"}
2026-08-20T18:15:58.213Z | ERROR | core/connection_oauth.go:426 | MCP initialize JSON-RPC call failed | {"server": "press-ranger", "error": "transport error: failed to send request: no valid token available, authorization required"}
2026-08-20T18:15:58.214Z | ERROR | managed/client.go:239 | Connection failed | {"server": "press-ranger", "error": "OAuth authentication required for press-ranger: login available via Web UI, system tray menu, or 'mcpproxy auth login' CLI command"}
2026-08-20T18:16:22.758Z | INFO  | managed/client.go:239 | Starting connection attempt | ...   <- exactly 30s later, forever

(Identical cycle fills every rotated server-press-ranger.log from Aug 6 to Aug 20 — ~40k attempts, ~120k HTTP requests, all against an endpoint that could never succeed without human OAuth login.)

[antongulin]

Root cause pinned down while writing the fix: the client-level exponential backoff and MaxConnectionRetries give-up already exist (StateManager.ShouldRetry), and the ConnectAll path in internal/upstream/manager.go correctly gates on them — but the supervisor's periodic per-server reconcile path (computeReconcilePlan → ActionConnect → ActorPoolSimple.ConnectServer → client.Connect()) bypasses that policy entirely, so the 30s ticker re-dials failed servers forever. StatePendingAuth servers are re-dialed too, even though the retry can't succeed until the user completes the OAuth login. Fix in #1014.