fix: add bounded jittered backoff to StartableToolSet retry path

docs/tools/lsp/index.md

@@ -197,6 +197,8 @@ Available Capabilities:
 
 LSP toolsets are managed by the same supervisor as MCP toolsets, so a crashed `gopls` (or any other language server) is reconnected automatically with exponential backoff. Use the [`lifecycle`](../../configuration/tools/index.md#toolset-lifecycle) block to tune the policy per toolset — for example, mark `gopls` as `strict` if your CI flow requires it to be available, or use `/toolset-restart gopls` from the TUI to force a reconnect when the server gets stuck.
 
+**Startup failure behaviour:** local process failures such as a missing binary or server-unavailable error fail fast — each turn retries immediately with no artificial delay. The backoff described in [Indexing failures, retries and backoff](../rag/index.md#indexing-failures-retries-and-backoff) applies only to structured HTTP rate-limit and server-error responses from model providers, which local LSP process errors are not.
+
 ```yaml
 toolsets:
   - type: lsp

docs/tools/mcp/index.md

@@ -243,6 +243,8 @@ See [Per-Toolset Model Routing](../../configuration/tools/index.md#per-toolset-m
 
 Local stdio and remote MCP servers are supervised: crashed servers reconnect automatically with exponential backoff. **Remote** MCP servers (Streamable HTTP / SSE) also reconnect after idle/clean connection closes — services like Notion and Linear periodically close idle connections, and Docker Agent reconnects transparently. Tune the policy with the `lifecycle` block:
 
+**Startup failure behaviour:** local process failures (missing binary, connection refused, authentication error) fail fast — each turn retries immediately with no artificial delay added. Only structured HTTP rate-limit (HTTP 429) and server error (HTTP 5xx) responses from a backing model provider trigger a backoff window. See [Indexing failures, retries and backoff](../rag/index.md#indexing-failures-retries-and-backoff) for the full policy.
+
 ```yaml
 toolsets:
   - type: mcp

docs/tools/rag/index.md

@@ -174,6 +174,86 @@ chunking:
 >
 > Currently supports Go (`.go`) files. More languages will be added. Falls back to plain text chunking for unsupported file types.
 
+## Indexing failures, retries and backoff
+
+When a knowledge-base fails to start — because the embedding provider is rate-limiting
+your requests or returning a transient server error — Docker Agent spaces out retry
+attempts with bounded exponential backoff instead of hammering the provider on
+every agent turn.
+
+### What triggers backoff
+
+Backoff applies only to **structured HTTP rate-limit (429), request timeout (408), and server error (5xx)**
+responses from the embedding or model provider. These are the signals that mean
+"slow down", and spacing them out gives the provider room to recover:
+
+| Failure kind | Behaviour |
+|---|---|
+| HTTP 429 (rate limit) | Backoff: next attempt delayed |
+| HTTP 5xx / overload (500, 502, 503, 504, 529) | Backoff: next attempt delayed |
+| HTTP 408 (request timeout) | Backoff: next attempt delayed |
+| Bad config, invalid model, wrong API key (other 4xx errors) | Fail fast: retried every turn with no added delay |
+| Context cancellation or agent shutdown | Immediate: no delay |
+
+### Retry policy and parameters
+
+The backoff is **bounded exponential with additive jitter**:
+
+- **Base delay**: 15 seconds
+- **Maximum delay**: 5 minutes
+- **Growth**: doubles after each consecutive retryable failure (15s → 30s → 1m → 2m → 4m → 5m)
+- **Jitter**: each wait is a random value in `[nominal, 1.2×nominal]` (additive 0–20%)
+  so concurrent knowledge-base sources spread their retries and avoid
+  hammering the provider together
+
+The gate is a lightweight wall-clock check — it creates no background threads or
+timers. A Stop command or agent shutdown takes effect immediately regardless of
+how much of the backoff window remains.
+
+### Operational impact
+
+**Before**: a rate-limited knowledge base was re-indexed on every agent turn —
+`max_indexing_concurrency × max_embedding_concurrency` concurrent provider calls
+could relaunch within milliseconds, easily tripping rate limits for both the
+knowledge base and the agent's own model calls.
+
+**After**: retries are spaced out and jittered so the provider has room to recover
+before the next attempt. The agent continues working with any other toolsets that
+are not affected.
+
+### What you will see
+
+- Docker Agent logs a single warning when a knowledge base first fails to start.
+  Repeated failures in between are logged at debug level only, so you are not
+  flooded with alerts on every turn. Recovery is intentionally silent — the
+  tool appearing in the agent's tool list is the signal that indexing succeeded.
+- The knowledge-base tool does not appear in the agent's tool list until indexing
+  succeeds. A successful start is silent — the tool is listed and the agent uses it.
+
+### Troubleshooting repeated 429 errors
+
+If you see persistent `429` errors in the logs:
+
+1. **Check provider rate limits.** Your embedding API key may have a low requests-per-minute
+   quota. Upgrading the plan or using a different API key can help.
+2. **Reduce concurrency.** The chunked-embeddings and semantic-embeddings strategies
+   accept `max_indexing_concurrency` (default `3`) and `max_embedding_concurrency`
+   (default `3`) parameters. Lowering these reduces simultaneous requests:
+
+   ```yaml
+   rag:
+     name: docs
+     path: ./knowledge-base
+     strategy:
+       type: chunked-embeddings
+       params:
+         max_indexing_concurrency: 1
+         max_embedding_concurrency: 1
+   ```
+
+3. **Use a model with a higher quota.** Some providers offer higher rate limits on
+   specific embedding model tiers.
+
 ## Debugging RAG
 
 Enable debug logging to see retrieval details:

pkg/agent/agent.go

@@ -645,10 +645,10 @@ func (a *Agent) ensureToolSetsAreStarted(ctx context.Context) {
 			continue
 		}
 		if toolSet.ShouldReportFailure() {
-			slog.WarnContext(ctx, "Toolset start failed; will retry on next turn", "agent", a.Name(), "toolset", desc, "error", err)
+			slog.WarnContext(ctx, "Toolset start failed; will retry (backoff may apply)", "agent", a.Name(), "toolset", desc, "error", err)
 			a.AddToolWarning(fmt.Sprintf("%s start failed: %v", desc, err))
 		} else {
-			slog.DebugContext(ctx, "Toolset still unavailable; retrying next turn", "agent", a.Name(), "toolset", desc, "error", err)
+			slog.DebugContext(ctx, "Toolset still unavailable; will retry (backoff may apply)", "agent", a.Name(), "toolset", desc, "error", err)
 		}
 	}
 }

pkg/model/provider/dmr/embed.go

@@ -17,6 +17,7 @@ import (
 	"github.com/openai/openai-go/v3"
 
 	"github.com/docker/docker-agent/pkg/model/provider/base"
+	"github.com/docker/docker-agent/pkg/model/provider/oaistream"
 	"github.com/docker/docker-agent/pkg/rag/types"
 )
 
@@ -50,7 +51,7 @@ func (c *Client) CreateBatchEmbedding(ctx context.Context, texts []string) (*bas
 		Model: c.ModelConfig.Model,
 	})
 	if err != nil {
-		return nil, fmt.Errorf("failed to create embeddings: %w", err)
+		return nil, fmt.Errorf("failed to create embeddings: %w", oaistream.WrapOpenAIError(err))
 	}
 
 	if len(response.Data) != len(texts) {

pkg/model/provider/openai/client.go

@@ -1165,7 +1165,7 @@ func (c *Client) CreateBatchEmbedding(ctx context.Context, texts []string) (*bas
 	response, err := client.Embeddings.New(ctx, params)
 	if err != nil {
 		slog.ErrorContext(ctx, "OpenAI batch embedding request failed", "error", err)
-		return nil, fmt.Errorf("failed to create batch embeddings: %w", err)
+		return nil, fmt.Errorf("failed to create batch embeddings: %w", oaistream.WrapOpenAIError(err))
 	}
 
 	if len(response.Data) != len(texts) {

pkg/modelerrors/modelerrors.go

@@ -781,3 +781,18 @@ func scalarString(v any) string {
 		return fmt.Sprint(v)
 	}
 }
+
+// RetryableHTTPStatus reports whether err contains an HTTP status that warrants
+// backoff (429 rate-limit, 408 request-timeout, or a 5xx server error). It
+// checks for a *StatusError in the chain first; if none is found it falls back
+// to matching \b([45]\d{2})\b in the error message, which can produce false
+// positives for port numbers or similar numeric patterns. Callers that need
+// strict StatusError-only classification should pre-filter with errors.As.
+// No context-error handling is performed.
+func RetryableHTTPStatus(err error) bool {
+	code := extractHTTPStatusCode(err)
+	if code == 0 {
+		return false
+	}
+	return code == http.StatusTooManyRequests || isRetryableStatusCode(code)
+}

pkg/modelerrors/modelerrors_test.go

@@ -944,3 +944,57 @@ func TestScalarStringEdgeCases(t *testing.T) {
 		})
 	}
 }
+
+func TestRetryableHTTPStatus(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name     string
+		err      error
+		expected bool
+	}{
+		// Retryable HTTP codes via *StatusError.
+		{name: "429 rate-limit StatusError", err: &StatusError{StatusCode: 429, Err: errors.New("rate limited")}, expected: true},
+		{name: "408 request timeout StatusError", err: &StatusError{StatusCode: 408, Err: errors.New("timeout")}, expected: true},
+		{name: "500 server error StatusError", err: &StatusError{StatusCode: 500, Err: errors.New("internal server error")}, expected: true},
+		{name: "503 unavailable StatusError", err: &StatusError{StatusCode: 503, Err: errors.New("service unavailable")}, expected: true},
+		{name: "529 overloaded StatusError", err: &StatusError{StatusCode: 529, Err: errors.New("overloaded")}, expected: true},
+		// Non-retryable HTTP codes via *StatusError.
+		{name: "400 bad request StatusError", err: &StatusError{StatusCode: 400, Err: errors.New("bad request")}, expected: false},
+		{name: "401 unauthorized StatusError", err: &StatusError{StatusCode: 401, Err: errors.New("unauthorized")}, expected: false},
+		{name: "403 forbidden StatusError", err: &StatusError{StatusCode: 403, Err: errors.New("forbidden")}, expected: false},
+		{name: "404 not found StatusError", err: &StatusError{StatusCode: 404, Err: errors.New("not found")}, expected: false},
+		// Plain string errors without HTTP codes: must stay non-retryable so
+		// MCP/LSP "connection refused" errors are not paced.
+		{name: "connection refused", err: errors.New("connection refused: dial tcp 127.0.0.1:9999"), expected: false},
+		{name: "no such host", err: errors.New("no such host: example.invalid"), expected: false},
+		// Plain text that DOES contain a retryable HTTP code: the regex fallback
+		// in extractHTTPStatusCode finds it. NOTE: RetryableHTTPStatus returns
+		// true here, but startBackoffRetryable (the toolset gate classifier)
+		// requires a *StatusError and returns false for the same input — the
+		// narrowing is deliberate to prevent port-number/chunk-count false positives.
+		{name: "503 in plain text", err: errors.New("upstream: 503 Service Unavailable"), expected: true},
+		{name: "429 in plain text", err: errors.New("provider said: 429 Too Many Requests"), expected: true},
+		// HTTP-status precedence: a StatusError{429} wrapped alongside
+		// context.DeadlineExceeded must return true — the HTTP signal wins.
+		{
+			name: "429 StatusError + DeadlineExceeded in chain",
+			err: fmt.Errorf("start budget exceeded: %w, provider said: %w",
+				context.DeadlineExceeded,
+				&StatusError{StatusCode: 429, Err: errors.New("rate limited")}),
+			expected: true,
+		},
+		// Bare context errors: no HTTP code, must return false.
+		{name: "bare DeadlineExceeded", err: context.DeadlineExceeded, expected: false},
+		{name: "bare Canceled", err: context.Canceled, expected: false},
+		{name: "nil error", err: nil, expected: false},
+	}
+
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+			got := RetryableHTTPStatus(tc.err)
+			assert.Equal(t, tc.expected, got)
+		})
+	}
+}

pkg/tools/builtin/rag/rag_backoff_test.go

@@ -0,0 +1,129 @@
+package rag
+
+// Regression tests proving that the real RAG ToolSet, when wrapped in
+// tools.NewStartable, correctly engages the backoff gate for structured
+// HTTP-status failures (e.g. a 429 rate-limit from the embedding provider)
+// and does NOT throttle plain (non-StatusError) failures.
+//
+// Gate enforcement is in tryStartLocked (TryStart paths only); all gate
+// assertions here use TryStart, not Start. A frozen fake clock supplied via
+// WithStartRetryClock gives deterministic windows without needing synctest.
+
+import (
+	"context"
+	"fmt"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/docker/docker-agent/pkg/modelerrors"
+	"github.com/docker/docker-agent/pkg/rag"
+	"github.com/docker/docker-agent/pkg/rag/strategy"
+	"github.com/docker/docker-agent/pkg/tools"
+)
+
+// countingStatusErrStrategy is a mockStrategy variant whose Initialize
+// returns a *modelerrors.StatusError and counts every invocation.
+type countingStatusErrStrategy struct {
+	mockStrategy
+
+	statusCode int
+	calls      atomic.Int32
+}
+
+func (s *countingStatusErrStrategy) Initialize(_ context.Context, _ []string, _ strategy.ChunkingConfig) error {
+	s.calls.Add(1)
+	return &modelerrors.StatusError{
+		StatusCode: s.statusCode,
+		Err:        fmt.Errorf("HTTP %d error from provider", s.statusCode),
+	}
+}
+
+// countingPlainErrStrategy always returns a plain (non-StatusError) error
+// so the backoff gate must never fire.
+type countingPlainErrStrategy struct {
+	mockStrategy
+
+	calls atomic.Int32
+}
+
+func (s *countingPlainErrStrategy) Initialize(_ context.Context, _ []string, _ strategy.ChunkingConfig) error {
+	s.calls.Add(1)
+	return assert.AnError
+}
+
+func buildRAGToolSet(t *testing.T, impl strategy.Strategy) *ToolSet {
+	t.Helper()
+	cfg := rag.Config{
+		StrategyConfigs: []strategy.Config{
+			{Name: "test-strategy", Strategy: impl},
+		},
+	}
+	mgr, err := rag.New(t.Context(), "test-rag", cfg, nil)
+	require.NoError(t, err)
+	return &ToolSet{manager: mgr, toolName: "test-rag"}
+}
+
+// TestRAGStartableBackoff_StatusErrorEngagesGate proves that the real RAG
+// ToolSet, when it fails with a *modelerrors.StatusError, arms the backoff
+// gate in TryStart: the immediate retry is gated and the underlying is not
+// called again until the window expires.
+//
+// A frozen fake clock (WithStartRetryClock) and identity jitter give exact,
+// deterministic windows without synctest complications from the RAG manager's
+// internal goroutines.
+func TestRAGStartableBackoff_StatusErrorEngagesGate(t *testing.T) {
+	t.Parallel()
+
+	counting := &countingStatusErrStrategy{statusCode: 429}
+	toolset := buildRAGToolSet(t, counting)
+
+	// Frozen clock: window = base + 0% jitter = exactly base.
+	// We advance the clock manually to control expiry.
+	now := time.Unix(1_000_000, 0)
+	s := tools.NewStartable(toolset,
+		tools.WithStartRetryJitter(func(d time.Duration) time.Duration { return d }), // identity
+		tools.WithStartRetryClock(func() time.Time { return now }),
+	)
+
+	// Attempt 1: TryStart invokes Initialize and arms the gate.
+	_, err := s.TryStart(t.Context())
+	require.Error(t, err, "expected failure on attempt 1")
+	assert.Equal(t, int32(1), counting.calls.Load(), "Initialize must be called on attempt 1")
+
+	// Immediate TryStart: gate must suppress it (clock not advanced).
+	_, err = s.TryStart(t.Context())
+	require.Error(t, err)
+	assert.Equal(t, int32(1), counting.calls.Load(),
+		"gate must suppress TryStart within the window")
+
+	// Advance clock past base window (base × 1.0 with identity jitter).
+	// Use 6 minutes to exceed any possible jittered window for any attempt.
+	now = now.Add(6 * time.Minute)
+
+	// Gate expired: TryStart must invoke Initialize again.
+	_, err = s.TryStart(t.Context())
+	require.Error(t, err, "still failing — expected error")
+	assert.Equal(t, int32(2), counting.calls.Load(),
+		"Initialize must be called again once the backoff window expires")
+}
+
+// TestRAGStartableBackoff_PlainErrorNoGate proves that a plain (non-StatusError)
+// failure does NOT arm the gate: every TryStart reaches Initialize.
+func TestRAGStartableBackoff_PlainErrorNoGate(t *testing.T) {
+	t.Parallel()
+
+	counting := &countingPlainErrStrategy{}
+	toolset := buildRAGToolSet(t, counting)
+	s := tools.NewStartable(toolset)
+
+	for i := int32(1); i <= 3; i++ {
+		_, err := s.TryStart(t.Context())
+		require.Error(t, err)
+		assert.Equal(t, i, counting.calls.Load(),
+			"plain error must not gate subsequent TryStart calls (attempt %d)", i)
+	}
+}

pkg/tools/export_test.go

@@ -1,6 +1,9 @@
 package tools
 
-import "context"
+import (
+	"context"
+	"time"
+)
 
 // ExportedPublishStopRequest exposes the request-publication half of
 // StopIfStarted for tests in the _test package: it leaves a pending stop
@@ -12,3 +15,16 @@ func (s *StartableToolSet) ExportedPublishStopRequest(ctx context.Context) {
 	s.stopRequested = true
 	s.stopRequestCtx = ctx
 }
+
+// ExportedComputeStartBackoff exposes computeStartBackoff for unit tests
+// that verify the bounds and cap behaviour without going through a full
+// Start cycle.
+func ExportedComputeStartBackoff(attempt int, jitterFn func(time.Duration) time.Duration) time.Duration {
+	return computeStartBackoff(attempt, jitterFn)
+}
+
+// Exported backoff bound constants for test assertions.
+const (
+	ExportedStartBackoffBase = startBackoffBase
+	ExportedStartBackoffMax  = startBackoffMax
+)

pkg/tools/mcp/mcp.go

@@ -96,7 +96,7 @@ func CreateToolSet(ctx context.Context, toolset latest.Toolset, runConfig *confi
 	case toolset.Command != "":
 		resolvedCommand, err := toolinstall.EnsureCommand(ctx, toolset.Command, toolset.Version)
 		if err != nil {
-			slog.WarnContext(ctx, "MCP command not yet available, will retry on next turn",
+			slog.WarnContext(ctx, "MCP command not yet available, will retry",
 				"command", toolset.Command, "error", err)
 			resolvedCommand = toolset.Command
 		}