test: add StartableToolSet backoff regression tests and docs

docs/tools/lsp/index.md

@@ -197,6 +197,8 @@ Available Capabilities:
 
 LSP toolsets are managed by the same supervisor as MCP toolsets, so a crashed `gopls` (or any other language server) is reconnected automatically with exponential backoff. Use the [`lifecycle`](../../configuration/tools/index.md#toolset-lifecycle) block to tune the policy per toolset — for example, mark `gopls` as `strict` if your CI flow requires it to be available, or use `/toolset-restart gopls` from the TUI to force a reconnect when the server gets stuck.
 
+**Startup failure behaviour:** local process failures such as a missing binary or server-unavailable error fail fast — each turn retries immediately with no artificial delay. The backoff described in [Indexing failures, retries and backoff](../rag/index.md#indexing-failures-retries-and-backoff) applies only to structured HTTP rate-limit and server-error responses from model providers, which local LSP process errors are not.
+
 ```yaml
 toolsets:
   - type: lsp

docs/tools/mcp/index.md

@@ -243,6 +243,8 @@ See [Per-Toolset Model Routing](../../configuration/tools/index.md#per-toolset-m
 
 Local stdio and remote MCP servers are supervised: crashed servers reconnect automatically with exponential backoff. **Remote** MCP servers (Streamable HTTP / SSE) also reconnect after idle/clean connection closes — services like Notion and Linear periodically close idle connections, and Docker Agent reconnects transparently. Tune the policy with the `lifecycle` block:
 
+**Startup failure behaviour:** local process failures (missing binary, connection refused, authentication error) fail fast — each turn retries immediately with no artificial delay added. Only structured HTTP rate-limit (HTTP 429) and server error (HTTP 5xx) responses from a backing model provider trigger a backoff window. See [Indexing failures, retries and backoff](../rag/index.md#indexing-failures-retries-and-backoff) for the full policy.
+
 ```yaml
 toolsets:
   - type: mcp

docs/tools/rag/index.md

@@ -174,6 +174,86 @@ chunking:
 >
 > Currently supports Go (`.go`) files. More languages will be added. Falls back to plain text chunking for unsupported file types.
 
+## Indexing failures, retries and backoff
+
+When a knowledge-base fails to start — because the embedding provider is rate-limiting
+your requests or returning a transient server error — Docker Agent spaces out retry
+attempts with bounded exponential backoff instead of hammering the provider on
+every agent turn.
+
+### What triggers backoff
+
+Backoff applies only to **structured HTTP rate-limit (429), request timeout (408), and server error (5xx)**
+responses from the embedding or model provider. These are the signals that mean
+"slow down", and spacing them out gives the provider room to recover:
+
+| Failure kind | Behaviour |
+|---|---|
+| HTTP 429 (rate limit) | Backoff: next attempt delayed |
+| HTTP 5xx / overload (500, 502, 503, 504, 529) | Backoff: next attempt delayed |
+| HTTP 408 (request timeout) | Backoff: next attempt delayed |
+| Bad config, invalid model, wrong API key (other 4xx errors) | Fail fast: retried every turn with no added delay |
+| Context cancellation or agent shutdown | Immediate: no delay |
+
+### Retry policy and parameters
+
+The backoff is **bounded exponential with additive jitter**:
+
+- **Base delay**: 15 seconds
+- **Maximum delay**: 5 minutes
+- **Growth**: doubles after each consecutive retryable failure (15s → 30s → 1m → 2m → 4m → 5m)
+- **Jitter**: each wait is a random value in `[nominal, 1.2×nominal]` (additive 0–20%)
+  so concurrent knowledge-base sources spread their retries and avoid
+  hammering the provider together
+
+The gate is a lightweight wall-clock check — it creates no background threads or
+timers. A Stop command or agent shutdown takes effect immediately regardless of
+how much of the backoff window remains.
+
+### Operational impact
+
+**Before**: a rate-limited knowledge base was re-indexed on every agent turn —
+`max_indexing_concurrency × max_embedding_concurrency` concurrent provider calls
+could relaunch within milliseconds, easily tripping rate limits for both the
+knowledge base and the agent's own model calls.
+
+**After**: retries are spaced out and jittered so the provider has room to recover
+before the next attempt. The agent continues working with any other toolsets that
+are not affected.
+
+### What you will see
+
+- Docker Agent logs a single warning when a knowledge base first fails to start.
+  Repeated failures in between are logged at debug level only, so you are not
+  flooded with alerts on every turn. Recovery is intentionally silent — the
+  tool appearing in the agent's tool list is the signal that indexing succeeded.
+- The knowledge-base tool does not appear in the agent's tool list until indexing
+  succeeds. A successful start is silent — the tool is listed and the agent uses it.
+
+### Troubleshooting repeated 429 errors
+
+If you see persistent `429` errors in the logs:
+
+1. **Check provider rate limits.** Your embedding API key may have a low requests-per-minute
+   quota. Upgrading the plan or using a different API key can help.
+2. **Reduce concurrency.** The chunked-embeddings and semantic-embeddings strategies
+   accept `max_indexing_concurrency` (default `3`) and `max_embedding_concurrency`
+   (default `3`) parameters. Lowering these reduces simultaneous requests:
+
+   ```yaml
+   rag:
+     name: docs
+     path: ./knowledge-base
+     strategy:
+       type: chunked-embeddings
+       params:
+         max_indexing_concurrency: 1
+         max_embedding_concurrency: 1
+   ```
+
+3. **Use a model with a higher quota.** Some providers offer higher rate limits on
+   specific embedding model tiers.
+
 ## Debugging RAG
 
 Enable debug logging to see retrieval details:

pkg/tools/builtin/rag/rag_backoff_test.go

@@ -0,0 +1,129 @@
+package rag
+
+// Regression tests proving that the real RAG ToolSet, when wrapped in
+// tools.NewStartable, correctly engages the backoff gate for structured
+// HTTP-status failures (e.g. a 429 rate-limit from the embedding provider)
+// and does NOT throttle plain (non-StatusError) failures.
+//
+// Gate enforcement is in tryStartLocked (TryStart paths only); all gate
+// assertions here use TryStart, not Start. A frozen fake clock supplied via
+// WithStartRetryClock gives deterministic windows without needing synctest.
+
+import (
+	"context"
+	"fmt"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+
+	"github.com/docker/docker-agent/pkg/modelerrors"
+	"github.com/docker/docker-agent/pkg/rag"
+	"github.com/docker/docker-agent/pkg/rag/strategy"
+	"github.com/docker/docker-agent/pkg/tools"
+)
+
+// countingStatusErrStrategy is a mockStrategy variant whose Initialize
+// returns a *modelerrors.StatusError and counts every invocation.
+type countingStatusErrStrategy struct {
+	mockStrategy
+
+	statusCode int
+	calls      atomic.Int32
+}
+
+func (s *countingStatusErrStrategy) Initialize(_ context.Context, _ []string, _ strategy.ChunkingConfig) error {
+	s.calls.Add(1)
+	return &modelerrors.StatusError{
+		StatusCode: s.statusCode,
+		Err:        fmt.Errorf("HTTP %d error from provider", s.statusCode),
+	}
+}
+
+// countingPlainErrStrategy always returns a plain (non-StatusError) error
+// so the backoff gate must never fire.
+type countingPlainErrStrategy struct {
+	mockStrategy
+
+	calls atomic.Int32
+}
+
+func (s *countingPlainErrStrategy) Initialize(_ context.Context, _ []string, _ strategy.ChunkingConfig) error {
+	s.calls.Add(1)
+	return assert.AnError
+}
+
+func buildRAGToolSet(t *testing.T, impl strategy.Strategy) *ToolSet {
+	t.Helper()
+	cfg := rag.Config{
+		StrategyConfigs: []strategy.Config{
+			{Name: "test-strategy", Strategy: impl},
+		},
+	}
+	mgr, err := rag.New(t.Context(), "test-rag", cfg, nil)
+	require.NoError(t, err)
+	return &ToolSet{manager: mgr, toolName: "test-rag"}
+}
+
+// TestRAGStartableBackoff_StatusErrorEngagesGate proves that the real RAG
+// ToolSet, when it fails with a *modelerrors.StatusError, arms the backoff
+// gate in TryStart: the immediate retry is gated and the underlying is not
+// called again until the window expires.
+//
+// A frozen fake clock (WithStartRetryClock) and identity jitter give exact,
+// deterministic windows without synctest complications from the RAG manager's
+// internal goroutines.
+func TestRAGStartableBackoff_StatusErrorEngagesGate(t *testing.T) {
+	t.Parallel()
+
+	counting := &countingStatusErrStrategy{statusCode: 429}
+	toolset := buildRAGToolSet(t, counting)
+
+	// Frozen clock: window = base + 0% jitter = exactly base.
+	// We advance the clock manually to control expiry.
+	now := time.Unix(1_000_000, 0)
+	s := tools.NewStartable(toolset,
+		tools.WithStartRetryJitter(func(d time.Duration) time.Duration { return d }), // identity
+		tools.WithStartRetryClock(func() time.Time { return now }),
+	)
+
+	// Attempt 1: TryStart invokes Initialize and arms the gate.
+	_, err := s.TryStart(t.Context())
+	require.Error(t, err, "expected failure on attempt 1")
+	assert.Equal(t, int32(1), counting.calls.Load(), "Initialize must be called on attempt 1")
+
+	// Immediate TryStart: gate must suppress it (clock not advanced).
+	_, err = s.TryStart(t.Context())
+	require.Error(t, err)
+	assert.Equal(t, int32(1), counting.calls.Load(),
+		"gate must suppress TryStart within the window")
+
+	// Advance clock past base window (base × 1.0 with identity jitter).
+	// Use 6 minutes to exceed any possible jittered window for any attempt.
+	now = now.Add(6 * time.Minute)
+
+	// Gate expired: TryStart must invoke Initialize again.
+	_, err = s.TryStart(t.Context())
+	require.Error(t, err, "still failing — expected error")
+	assert.Equal(t, int32(2), counting.calls.Load(),
+		"Initialize must be called again once the backoff window expires")
+}
+
+// TestRAGStartableBackoff_PlainErrorNoGate proves that a plain (non-StatusError)
+// failure does NOT arm the gate: every TryStart reaches Initialize.
+func TestRAGStartableBackoff_PlainErrorNoGate(t *testing.T) {
+	t.Parallel()
+
+	counting := &countingPlainErrStrategy{}
+	toolset := buildRAGToolSet(t, counting)
+	s := tools.NewStartable(toolset)
+
+	for i := int32(1); i <= 3; i++ {
+		_, err := s.TryStart(t.Context())
+		require.Error(t, err)
+		assert.Equal(t, i, counting.calls.Load(),
+			"plain error must not gate subsequent TryStart calls (attempt %d)", i)
+	}
+}