Skip to content

RAG semantic-embeddings indexing can trigger a self-sustaining rate-limit retry storm #4060

Description

@steilerDev

Summary

An agent configured with a rag semantic-embeddings strategy over a knowledge base large enough that per-chunk LLM enrichment can't finish within the toolset-start timeout will fall into a self-sustaining retry storm against the configured model provider, because several independent pieces of behavior compound:

  1. Toolset start (including RAG indexing) is bounded by a hardcoded, non-configurable timeout (DefaultStartTimeout = 30 * time.Second, pkg/tools/startable.go:381), shared between the cold-start probe and every subsequent restart attempt. There's no way to grant a RAG-heavy toolset a larger start budget.
  2. A failed/aborted toolset start leaves s.started == false with no backoff or cooldown before the next attempt (startLocked, pkg/tools/startable.go:424).
  3. Agent.Tools() calls ensureToolSetsAreStarted on every agentic loop iteration, not once per user turn — confirmed by the loop's own comments that getTools() is "the single authoritative source ... each iteration" (pkg/runtime/loop.go:1634, see also L918-922 and the call sites at L382/L486/L1643). So a stuck/failing RAG toolset gets a fresh start attempt on every loop step of the current turn, with zero delay between attempts.
  4. Per-file indexing progress is only persisted after a file's embeddings fully succeed (indexFile, pkg/rag/strategy/vector_store.go:646-658). Any abort mid-file (timeout, canceled context, or a classified non-retryable model error) discards all progress for that file, so the next attempt reprocesses the same chunks from scratch.
  5. A 429 from the model provider is correctly classified as non-retryable for the current request (avoiding hammering a single failing call — see classifyModelCallError/errIndexingAborted in pkg/rag/strategy/indexing_errors.go and isRetryableStatusCode in pkg/modelerrors/modelerrors.go), but nothing throttles the next full restart triggered by (2)+(3). The circuit breaker resets instantly instead of backing off.

Combined, these mean: indexing concurrency (max_indexing_concurrency * max_embedding_concurrency concurrent chat-model calls per strategy instance) gets relaunched in full on every loop iteration until the KB backlog is fully indexed or the turn ends, with no inter-attempt delay. For a KB backlog large enough to routinely exceed 30s, this produces a sustained, rapidly-repeating burst of concurrent model calls against the provider — easily reaching thousands of requests/minute — which:

  • Trips provider-side rate limiting (429s) far faster than a human would expect from the configured concurrency numbers alone, since the same burst repeats every loop iteration rather than once.
  • Since the RAG chat-model traffic and the agent's own primary conversational model calls share the same provider/gateway, saturating it with retry-storm traffic causes the agent's own turn-completing model calls to get rate-limited too — with no retry/fallback configured by default, that can outright fail the turn ("All models failed"), not just degrade it.
  • Overflows the fixed-size RAG event channel (make(chan types.Event, 500), pkg/rag/builder.go:235) since each repeated attempt emits a fresh burst of usage/indexing_progress events faster than the single consumer goroutine can drain them relative to the burst rate, silently dropping observability events.

Expected behavior

Some combination of:

  • An exponential (or at least fixed-minimum) backoff between toolset-start retries after a failure, so a stuck RAG index doesn't relaunch its full concurrent burst on every loop iteration.
  • A start timeout that can be configured per-toolset (or is simply longer/adaptive for strategies doing bulk LLM-backed indexing), decoupled from the "is this toolset wedged" probe timeout.
  • Optionally, partial progress checkpointing at the chunk level (not just per-file) so an aborted run doesn't discard everything already embedded.

Reproduction sketch

  1. Configure an agent with a rag strategy of type: semantic-embeddings with a non-trivial max_indexing_concurrency / max_embedding_concurrency product, pointed at a knowledge base with enough uncached/changed content that the per-chunk LLM summarization pass cannot complete in ~30s.
  2. Start a session and watch debug logs: repeated "Toolset start failed; will retry on next turn" / "Toolset still unavailable; retrying next turn" messages fire within the same turn, seconds apart, each accompanied by a fresh burst of concurrent chat-model calls for the same files/chunks that just failed.
  3. Once the provider starts returning 429s, the same files continue being reprocessed (and re-failing) every loop iteration, and other model calls made by the agent in the same turn/session may start failing with 429 as well.

Environment

Observed via cagent's debug log with a RAG semantic-embeddings strategy backed by an Anthropic model as the chat_model, called through an internal AI Gateway. The mechanism described above is provider-agnostic — it's a property of the toolset-start/retry/indexing lifecycle, not specific to Anthropic or any particular gateway.

Metadata

Metadata

Assignees

Labels

area/ragFor work/issues that have to do with the RAG featuresarea/runtimeRuntime engine, agent loop execution, tool dispatch, loop detectionarea/toolsFor features/issues/fixes related to the usage of built-in and MCP tools

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions