Summary
An agent configured with a rag semantic-embeddings strategy over a knowledge base large enough that per-chunk LLM enrichment can't finish within the toolset-start timeout will fall into a self-sustaining retry storm against the configured model provider, because several independent pieces of behavior compound:
- Toolset start (including RAG indexing) is bounded by a hardcoded, non-configurable timeout (
DefaultStartTimeout = 30 * time.Second, pkg/tools/startable.go:381), shared between the cold-start probe and every subsequent restart attempt. There's no way to grant a RAG-heavy toolset a larger start budget.
- A failed/aborted toolset start leaves
s.started == false with no backoff or cooldown before the next attempt (startLocked, pkg/tools/startable.go:424).
Agent.Tools() calls ensureToolSetsAreStarted on every agentic loop iteration, not once per user turn — confirmed by the loop's own comments that getTools() is "the single authoritative source ... each iteration" (pkg/runtime/loop.go:1634, see also L918-922 and the call sites at L382/L486/L1643). So a stuck/failing RAG toolset gets a fresh start attempt on every loop step of the current turn, with zero delay between attempts.
- Per-file indexing progress is only persisted after a file's embeddings fully succeed (
indexFile, pkg/rag/strategy/vector_store.go:646-658). Any abort mid-file (timeout, canceled context, or a classified non-retryable model error) discards all progress for that file, so the next attempt reprocesses the same chunks from scratch.
- A 429 from the model provider is correctly classified as non-retryable for the current request (avoiding hammering a single failing call — see
classifyModelCallError/errIndexingAborted in pkg/rag/strategy/indexing_errors.go and isRetryableStatusCode in pkg/modelerrors/modelerrors.go), but nothing throttles the next full restart triggered by (2)+(3). The circuit breaker resets instantly instead of backing off.
Combined, these mean: indexing concurrency (max_indexing_concurrency * max_embedding_concurrency concurrent chat-model calls per strategy instance) gets relaunched in full on every loop iteration until the KB backlog is fully indexed or the turn ends, with no inter-attempt delay. For a KB backlog large enough to routinely exceed 30s, this produces a sustained, rapidly-repeating burst of concurrent model calls against the provider — easily reaching thousands of requests/minute — which:
- Trips provider-side rate limiting (429s) far faster than a human would expect from the configured concurrency numbers alone, since the same burst repeats every loop iteration rather than once.
- Since the RAG chat-model traffic and the agent's own primary conversational model calls share the same provider/gateway, saturating it with retry-storm traffic causes the agent's own turn-completing model calls to get rate-limited too — with no retry/fallback configured by default, that can outright fail the turn (
"All models failed"), not just degrade it.
- Overflows the fixed-size RAG event channel (
make(chan types.Event, 500), pkg/rag/builder.go:235) since each repeated attempt emits a fresh burst of usage/indexing_progress events faster than the single consumer goroutine can drain them relative to the burst rate, silently dropping observability events.
Expected behavior
Some combination of:
- An exponential (or at least fixed-minimum) backoff between toolset-start retries after a failure, so a stuck RAG index doesn't relaunch its full concurrent burst on every loop iteration.
- A start timeout that can be configured per-toolset (or is simply longer/adaptive for strategies doing bulk LLM-backed indexing), decoupled from the "is this toolset wedged" probe timeout.
- Optionally, partial progress checkpointing at the chunk level (not just per-file) so an aborted run doesn't discard everything already embedded.
Reproduction sketch
- Configure an agent with a
rag strategy of type: semantic-embeddings with a non-trivial max_indexing_concurrency / max_embedding_concurrency product, pointed at a knowledge base with enough uncached/changed content that the per-chunk LLM summarization pass cannot complete in ~30s.
- Start a session and watch debug logs: repeated
"Toolset start failed; will retry on next turn" / "Toolset still unavailable; retrying next turn" messages fire within the same turn, seconds apart, each accompanied by a fresh burst of concurrent chat-model calls for the same files/chunks that just failed.
- Once the provider starts returning 429s, the same files continue being reprocessed (and re-failing) every loop iteration, and other model calls made by the agent in the same turn/session may start failing with 429 as well.
Environment
Observed via cagent's debug log with a RAG semantic-embeddings strategy backed by an Anthropic model as the chat_model, called through an internal AI Gateway. The mechanism described above is provider-agnostic — it's a property of the toolset-start/retry/indexing lifecycle, not specific to Anthropic or any particular gateway.
Summary
An agent configured with a
ragsemantic-embeddingsstrategy over a knowledge base large enough that per-chunk LLM enrichment can't finish within the toolset-start timeout will fall into a self-sustaining retry storm against the configured model provider, because several independent pieces of behavior compound:DefaultStartTimeout = 30 * time.Second,pkg/tools/startable.go:381), shared between the cold-start probe and every subsequent restart attempt. There's no way to grant a RAG-heavy toolset a larger start budget.s.started == falsewith no backoff or cooldown before the next attempt (startLocked,pkg/tools/startable.go:424).Agent.Tools()callsensureToolSetsAreStartedon every agentic loop iteration, not once per user turn — confirmed by the loop's own comments thatgetTools()is "the single authoritative source ... each iteration" (pkg/runtime/loop.go:1634, see also L918-922 and the call sites at L382/L486/L1643). So a stuck/failing RAG toolset gets a fresh start attempt on every loop step of the current turn, with zero delay between attempts.indexFile,pkg/rag/strategy/vector_store.go:646-658). Any abort mid-file (timeout, canceled context, or a classified non-retryable model error) discards all progress for that file, so the next attempt reprocesses the same chunks from scratch.classifyModelCallError/errIndexingAbortedinpkg/rag/strategy/indexing_errors.goandisRetryableStatusCodeinpkg/modelerrors/modelerrors.go), but nothing throttles the next full restart triggered by (2)+(3). The circuit breaker resets instantly instead of backing off.Combined, these mean: indexing concurrency (
max_indexing_concurrency * max_embedding_concurrencyconcurrent chat-model calls per strategy instance) gets relaunched in full on every loop iteration until the KB backlog is fully indexed or the turn ends, with no inter-attempt delay. For a KB backlog large enough to routinely exceed 30s, this produces a sustained, rapidly-repeating burst of concurrent model calls against the provider — easily reaching thousands of requests/minute — which:"All models failed"), not just degrade it.make(chan types.Event, 500),pkg/rag/builder.go:235) since each repeated attempt emits a fresh burst ofusage/indexing_progressevents faster than the single consumer goroutine can drain them relative to the burst rate, silently dropping observability events.Expected behavior
Some combination of:
Reproduction sketch
ragstrategy oftype: semantic-embeddingswith a non-trivialmax_indexing_concurrency/max_embedding_concurrencyproduct, pointed at a knowledge base with enough uncached/changed content that the per-chunk LLM summarization pass cannot complete in ~30s."Toolset start failed; will retry on next turn"/"Toolset still unavailable; retrying next turn"messages fire within the same turn, seconds apart, each accompanied by a fresh burst of concurrent chat-model calls for the same files/chunks that just failed.Environment
Observed via
cagent's debug log with a RAGsemantic-embeddingsstrategy backed by an Anthropic model as the chat_model, called through an internal AI Gateway. The mechanism described above is provider-agnostic — it's a property of the toolset-start/retry/indexing lifecycle, not specific to Anthropic or any particular gateway.