docs(ai-chat): add prompt caching guide#3951
Conversation
|
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
✅ Files skipped from review due to trivial changes (1)
📜 Recent review details⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
WalkthroughA new MDX documentation page 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
## Summary
`chat.agent`'s system prompt (the `chat.prompt` text plus any skills
preamble) could not carry a provider cache breakpoint, so the largest
and most stable part of the prompt re-paid full input price on every
turn. `chat.toStreamTextOptions()` now emits the system prompt as a
structured message carrying `providerOptions` when you opt in, so a
provider can cache the system block. Without an option, `system` stays a
plain string, so existing behavior is unchanged.
## API
Three ways to opt in (most specific wins, no deep merge):
```ts
// Anthropic sugar
chat.toStreamTextOptions({ cacheControl: { type: "ephemeral" } });
// provider-agnostic (also covers Amazon Bedrock's cachePoint)
chat.toStreamTextOptions({ systemProviderOptions: { anthropic: { cacheControl: { type: "ephemeral" } } } });
// at the definition site
chat.prompt.set(SYSTEM_PROMPT, { providerOptions: { anthropic: { cacheControl: { type: "ephemeral" } } } });
```
The `cacheControl` shorthand is Anthropic-only; `systemProviderOptions`
is the general form. Pairs with a `prepareMessages` cache breakpoint to
cache the conversation prefix too.
Docs guide: #3951
ericallam
force-pushed
the
docs/chat-prompt-caching
branch
from
11c4746 to
4fa31fc
Compare
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
@trigger.dev/build
trigger.dev
@trigger.dev/core
@trigger.dev/python
@trigger.dev/react-hooks
@trigger.dev/redis-worker
@trigger.dev/rsc
@trigger.dev/schema-to-json
@trigger.dev/sdk
commit: |
|
@claude can you update these doc edits to use the latest version of the sonnet model claude-sonnet-4-6 instead of claude-sonnet-4-5 |
Co-authored-by: Eric Allam <ericallam@users.noreply.github.com>
|
Claude finished @ericallam's task in 32s —— View job Updating model version in prompt-caching guide
Already done! The model references were already updated to |
2 participants
Summary
New
/ai-chat/prompt-cachingguide covering how to cache a chat agent's prompt prefix with Anthropic prompt caching: the system prompt, the conversation history (aprepareMessagesbreakpoint), and how caching interacts with compaction. It also shows how to verify cache hits via usage and the dashboard, the prefix-stability footguns, and an "Other providers" section (OpenAI and Google cache automatically; Amazon Bedrock usescachePointthroughsystemProviderOptions).Registered under Features in the AI Agents nav, next to Compaction.