Skip to content

AI assistant cost optimization: Keep prompt history byte-stable so the cache prefix survives #5789

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jurgenwerk wants to merge 7 commits into
base: main
Choose a base branch
from
Open

AI assistant cost optimization: Keep prompt history byte-stable so the cache prefix survives #5789

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
0ae799e
Keep prompt history byte-stable so the cache prefix survives
jurgenwerk Aug 14, 2026
f6f3d33
Stop eliding past code blocks from prompt history
jurgenwerk Aug 19, 2026
45c8a78
Merge remote-tracking branch 'origin/main' into prompt-cache-stability
jurgenwerk Aug 19, 2026
6006252
Accumulate tools first-wins in first-seen order so the array stops ch…
jurgenwerk Aug 20, 2026
c732bc8
Stop injecting per-card patchCardInstance tools on interactive messages
jurgenwerk Aug 20, 2026
41a4814
Fix CI for the tools-array changes: type-safe assertion, retargeted p…
jurgenwerk Aug 20, 2026
d0eaaae
Remove tools only when their skill is disabled, and keep the editabil…
jurgenwerk Aug 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions packages/ai-bot/lib/responder.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -309,6 +309,12 @@ export class Responder {
let cachedTokens = (chunk.usage as any).prompt_tokens_details
?.cached_tokens;
let costUsd = (chunk.usage as any).cost;
// OpenRouter also stamps each chunk with the provider that served the
// request and the generation id. Recording them alongside the counts
// makes a cache miss attributable: a cachedTokens collapse with a
// provider change is a routing miss, not a prompt-shape bug.
let provider = (chunk as any).provider;
let generationId = chunk.id;
Comment on lines +316 to +317

@habdelra habdelra Aug 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Claude Code 🤖] Test gap — non-blocking: these two fields have no coverage, and the test that looks like it covers them is structurally blind to them.

packages/ai-bot/tests/responding-test.ts asserts the usage payload with a whole-object assert.deepEqual(JSON.parse(last.content.data as string).usage, { promptTokens, completionTokens, cachedTokens, costUsd }). That assertion would catch an unexpected extra key — except the fixture chunk it feeds onChunk is { choices: [], usage: { … } } with no id and no provider, so generationId and provider both come out undefined and are stripped by the typeof … === 'string' spread. A real OpenRouter chunk always carries id, so the assertion has quietly stopped describing the production shape of usage.

Two lines fix it: add id: 'gen-abc123' and provider: 'Anthropic' to that fixture chunk and the corresponding generationId / provider entries to the expected object. That also pins the typeof guards, which are the only thing keeping a non-string id out of the room event.

Generated by Claude Code

// Hand the counts to the publisher so they ride on the final room
// event. When the final edit has already gone out (the usage chunk
// trails the finish chunk), finalize() sends one more edit to carry
Expand All @@ -318,6 +324,8 @@ export class Responder {
completionTokens: chunk.usage.completion_tokens,
...(typeof cachedTokens === 'number' ? { cachedTokens } : {}),
...(typeof costUsd === 'number' ? { costUsd } : {}),
...(typeof provider === 'string' ? { provider } : {}),
...(typeof generationId === 'string' ? { generationId } : {}),
};
log.info(
`Request used ${chunk.usage.prompt_tokens} prompt tokens (${
Expand Down
11 changes: 11 additions & 0 deletions packages/ai-bot/main.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,17 @@ class Assistant {
// the cast.
(request as Record<string, unknown>).usage = { include: true };

// Prompt caches live per provider, and the router is otherwise free to
// spread a room's requests across providers — which turns a warm cache
// prefix into a full-price miss mid-conversation. Bias Anthropic-model
// requests to Anthropic itself, keeping fallbacks for availability.
if (this.getModel(prompt).startsWith('anthropic/')) {
(request as Record<string, unknown>).provider = {
order: ['anthropic'],
allow_fallbacks: true,
};
}

if (prompt.reasoningEffort !== undefined) {
request.reasoning_effort = prompt.reasoningEffort;
}
Expand Down
Loading
Loading