Skip to content

fix(server): retain Responses cache after tool turns - #714

Open
blake-gj wants to merge 1 commit into
antirez:mainfrom
blake-gj:fix/responses-cache-after-tool-turns
Open

fix(server): retain Responses cache after tool turns#714
blake-gj wants to merge 1 commit into
antirez:mainfrom
blake-gj:fix/responses-cache-after-tool-turns

Conversation

@blake-gj

@blake-gj blake-gj commented Aug 6, 2026

Copy link
Copy Markdown

Problem

Stateless Responses clients replay the typed output items from the previous
turn. When reasoning.summary is enabled, that replay includes the emitted
final-answer reasoning summary.

DS4's live visible key intentionally omitted final-answer summaries for older
clients that do not replay them. After a tool-output continuation, however, the
live sampled KV can tokenize differently from a freshly rendered prompt at the
tool boundary. A client that did replay the summary then missed both the exact
token prefix and the visible-prefix fallback, forcing a cold prefill on its
next user turn.

Four consecutive production occurrences at 18k-39k input tokens stopped at the
preceding tool-call frontier and spent 12.6-26.9 seconds cold-prefilling.

Change

  • Make the primary live visible key mirror the emitted final reasoning summary.
  • Retain one summary-free alternate key at the same KV frontier for older
    clients.
  • Accept either key in the existing visible-prefix continuation path.
  • Cover both replay shapes and the live-frontier guard in the server tests.

This adds no protocol or configuration flag. Durable disk KV continues to use
only the primary, faithful replay shape.

Validation

Clean branch based on b030961:

  • git diff --check
  • make clean && make — warning-free Metal build on an Apple M4 Pro
  • make cpu — warning-free CPU build
  • make ds4_test && ./ds4_test --server
  • ./ds4-eval --self-test-extractors
  • Agent, Q4_K, MXFP4, layer-pack, multi-GPU placement, GPU-argument, and CLI
    suites all passed

The aggregate make test reached the default model-backed long-context group
and stopped because this Mac has no ds4flash.gguf; all remaining
model-independent targets were run directly and passed.

The same change was also built for CUDA sm_120 on an RTX 5090 and RTX PRO
6000, and for ROCm gfx1151 on a Ryzen AI MAX+ 395. A three-host distributed,
streamed /v1/responses regression used a DeepSeek-V4-Flash hybrid Q2/Q4 model
at 256K context:

Those GPU builds emitted the pre-existing cache_source -Wstring-compare
warning and no new warning from this diff.

Request Input Cached Evaluated Prefill
Initial tool request 419 0 419 1.100 s
Tool-output continuation 524 505 19 0.130 s
Next user turn after final answer 563 557 6 0.058 s

The last request matched responses-visible and retained 98.9% of its prompt;
this is the transition that cold-prefilled before the change.

Responses clients can replay final-answer reasoning summaries on the next stateless turn. The live visible key omitted that summary, so token-boundary differences after a tool continuation could defeat exact matching and force a cold prefill.

Remember the emitted-summary replay as the primary key and retain a summary-free alternate for older clients. Cover both shapes in server tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant