fix(server): retain Responses cache after tool turns - #714
Open
blake-gj wants to merge 1 commit into
Open
Conversation
Responses clients can replay final-answer reasoning summaries on the next stateless turn. The live visible key omitted that summary, so token-boundary differences after a tool continuation could defeat exact matching and force a cold prefill. Remember the emitted-summary replay as the primary key and retain a summary-free alternate for older clients. Cover both shapes in server tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Stateless Responses clients replay the typed output items from the previous
turn. When
reasoning.summaryis enabled, that replay includes the emittedfinal-answer reasoning summary.
DS4's live visible key intentionally omitted final-answer summaries for older
clients that do not replay them. After a tool-output continuation, however, the
live sampled KV can tokenize differently from a freshly rendered prompt at the
tool boundary. A client that did replay the summary then missed both the exact
token prefix and the visible-prefix fallback, forcing a cold prefill on its
next user turn.
Four consecutive production occurrences at 18k-39k input tokens stopped at the
preceding tool-call frontier and spent 12.6-26.9 seconds cold-prefilling.
Change
clients.
This adds no protocol or configuration flag. Durable disk KV continues to use
only the primary, faithful replay shape.
Validation
Clean branch based on
b030961:git diff --checkmake clean && make— warning-free Metal build on an Apple M4 Promake cpu— warning-free CPU buildmake ds4_test && ./ds4_test --server./ds4-eval --self-test-extractorssuites all passed
The aggregate
make testreached the default model-backed long-context groupand stopped because this Mac has no
ds4flash.gguf; all remainingmodel-independent targets were run directly and passed.
The same change was also built for CUDA
sm_120on an RTX 5090 and RTX PRO6000, and for ROCm
gfx1151on a Ryzen AI MAX+ 395. A three-host distributed,streamed
/v1/responsesregression used a DeepSeek-V4-Flash hybrid Q2/Q4 modelat 256K context:
Those GPU builds emitted the pre-existing
cache_source-Wstring-comparewarning and no new warning from this diff.
The last request matched
responses-visibleand retained 98.9% of its prompt;this is the transition that cold-prefilled before the change.