fix(implications): move per-call target to end of prompt to enable prompt caching - #61
Open
fashionmascherine-svg wants to merge 1 commit into
Conversation
IMPLICATION_PROMPT places the per-call {target_title} before the large, run-static {group_titles_text} block and the instructions. Prompt caching keys on the common prefix, so the changing target at the top makes the ~35-44k static tokens uncacheable and every call is billed full input (~0.2% cache hit measured).
Moving the target block to the end makes the static prefix cacheable. Measured on DeepSeek v4-flash: cache-hit 0.2% -> ~40% aggregate (~80% on the implications step), ~40% lower run cost on a small tag; larger tags benefit more. Output unchanged in order of magnitude (coverage identical).
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe implication prompt is reorganized so target-event instructions appear later, require exact titles from available events, use a stricter target format, and require only the specified JSON output. ChangesImplication prompt update
Estimated code review effort: 1 (Trivial) | ~5 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
IMPLICATION_PROMPTplaces the per-call{target_title}before the large, run-static{group_titles_text}block and the instruction body. Prompt caching keys on the longest common prefix, so a target that changes every call makes the whole ~35–44k-token static section uncacheable — every implications call is billed full input.Fix
Move the
## TARGET EVENTblock to the end of the prompt (after the available-events list and the instructions). The static prefix (events list + instructions) becomes a stable, cacheable prefix across all calls in a run. No change to the task semantics or output schema.Measured (DeepSeek
deepseek-chat/ v4-flash)Same input, controlled before/after run:
Diff
Two hunks in
backend/core/steps/implications.py:Summary by CodeRabbit