[https://nvbugs/6438658][fix] Fix KV cache estimation capacity#16545
Draft
jiaganc wants to merge 1 commit into
Draft
[https://nvbugs/6438658][fix] Fix KV cache estimation capacity#16545jiaganc wants to merge 1 commit into
jiaganc wants to merge 1 commit into
Conversation
Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com>
jiaganc
force-pushed
the
codex/fix-kv-estimation-cuda-graph-warmup
branch
from
July 17, 2026 14:27
084262f to
9917bcf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@coderabbitai summary
Description
Fix KV cache capacity estimation so the resulting cache can satisfy runtime warmup and resume requirements.
Without these constraints, pool-ratio estimation can allocate a pool layout that cannot run CUDA graph warmup. Ignoring resume utilization during distributed quota synchronization can also remove the reserved headroom and overestimate usable token capacity.
max_util_for_resumeheadroom when synchronizing KV cache capacity across distributed ranks.KVCacheManagerV2.Test Coverage
max_util_for_resumevalidation.pre-commitpassed for all changed files.python3 -m py_compilepassed for the changed Python files.tensorrt_llm.bindings.PR Checklist
GitHub Bot Help
To see a list of available CI bot commands, comment
/bot help.