[Blog] Sticky Until Saturated: Token-Aware Routing in llm-d - #462
[Blog] Sticky Until Saturated: Token-Aware Routing in llm-d#462kaushikmitr wants to merge 1 commit into
Conversation
✅ Deploy Preview for llm-d ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Unsigned commits detected! Please sign your commits. For instructions on how to set up GPG/SSH signing and verify your commits, please see GitHub Documentation. |
114bb32 to
ce8bc37
Compare
There was a problem hiding this comment.
Pull request overview
This PR adds a new long-form technical blog post to the site, explaining how to select llm-d router scheduling configurations based on whether a workload is prefill-compute-bound or decode-slot-bound, including benchmark results and a derivation/calibration recipe for the saturation threshold.
Changes:
- Adds a new blog post covering bottleneck-matched scheduling configurations and operational guidance.
- Includes benchmark result discussion across three workloads and a derivation for τ / saturation override configuration.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
fe6e4b9 to
6917356
Compare
ahg-g
left a comment
ahg-g
left a comment
There was a problem hiding this comment.
one nit, otherwise looks good
|
|
||
| *Matching scheduler configurations to workload constraints in the llm-d router* | ||
|
|
||
| The llm-d router's default configuration has changed to something an operator can reason about: **identify what limits the workload's throughput, and route on the signal that directly tracks that limit.** We call the result a *bottleneck-matched configuration* (or *matched configuration* for short). For prefill-bound traffic (long prompts), the router keeps requests on cache-warm pods and picks the one with the least uncached prefill work in flight (`prefix-cache-affinity-filter + token-load-scorer`); for decode-bound traffic (long outputs), it picks the pod with the fewest active streams (`active-request-scorer`). Because one signal matches one bottleneck, the scheduler is legible: an operator can predict what it will do, why it will do it, and how it will degrade under load. The previous default, a four-signal weighted blend (prefix-cache match, queue depth, KV utilization, LRU), produced emergent behavior that was hard to predict and harder to tune. |
There was a problem hiding this comment.
can we use endpoint instead of pod? A pod is not 1:1 with a model server (think multi-node)
| @@ -0,0 +1,326 @@ | |||
| --- | |||
| title: "Bottleneck-Aware Scheduling for LLM Inference" | |||
There was a problem hiding this comment.
This is a mundane title, how about "Stick or Spill: The Balancing Act of KV-Cache Affinity and Load Distribution"
There was a problem hiding this comment.
updated to Stick or Spill: Token-Aware Routing for LLM Inference Want to have token awareness in the title
| tags: [blog, scheduling, inference, sig-benchmarking] | ||
| --- | ||
|
|
||
| # Bottleneck-Aware Scheduling for LLM Inference |
There was a problem hiding this comment.
the blog should mention that this solution also aims to address hot spotting that most other routers suffer from because of over indexing on kv-cache affinity.
There was a problem hiding this comment.
added this:
Two failure modes motivated the change. The first is the previous default itself: a four-signal weighted blend (prefix-cache match, queue depth, KV utilization, LRU) whose emergent behavior was hard to predict and harder to tune. The second is the hot spotting most routers suffer from over-indexing on KV-cache affinity: affinity concentrates traffic on cache-warm endpoints, and without an explicit saturation release the warm endpoint keeps absorbing load past the point where a cold endpoint would serve the request faster. Token-aware routing pairs one signal with one bottleneck and one calibrated limit, so the scheduler is legible: an operator can predict what it will do, why it will do it, and how it will degrade under load.
|
|
||
| <!-- truncate --> | ||
|
|
||
| This post is the analysis behind that shift. It develops the framework across three workloads spanning the two bottleneck regimes (prefill compute, decode slots), derives the configuration's one threshold in closed form from a single hardware calibration (see [Deriving τ from the Hardware](#deriving-tau-from-the-hardware)), and evaluates the latency-predictor pipeline as a workload-agnostic alternative for operators with high-variance traffic or an uncharacterized bottleneck. The experiments were run against the affinity filter's earlier parameterization; the shipped interface has since been redesigned around the calibration quantity derived here (`peakPrefillThroughput`, see [the saturation override](#the-saturation-override)), and the calibration procedure is distributed with the router as a [shared recipe](https://github.com/llm-d/llm-d/tree/main/guides/recipes/router/calibration) with a per-(model, accelerator) [configuration matrix](https://github.com/llm-d/llm-d/blob/main/guides/recipes/router/calibration/configuration-matrix.md). |
There was a problem hiding this comment.
not a lot will be familiar with the latency predictor work, I think the description here should be agnostic to that otherwise people may confuse this as something that requires the predictor.
Also, I would suggest to simplify the description of the algorithm, something along the lines: we prioritize affinity unless the load on the server is above a specific limit, at which point we ignore affinity and pick the endpoint based on load only. For prefill, we measure load using uncached prefill tokens, for decode we use inflight requests.
Then describe how the threshold is set in a separate detailed section.
There was a problem hiding this comment.
this:
The llm-d router's default configuration has changed to something an operator can reason about, built on a single methodology: token-aware routing. The scheduler prioritizes KV-cache affinity, keeping each request on the endpoint that already holds its prefix, unless the load on that endpoint exceeds a calibrated limit; past the limit it ignores affinity and picks the endpoint by load alone. Load is measured in tokens matched to the workload's bottleneck, giving two configurations: for prefill-bound traffic (long prompts), prefix-cache affinity + token load (prefix-cache-affinity-filter + token-load-scorer), routing on uncached prefill tokens in flight; for decode-bound traffic (long outputs), prefix-cache affinity + active requests (prefix-cache-affinity-filter + active-request-scorer), routing on active streams. We call each pairing of the affinity filter with a load signal a bottleneck-matched configuration (or matched configuration for short).
73ad318 to
eeccbdf
Compare
kaushikmitr
changed the title
eeccbdf to
8213015
Compare
168acae to
142f133
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 14 changed files in this pull request and generated no new comments.
Suppressed comments (1)
blog/2026-08-04_bottleneck-aware-scheduling-for-llm-inference.mdx:83
- Grammatical typo: "a endpoint" should be "an endpoint".
**Decode-bound (slot-limited).** The bottleneck is the number of concurrent decode streams a endpoint can sustain. Prompts are short, outputs are long, prefill completes quickly, and the cluster spends most of its time generating output tokens. KV-cache pressure builds with concurrent requests, not with per-request size. The right scheduler signal is **running request count per endpoint**, sending new work to the endpoint with the fewest active streams keeps decode batches uniformly loaded. An alternative token-denominated signal is tokens in process: the in-flight load producer tracks tokens yet to be prefilled, and can be configured (`addEstimatedOutputTokens=true`) to also hold a ratio-based output estimate until the response completes, approximating KV-cache occupancy. Once a request is past prefill, its actual tokens in KV cache could in principle be tracked as the decode-load signal, from the router's event-driven KV-block index rather than the engine's polled KV-usage metric, which lags dispatch. In practice the simple active-request count works well, and it is the form we benchmark here. Reasoning (mean 1000 token prompts, mean 8000 output tokens) is our representative.
142f133 to
2863ed9
Compare
Adds a new post on matching llm-d router configurations to workload bottlenecks (prefill compute vs decode slots), with benchmark results across three workloads, the tau saturation-threshold derivation, and the calibration recipe that now ships as the router default. Signed-off-by: Kaushik Mitra <kaushikmitra@google.com>
2863ed9 to
cca72fe
Compare
Adds a new post on matching llm-d router configurations to workload bottlenecks (prefill compute vs decode slots), with benchmark results across three workloads, the tau saturation-threshold derivation, and the calibration recipe that now ships as the router default.