Design: inference-aware fleet routing - #364
Conversation
067c3db to
64d3845
Compare
The control plane's fleet gateway routes by path and static weight only, so it is cache-blind: a multi-turn chat can land on a different cluster each turn and pay full prefill every time, and the fleet can't express request priority. This design adds cache-locality affinity and criticality, composed against the Gateway API standard (SessionPersistence and GAIE) so a conformant gateway works without a codepath per vendor. It covers modelplaneai#71 and modelplaneai#8, and relates to the Traefik choice in modelplaneai#89. Signed-off-by: Dennis Ramdass <dennis@upbound.io>
64d3845 to
bca9c91
Compare
Anchor the design on the user-facing resources: what a user authors (ModelService, ModelEndpoint) and how the composition functions and two gateways implement it, before proposing affinity and criticality on the ModelService. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>
| more gateways qualify with no new Modelplane code. That is the point of composing | ||
| to the standard: the supported set grows without the design diverging. |
There was a problem hiding this comment.
Why does supported set matter? Do we want to let folks bring their own preferred GAIE implementation? If so why?
There was a problem hiding this comment.
Took a stance: we target one gateway (Envoy Gateway + GAIE + Envoy AI Gateway), not arbitrary implementations. Composing to the standard is about getting off Traefik's beyond-spec behavior and onto the inference-aware features — not BYO. The InferenceGateway backend discriminator just stays as the seam if a platform team ever needs a different conformant gateway; breadth of support is a consequence, not the goal. Reworded the "supported set grows" line into a new "Which gateway" section.
| rather than sit in a CR. **This document does not propose building it.** The | ||
| adapter belongs with a provider-auth design, not with fleet routing. Until it | ||
| exists, heterogeneous-path provider aggregation stays a Traefik-backend concern, | ||
| and the standard, inference-aware backend serves self-hosted fleets. |
There was a problem hiding this comment.
I think if this doc proposes moving the InferenceGateway implementation back to a GAIE implementation, it needs to propose a solution for heterogenous rewrites. I'm open to a component that does the rewrite for us if you think it's the best path.
There was a problem hiding this comment.
Agreed it needed a real answer — added one, in a new "Heterogeneous rewrites without the beyond-spec feature" section. Two cases:
- Self-hosted per-endpoint paths move to an ext_proc rewrite (
EnvoyExtensionPolicy/extProc), which rides GAIE's EPP since that's already an ext_proc in the request path — or normalize to one deployment-scoped path so a single rule-level rewrite covers it. - External providers are handled by Envoy AI Gateway's
AIServiceBackend+BackendSecurityPolicy(path/schema translation + credential injection). That's a built-in feature of the gateway we're adopting, not a Modelplane-built adapter, so it also resolves the provider-auth piece the doc used to punt.
Both put the rewrite in an ext_proc rather than a beyond-spec gateway feature, which is what lets the standard profile replace Traefik. (Also reworked the doc around ModelService — that reframe hadn't actually been pushed before.)
The doc hand-waved normalize-path-aggregation-first and punted provider aggregation to an unbuilt adapter. Replace that with a concrete answer: per-endpoint rewrites move to an ext_proc (EnvoyExtensionPolicy/extProc, riding GAIE's EPP), and provider aggregation is a built-in Envoy AI Gateway feature (AIServiceBackend + BackendSecurityPolicy). State up front what the doc proposes and which gateway it targets, so an approval is unambiguous. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>
2 participants
Description of your changes
A design doc, not code yet — opening as a draft for direction before implementation.
The control plane's fleet gateway routes by path and static weight only, so it's cache-blind: a multi-turn chat can land on a different cluster each turn and pay full prefill every time (~800ms TTFT vs ~150ms warm), and the fleet can't express request priority. The per-cluster EPP is already inference-aware; the fleet layer isn't.
This proposes cache-locality affinity (Session / ClientIP) and criticality on the fleet gateway, composed against the Gateway API standard —
SessionPersistence(GEP-1619) and GAIE (InferencePool/InferenceModel) — rather than any one gateway's own resources, so a conformant gateway works with one codepath instead of a per-vendor fork. The fleet routes to a cluster and delegates the replica pick to that cluster's EPP (move requests, not KV state). Covers #71 and #8.Two things I'd flag for judgment:
URLRewrite(the reason we're on Traefik,compose-model-serviceputs URLRewrite filter under backendRef, breaking the data path #85/Switch control plane gateway from Envoy Gateway to Traefik #89 — it's beyond the Gateway API spec, which only allowsURLRewriteat the rule level). The doc proposes relocating it (deployment-scoped replica paths, GAIE canary weighting, a per-provider adapter) so the fleet gateway needs only the standard. Honest cost: Traefik handles provider path/TLS for free today.The standard is treated as a baseline with clean extension seams (the
backenddiscriminator, the configurable EPP in #179, the ext_proc chain) so richer routing can attach without forking core.design/routing.mdonly; no code.I have:
Run(design doc only, no code paths)nix flake check(or./nix.sh flake check) and made sure it passes.Added or updated tests covering any composition function changes.(design doc only)git commit -s.