[docs] Add Model Routing user documentation - #1124
Conversation
github-actions
Bot
added
doc-label-missing
wenjin272
left a comment
wenjin272
left a comment
There was a problem hiding this comment.
Thanks for putting together this comprehensive documentation and for covering the model routing behavior in depth. I left a few comments about simplifying the information architecture, making the examples annotation-first, and reducing implementation-level detail.
|
|
||
| # Model Routing Reference | ||
|
|
||
| Reference material for [Model Routing]({{< ref "docs/development/model_routing" >}}): event and response fields, metrics, validation stages, and descriptor keys. |
There was a problem hiding this comment.
I’m not sure a separate Reference page is warranted here. It adds another sidebar level for a relatively small amount of content, which is inconsistent with most other development topics. Could we keep Model Routing as a single page, merge the event and metric fields into Observability, and place validation notes near the relevant configuration sections? The descriptor keys could be omitted or moved to Javadocs.
There was a problem hiding this comment.
Done: the Reference subpage is gone and Model Routing is a single page again. The event and response fields and the metrics now sit in Observability, the validation timing is a short note under a small Advanced section, and the descriptor keys are removed.
| What works with routing, and what does not: | ||
|
|
||
| - Any registered `CHAT_MODEL` from any provider can be a candidate or a judge. Tool calls work: the model that answered the initial request is kept for every tool-call round of that request. | ||
| - Routers are declared with `addResource` on the execution environment or on the agent. There is no annotation for declaring a router inside an agent class, and the YAML API has no section for routers. |
There was a problem hiding this comment.
The examples currently register all resources through addResource, while our recommended Agent-facing style is annotation-based. Connections, candidate models, and the judge can already use @ChatModelConnection and @ChatModelSetup, but MODEL_ROUTER has no corresponding annotation. Could we add or track a dedicated annotation such as @ModelRouterSetup (@ModelRouter would conflict with the existing builder class), and then make the documentation examples fully annotation-first? This can be implemented in a separate PR if we want to keep this one documentation-only.
There was a problem hiding this comment.
Done for everything that has an annotation: the quick start now declares the connection, both candidates and the judge with @ChatModelConnection / @ChatModelSetup inside the agent class. The router itself is still registered with Agent.addResource in the constructor, and the page says why. I opened #1133 for a @ModelRouterSetup annotation (name per your suggestion) so this PR stays documentation-only.
|
|
||
| ## Routing Strategies | ||
|
|
||
| A `RoutingStrategy` is a serializable declaration produced by a `Strategies` factory. It travels in the agent plan as a type tag plus arguments; the executor for that type runs on the TaskManager. Every strategy either selects a candidate or abstains. |
There was a problem hiding this comment.
Overall, this page feels too implementation-heavy for user documentation. Details such as Plan type tags, TaskManager executors, durable call IDs, provider-specific finish-reason behavior, validation stages, and descriptor wire keys obscure the main usage flow. Could we shorten it around the overview, quick start, strategies, fallback, and observability, and move lower-level details to Javadocs or a short advanced subsection?
There was a problem hiding this comment.
Done: the page is now Overview, Quick Start, Routing Strategies, Default Model and Fallback, Observability, a short Advanced section (validation timing and recovery in two paragraphs), and Examples. Plan type tags, TaskManager executors, durable call ids, the per-provider finish-reason list and the validation stages are gone from the user path.
wenjin272
added
doc-included
… internals Addresses the review on apache#1124: - Merge the Reference subpage back into the Model Routing page: event and metric fields move into Observability, validation timing becomes a short note under Advanced, descriptor keys are dropped. - Make the examples annotation-first: the connection, the candidates and the judge are declared with @ChatModelConnection / @ChatModelSetup inside the agent; the router, which has no annotation yet, is registered with Agent.addResource in the constructor. - Cut implementation detail from the user path: plan type tags, TaskManager wording, durable call ids, the per-provider finish-reason list and the validation stages; recovery is one paragraph. Generated-by: Claude Code 2.1.273 (Claude Fable 5.1)
Adds docs/development/model_routing: a guide page covering router declaration, the rule-based, LLM-judge and custom strategies, default model and fallback with the error-handling policy table, durability and replay, and observability; plus a Reference subpage with the routing event and response fields, metrics, validation stages, and descriptor keys. Cross-links from Chat Models, Workflow Agent, and Monitoring (new rows for the routingDecisionLatencyMs histogram and the _model_routing_event type). Sibling page weights shift by one so the page sits after Chat Models; this also resolves the pre-existing MCP/Skills weight tie. Covers the doc-needed label on apache#964 and apache#1042.
… internals Addresses the review on apache#1124: - Merge the Reference subpage back into the Model Routing page: event and metric fields move into Observability, validation timing becomes a short note under Advanced, descriptor keys are dropped. - Make the examples annotation-first: the connection, the candidates and the judge are declared with @ChatModelConnection / @ChatModelSetup inside the agent; the router, which has no annotation yet, is registered with Agent.addResource in the constructor. - Cut implementation detail from the user path: plan type tags, TaskManager wording, durable call ids, the per-provider finish-reason list and the validation stages; recovery is one paragraph. Generated-by: Claude Code 2.1.273 (Claude Fable 5.1)
…xception and context budget wording Generated-by: Claude Code 2.1.273 (Claude Fable 5.1)
766360c to
5dceae0
Compare
wenjin272
left a comment
wenjin272
left a comment
There was a problem hiding this comment.
Thanks for addressing my comments. LGTM
2 participants
Adds user documentation for pluggable model routing (#964, #1042), closing the documentation checkbox in #1062 and the
doc-neededlabel on both PRs.New pages
docs/development/model_routing: declaring a router, the rule-based, LLM-judge and custom strategies, default model and fallback with the error-handling policy table, durability and replay, observability, examples.docs/development/model_routing/reference: routing event and response fields, metrics, validation stages, descriptor keys.Other changes
routingDecisionLatencyMsand_model_routing_event.Verification
hugo-booksubmodule; sidebar and all cross-page anchors verified.flink-agents-apijar built from main.Java only for now; the page states that Python parity is tracked separately.
🤖 Generated with Claude Code