From 43996395536ea7a2a39fe5743daf145224a48048 Mon Sep 17 00:00:00 2001 From: Bihan Rana Date: Wed, 16 Sep 2026 11:50:10 +0545 Subject: [PATCH 1/2] [Presets] Update system_prompt.md for router usage --- .../presets/resources/system_prompt.md | 44 +++++++++++++++---- 1 file changed, 35 insertions(+), 9 deletions(-) diff --git a/src/dstack/_internal/cli/services/presets/resources/system_prompt.md b/src/dstack/_internal/cli/services/presets/resources/system_prompt.md index 101191b94..836e64587 100644 --- a/src/dstack/_internal/cli/services/presets/resources/system_prompt.md +++ b/src/dstack/_internal/cli/services/presets/resources/system_prompt.md @@ -260,9 +260,9 @@ mindful of which specific change was the root cause. {"resources": {...}, "context_length": ..., "benchmark": {...}, "learned": ..., "failed": ...} ``` -2. In case the task is using node groups (e.g. for PD disaggregation), - instead of a single `resources` it includes `groups`, and the fields are - these and no others: +2. In case the task is using node groups (a router group plus worker + groups, see `## Fleet Topology`), instead of a single `resources` it + includes `groups`, and the fields are these and no others: ``` {"groups": [[{...}], [{...}, {...}], [{...}]], "context_length": ..., "benchmark": {...}, "learned": ..., "failed": ...} @@ -321,9 +321,9 @@ serving engine: use `dataset` and `concurrency``co `shared_prefix_tokens` from `constraints.json` and measure all trials the same way so that their results are comparable with each other. -In case the task is using PD disaggregation, run benchmarks via SSH inside -the router node, directly against the router engine. Never benchmark prefill -or decode workers — each handles only part of a request. +In case the task is using a router (see `## Fleet Topology`), run benchmarks +via SSH inside the router node, directly against the router, never against a +worker, so that results stay comparable across trials. Before any benchmark, reset the serving engine's prefix cache, or restart the @@ -506,6 +506,32 @@ Use these offers when selecting fleet, backend, and hardware. To classify each backend's capabilities, fetch `https://dstack.ai/docs/concepts/backends.md` and classify from the fetched document, not from memory. VM-based backends are listed under `## VM-based` (they support idle instances and instance volumes). Kubernetes backend is listed under `## Container-based`, but supports instance volumes and thus is preferred over other container-based backends. SSH fleets can be treated as VM-based backends as they support both idle instances (its equivalent) and instance volumes. +## Fleet Topology + +If a fleet has `placement: cluster` and a CPU-only instance, that instance +must host a router (see `https://dstack.ai/docs/concepts/services/#router`), +and every trial and the final service run behind it. The GPU instances run +the serving engine as workers in one of two forms: aggregated, where each +worker handles both prefill and decode, or PD disaggregated, where the +workers are split into prefill and decode groups. Use `router: sglang` +(Shepherd Model Gateway, SMG) and connect the workers over gRPC, not HTTP. +Report the topology decision via `progress` before trial 1. + +Why: with an SMG router, request parsing, serialization, and tokenization +move from the serving engine to the router, and workers only process already +tokenized inputs, so latency improves just by introducing the router. With +multiple workers, the router improves performance in two ways. + +First, cache-aware routing tracks each worker's KV-cache state and sends a +request to the worker that already holds its prefix. + +Second, load modeling predicts how long a request will take to complete on +each worker and the router's `least_load` policy sends the request to the +worker predicted to finish it soonest. + +Connecting the workers over gRPC is required for SMG's cache-aware routing +and load modeling. + # Final Service Once the trials are over, pick the best trial that has not been verified yet @@ -565,9 +591,9 @@ trial benchmarks so that the results are comparable with each other. Attach to the service with `dstack attach `, which enables `ssh ` into the replica. -In case the service is using PD disaggregation, run the final benchmark -inside the router replica, directly against the router engine. Attach to it -via `dstack attach --replica --job `. +In case the service is using a router (see `## Fleet Topology`), run the +final benchmark inside the router replica, directly against the router. +Attach to it via `dstack attach --replica --job `. If the service or its benchmark cannot be completed, stop that service, pick the next-best trial, and repeat, until a service is verified or there From 947189620e7706b0d077659dd7933b64d32d4a89 Mon Sep 17 00:00:00 2001 From: Bihan Rana Date: Thu, 24 Sep 2026 13:05:51 +0545 Subject: [PATCH 2/2] Resolve Review Comments --- skills/dstack-prototyping/SKILL.md | 22 +++++++++-- .../presets/resources/system_prompt.md | 39 +++++-------------- 2 files changed, 28 insertions(+), 33 deletions(-) diff --git a/skills/dstack-prototyping/SKILL.md b/skills/dstack-prototyping/SKILL.md index 552d00dad..d054d8cb4 100644 --- a/skills/dstack-prototyping/SKILL.md +++ b/skills/dstack-prototyping/SKILL.md @@ -101,19 +101,33 @@ command, resources, cache, or model behavior needs to change, go back to a task. If the tested serving setup is still right and only the dstack service configuration is wrong, fix the configuration and submit the service again. -## PD disaggregation +## Router -If the intention is to use PD disaggregation: +If a fleet has `placement: cluster` and a CPU-only instance, it is generally +recommended to use a router configuration, regardless of whether the workers +are aggregated or PD disaggregated, with the router on the CPU-only instance. +Whenever possible, connect the workers over gRPC, not HTTP: with a gRPC +router, request parsing, serialization, and tokenization move from the +serving engine to the router, so latency improves just by introducing it. + +When using a router: - Use node groups for the task and replica groups for the service: tasks' node groups are the equivalent of services' replica groups. -- In both cases, you run a router and prefill/decode workers separately, and - you need to use a fleet with an interconnect (`placement: cluster`). - With tasks, still use `sleep infinity` even when using `groups` (set it in each group's `commands`; top-level `commands` is not allowed with `groups`), and run the actual commands on each node interactively over SSH. - When testing inference, call the router endpoint, not the workers directly (unless you want to test if they are alive). +- Look for "Prototyping services" in `https://dstack.ai/docs/concepts/tasks.md` + and "Router" in `https://dstack.ai/docs/concepts/services.md`. + +## PD disaggregation + +If the intention is to use PD disaggregation: + +- Follow `## Router`: the router and the prefill/decode workers run as + separate groups, and the fleet needs an interconnect (`placement: cluster`). - Look for "Node groups" and "PD disaggregation" in `https://dstack.ai/docs/concepts/tasks.md` and "Replica groups" and "PD disaggregation" in `https://dstack.ai/docs/concepts/services.md`. diff --git a/src/dstack/_internal/cli/services/presets/resources/system_prompt.md b/src/dstack/_internal/cli/services/presets/resources/system_prompt.md index 836e64587..1a11b74ab 100644 --- a/src/dstack/_internal/cli/services/presets/resources/system_prompt.md +++ b/src/dstack/_internal/cli/services/presets/resources/system_prompt.md @@ -261,7 +261,7 @@ mindful of which specific change was the root cause. ``` 2. In case the task is using node groups (a router group plus worker - groups, see `## Fleet Topology`), instead of a single `resources` it + groups, see `## Cluster Placement`), instead of a single `resources` it includes `groups`, and the fields are these and no others: ``` @@ -321,8 +321,8 @@ serving engine: use `dataset` and `concurrency``co `shared_prefix_tokens` from `constraints.json` and measure all trials the same way so that their results are comparable with each other. -In case the task is using a router (see `## Fleet Topology`), run benchmarks -via SSH inside the router node, directly against the router, never against a +In case the task is using a router (see `## Cluster Placement`), run benchmarks +via SSH inside the router node, directly against the router engine, never against a worker, so that results stay comparable across trials. @@ -506,31 +506,12 @@ Use these offers when selecting fleet, backend, and hardware. To classify each backend's capabilities, fetch `https://dstack.ai/docs/concepts/backends.md` and classify from the fetched document, not from memory. VM-based backends are listed under `## VM-based` (they support idle instances and instance volumes). Kubernetes backend is listed under `## Container-based`, but supports instance volumes and thus is preferred over other container-based backends. SSH fleets can be treated as VM-based backends as they support both idle instances (its equivalent) and instance volumes. -## Fleet Topology +## Cluster Placement -If a fleet has `placement: cluster` and a CPU-only instance, that instance -must host a router (see `https://dstack.ai/docs/concepts/services/#router`), -and every trial and the final service run behind it. The GPU instances run -the serving engine as workers in one of two forms: aggregated, where each -worker handles both prefill and decode, or PD disaggregated, where the -workers are split into prefill and decode groups. Use `router: sglang` -(Shepherd Model Gateway, SMG) and connect the workers over gRPC, not HTTP. -Report the topology decision via `progress` before trial 1. - -Why: with an SMG router, request parsing, serialization, and tokenization -move from the serving engine to the router, and workers only process already -tokenized inputs, so latency improves just by introducing the router. With -multiple workers, the router improves performance in two ways. - -First, cache-aware routing tracks each worker's KV-cache state and sends a -request to the worker that already holds its prefix. - -Second, load modeling predicts how long a request will take to complete on -each worker and the router's `least_load` policy sends the request to the -worker predicted to finish it soonest. - -Connecting the workers over gRPC is required for SMG's cache-aware routing -and load modeling. +If a fleet has `placement: cluster`, decide before trial 1 whether to use a +router, following `## Router` in the +`$/dstack-prototyping` skill, and +report the decision via `progress`. # Final Service @@ -591,8 +572,8 @@ trial benchmarks so that the results are comparable with each other. Attach to the service with `dstack attach `, which enables `ssh ` into the replica. -In case the service is using a router (see `## Fleet Topology`), run the -final benchmark inside the router replica, directly against the router. +In case the service is using a router (see `## Cluster Placement`), run the +final benchmark inside the router replica, directly against the router engine. Attach to it via `dstack attach --replica --job `. If the service or its benchmark cannot be completed, stop that service,