Skip to content

Kimi K3 B200 Day 0 AgentX Pipeline Parallelism (rebased onto main; non-DSpark as it doesn't work with Pipeline yet, offloading & TP16 and DEP8PP2 to be done in followup PR) #2400

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
functionstackx wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Kimi K3 B200 Day 0 AgentX Pipeline Parallelism (rebased onto main; non-DSpark as it doesn't work with Pipeline yet, offloading & TP16 and DEP8PP2 to be done in followup PR) #2400

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion MODELS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ This document tracks every model benchmarked by InferenceX-e2e: when it was adde
| Model architecture class | Prefix | Date added | Active scenarios | Deprecated scenarios |
|---|---|---|---|---|
| Qwen3.8 2.4T | `qwen3.8` | TBD | Agentic coding | |
| Kimi-K3 | `kimik3` | 2026-07-27 | Agentic coding | |
| Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | Agentic coding | |
| GLM-5.2 | `glm5.2` | 2026-07-18 ([#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)) | Agentic coding | |
| MiniMax-M3 | `minimaxm3` | 2026-06-12 ([#1724](https://github.com/SemiAnalysisAI/InferenceX/pull/1724)) | Single-turn 8k1k, Agentic coding | Single-turn 1k1k |
| DeepSeek-V4-Pro | `dsv4` | 2026-04-24 ([#1130](https://github.com/SemiAnalysisAI/InferenceX/pull/1130)) | Single-turn 8k1k, Agentic coding | Single-turn 1k1k |
Expand Down
2 changes: 1 addition & 1 deletion MODELS_zh.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
| 模型架构类别 | 前缀 | 加入日期 | 启用场景 | 已弃用场景 |
|---|---|---|---|---|
| Qwen3.8 2.4T | `qwen3.8` | 待定 | 智能体编码 | |
| Kimi-K3 | `kimik3` | 2026-07-27 | 智能体编码 | |
| Kimi-K3 | `kimik3` | 2026-07-27 ([#2391](https://github.com/SemiAnalysisAI/InferenceX/pull/2391)) | 智能体编码 | |
| GLM-5.2 | `glm5.2` | 2026-07-18（[#2268](https://github.com/SemiAnalysisAI/InferenceX/pull/2268)） | 智能体编码 | |
| MiniMax-M3 | `minimaxm3` | 2026-06-12（[#1724](https://github.com/SemiAnalysisAI/InferenceX/pull/1724)） | 单轮 8k1k、智能体编码 | 单轮 1k1k |
| DeepSeek-V4-Pro | `dsv4` | 2026-04-24（[#1130](https://github.com/SemiAnalysisAI/InferenceX/pull/1130)） | 单轮 8k1k、智能体编码 | 单轮 1k1k |
Expand Down
140 changes: 140 additions & 0 deletions benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
name: "kimik3-vllm-agg-b200-tp8pp2-retention0-agentic"

# Kimi-K3 MXFP4 B200 AGGREGATED TP8 x PP2 agentic recipe (2 nodes / 16 GPUs).
# The native MXFP4 checkpoint (2.8T total params, ~1.4TB of weights) does not
# fit one 8xB200 node, so TP8 shards attention/dense (/8) and PP2 splits the
# 93 layers (/2) across 16 GPUs. Plain TP (NOT TEP): expert parallelism is
# deliberately off, so the 896 routed experts are TP-sharded inside each
# pipeline stage. Node allocation = tp*pp/gpus_per_node = 8*2/8 = 2 nodes.
# Aggregated (single worker, decode num-worker 0) — no P/D split, no NIXL.
# VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION fuses the K3 LatentMoE tail path in
# the kimi-k3 bring-up image.
model:
path: "kimik3"
container: "vllm/vllm-openai:kimi-k3"
precision: "fp4"

identity:
model:
repo: "moonshotai/Kimi-K3"
container:
image: "vllm/vllm-openai:kimi-k3"

# Direct vLLM serving (frontend.type: vllm, srt-slurm PR #278 + the
# multinode extension on functionstackx/srt-slurm-nv branch
# klaud/direct-vllm-multinode): `vllm serve` owns the OpenAI port itself, so
# no Dynamo frontend/worker is involved and no dynamo install is needed.
dynamo:
install: false

slurm:
time_limit: "8:00:00"

health_check:
interval_seconds: 10
max_attempts: 1440

Check warning on line 35 in benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yaml

View check run for this annotation

Claude / Claude Code Review

Recipe max_attempts=1440 silently clamped to 720 by launcher sed 

This recipe sets `health_check.max_attempts: 1440` (a deliberate 4h window for the ~1.4TB MXFP4 weight load), but `runners/launch_b200-dgxc.sh` unconditionally runs a sed that rewrites any `max_attempts: N` line to 720 before calling `srtctl apply` on every multinode config. This silently halves the intended window to 2h, making the 1440 value dead/misleading config (harmless today since 720×10s=2h still comfortably exceeds `VLLM_ENGINE_READY_TIMEOUT_S=3600`, and the validation sweep succeeded u
Comment on lines +33 to +35

@claude claude Bot Jul 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 This recipe sets health_check.max_attempts: 1440 (a deliberate 4h window for the ~1.4TB MXFP4 weight load), but runners/launch_b200-dgxc.sh unconditionally runs a sed that rewrites any max_attempts: N line to 720 before calling srtctl apply on every multinode config. This silently halves the intended window to 2h, making the 1440 value dead/misleading config (harmless today since 720×10s=2h still comfortably exceeds VLLM_ENGINE_READY_TIMEOUT_S=3600, and the validation sweep succeeded under the clamped window).

Extended reasoning...

What happens: The new recipe agg-b200-tp8pp2-agentic.yaml declares health_check.interval_seconds: 10 and max_attempts: 1440, i.e. a 4-hour health-check budget, with an inline comment explicitly framing this as generous headroom for loading ~1.4TB of MXFP4 weights off shared Lustre. However, runners/launch_b200-dgxc.sh contains this pre-existing line (unchanged by this PR, but now exercised by this PR's new multinode recipe):

sed -i 's/^  max_attempts: [0-9]*/  max_attempts: 720/' "${CONFIG_FILE%%:*}"\n```

This sed runs unconditionally for every multinode job right before `srtctl apply`. It matches any line beginning with exactly two spaces followed by `max_attempts: <digits>` — which matches this recipe's `  max_attempts: 1440` line at line 35 — and rewrites it to `720` regardless of what the recipe author set.

**Why existing code doesn't prevent it:** The sed was added earlier (per its own comment) to *bump up* a smaller 360-attempt default to 720 for large models like DSR1-FP8. It was written as an unconditional overwrite rather than a floor/max, so it works correctly as a bump only when the recipe's own value is below 720; it silently clamps downward when a recipe (like this one, loading the largest checkpoint in the repo) deliberately declares something larger.

**Step-by-step proof:**
1. Recipe file declares (lines 33-35):
   ```yaml\n   health_check:\n     interval_seconds: 10\n     max_attempts: 1440\n   ```
2. `launch_b200-dgxc.sh` is invoked for this config with `IS_MULTINODE=true`.
3. Before `srtctl apply`, the script runs `sed -i 's/^  max_attempts: [0-9]*/  max_attempts: 720/' "${CONFIG_FILE%%:*}"`.
4. The regex `^  max_attempts: [0-9]*` matches `  max_attempts: 1440` (two-space indent, digits after the colon), rewriting the line in place to `  max_attempts: 720`.
5. `srtctl apply` is then called on the *mutated* file, so the effective health-check window becomes `720 * 10s = 7200s` (2h), not the author's intended `1440 * 10s = 14400s` (4h).

**Impact:** Today this is benign — the recipe's own `VLLM_ENGINE_READY_TIMEOUT_S=3600` (1h) is the binding constraint on engine readiness and sits well under even the clamped 2h health-check window, and the PR's linked validation run (30385435921) completed successfully with the sed already in effect. So no concrete failure results from merging as-is. But the `1440` value is dead config: it communicates author intent ("I need up to 4h for this 1.4TB load") that is silently discarded, and a future reader/maintainer could reasonably believe the recipe controls its own timeout when it does not. If a future checkpoint or FS slowdown pushes weight-load time past 2h, the author's actual safety margin will not be what the recipe file claims.

**Suggested fix:** Either drop the `1440` value in the recipe (since it's currently a no-op) and note that the effective cap is the launcher's 720, or change the launcher's sed to only raise the value (e.g. compare and take the max) rather than unconditionally overwriting it, so recipes can opt into a longer window than the 720 default without being silently clamped.

</details>

<!-- bhfn:1:nit -->


resources:
gpu_type: "b200"
gpus_per_node: 8
agg_nodes: 2
agg_workers: 1
gpus_per_agg: 16

infra:
etcd_nats_dedicated_node: false
nats_max_payload_mb: 32

frontend:
# Direct vLLM OpenAI server (srt-slurm PR #278): the vllm serve leader owns
# the public port; rank-1 runs a headless engine worker (vLLM-native
# multi-node TP8xPP2 via --master-addr/--nnodes/--node-rank, carried by the
# functionstackx/srt-slurm-nv klaud/direct-vllm-multinode branch).
type: vllm
enable_multiple_frontends: false

backend:
type: vllm
connector: null
aggregated_environment:
VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION: "1"
VLLM_SERVER_DEV_MODE: "1"
# ~1.4TB of MXFP4 weights off shared Lustre: keep the engine-ready window
# generous, and let one long AgentX request hold a PP stage beyond vLLM's
# 300-second model-execution default.
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS: "1800"
# Prefix-cache retention (variant G): 0, on the otherwise-unchanged
# GPU-resident variant D config. Any positive value must be a multiple of
# Kimi-K3's KDA-hybrid scheduler_block_size (3145728; the GB recipes'
# 32768 is hard-rejected at engine init — verified on this PR family), so
# 0 is the only value below one 3.1M-token scheduler block.
VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "0"
# No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: the GB200/GB300 AgentX value
# (32768) hard-fails engine init on Kimi-K3 — the KDA hybrid gives it a
# scheduler_block_size of 3145728 and the interval must be a multiple of
# it ("VLLM_PREFIX_CACHE_RETENTION_INTERVAL (32768) must be non-negative
# and a multiple of scheduler_block_size (3145728)"). Default retention
# served fine in earlier runs.

Check warning on line 78 in benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yaml

View check run for this annotation

Claude / Claude Code Review

Contradictory stale comment for VLLM_PREFIX_CACHE_RETENTION_INTERVAL 

The comment block on lines 73-78 of the new recipe file is stale and contradicts the line directly above it: line 72 sets `VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "0"`, but the leftover comment opens with 'No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: ...' and closes with 'Default retention served fine in earlier runs' — text describing a mutually-exclusive earlier variant where the var was left unset. This is comment-only with zero functional impact, but should be deleted so a future maintainer doesn'
Comment on lines +72 to +78

@claude claude Bot Jul 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The comment block on lines 73-78 of the new recipe file is stale and contradicts the line directly above it: line 72 sets VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "0", but the leftover comment opens with 'No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: ...' and closes with 'Default retention served fine in earlier runs' — text describing a mutually-exclusive earlier variant where the var was left unset. This is comment-only with zero functional impact, but should be deleted so a future maintainer doesn't mistakenly believe the var is unset.

Extended reasoning...

What the bug is

agg-b200-tp8pp2-agentic.yaml sets VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "0" on line 72, with a correct explanatory comment on lines 67-71 describing why 0 is the only valid value below the 3.1M-token KDA-hybrid scheduler block (any positive value must be a multiple of scheduler_block_size = 3145728, and the GB200/GB300 AgentX value 32768 hard-fails engine init).

Immediately below that, lines 73-78 are a second comment block that begins "No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: the GB200/GB300 AgentX value (32768) hard-fails engine init on Kimi-K3..." and ends "Default retention served fine in earlier runs." This block describes the opposite decision — leaving the variable unset entirely and falling back to the vLLM default retention interval.

Why this is contradictory

Both blocks agree on the underlying fact (32768 is rejected, values must be multiples of 3145728), but they describe two mutually exclusive configuration choices:

  • Lines 67-72: "we pin the var to 0."
  • Lines 73-78: "we do NOT set the var at all (No VLLM_PREFIX_CACHE_RETENTION_INTERVAL), relying on the default."

Only one of these is actually reflected in the YAML — the var IS set, to "0" (line 72). The second block is dead, leftover text from an earlier recipe variant (the perf-changelog entry documents at least two prior experiment variants, D and G, that iterated on this exact setting) that was never deleted when the approach changed from "leave unset" to "pin to 0".

Why nothing catches this

This is a plain YAML comment inside aggregated_environment, so it has no effect on parsing or runtime behavior — VLLM_PREFIX_CACHE_RETENTION_INTERVAL resolves to "0" regardless of what the comment says. No linter or CI check inspects comment prose for internal consistency, so this only surfaces via manual code review.

Impact

Zero functional/runtime impact — the engine will boot with the interval correctly pinned to 0 either way. The risk is purely to future maintainability: an engineer skimming this file later (e.g., while porting the recipe to a new GPU SKU or troubleshooting a prefix-cache issue) could read the second block, conclude the variable is unset, and either add a duplicate/conflicting entry or misdiagnose an unrelated issue based on a false premise about the current config.

Proof / walkthrough

  1. Read line 72: VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "0" — the key is present and set.
  2. Read lines 73-78 immediately after: the comment text opens "No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: the GB200/GB300 AgentX value (32768) hard-fails engine init..." — grammatically and semantically this asserts the variable is absent/unset.
  3. The same block closes with "Default retention served fine in earlier runs," which only makes sense as a rationale for not setting the variable (i.e., relying on the built-in default) — directly at odds with the pinned "0" one line above.
  4. Since YAML has exactly one VLLM_PREFIX_CACHE_RETENTION_INTERVAL key-value pair on line 72, and comments are inert, the true, executed behavior is "the var is 0." The second comment block is therefore stale narration that no longer matches the file it sits in.

Suggested fix

Delete the stale second comment block (lines 73-78) entirely; the accurate rationale is already fully captured by lines 67-71.

NCCL_CUMEM_ENABLE: "1"
TILELANG_CLEANUP_TEMP_FILES: "1"
UCX_MEMTYPE_CACHE: "n"
UCX_MEMTYPE_REG_WHOLE: "n"
UCX_NET_DEVICES: "mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_10:1,mlx5_11:1"
HF_HUB_CACHE: "/hf_hub_cache"
HUGGINGFACE_HUB_CACHE: "/hf_hub_cache"
vllm_config:
aggregated:
served-model-name: "moonshotai/Kimi-K3"
tensor-parallel-size: 8
pipeline-parallel-size: 2
trust-remote-code: true
load-format: fastsafetensors
moe-backend: auto
# 0.90, not 0.95: the flashinfer trtllm MXFP4 MoE kernel allocates a
# ~1.6 GiB runtime workspace OUTSIDE vLLM's memory pool on the first
# forward; at 0.95 a 178 GiB B200 has only ~1.35 GiB free and the first
# warmup request OOMs (seen on the dynamo-frontend variants). 0.90
# matches the GB200/GB300 agentic recipes.
gpu-memory-utilization: 0.90
no-enable-flashinfer-autotune: true
# kimi_k3 parsers via the native vllm serve OpenAI-frontend flags —
# legitimate here because this recipe serves directly with vllm serve
# (frontend.type: vllm), not through the dynamo worker entrypoint that
# rejects them.
enable-auto-tool-choice: true
tool-call-parser: kimi_k3
reasoning-parser: kimi_k3
# No explicit max-model-len: let vLLM derive the native 1M window from
# the model config (agentic trajectories blow past any small cap, and
# K3's KDA layers keep per-token KV small — only the 24 gated-MLA
# layers hold cache). Prefix caching stays on (default) for trajectory
# reuse. Cap prefill chunks so a single long request cannot OOM a
# pipeline stage; let vLLM pick max-num-seqs.
max-num-batched-tokens: 8192

sbatch_directives:
segment: "1"

srun_options:
container-remap-root: ""

benchmark:
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh
env:
INFMAX_CONTAINER_WORKSPACE: "/infmax-workspace"
RESULT_DIR: "/logs/agentic"
PORT: "8000"
# Keep the aggregate worker in the multinode result schema so ingestion
# uses the zero decode-worker count instead of duplicating TP into P and D.
IS_MULTINODE: "true"
# aiperf's conv-aware routing emits nvext.session_control, a removed POC
# field this dynamo build 400-rejects at warmup (schema moved to
# router/routing_constraints/agent_hints). Same opt-out as the GB300
# aggregate AgentX recipes — and with a single aggregate worker there is
# no P/D routing to bind anyway.
AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0"
AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache"
HF_HUB_CACHE: "/hf_hub_cache"
WEKA_LOADER_OVERRIDE: "semianalysis_cc_traces_weka_062126"
45 changes: 45 additions & 0 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8388,3 +8388,48 @@ qwen3.5-fp8-gb200-dynamo-sglang-mtp:
tp: 16
ep: 16
dp-attn: true

# Kimi-K3 MXFP4 B200 aggregated vLLM via Dynamo (TP8 x PP2, 2 nodes / 16
# GPUs), agentic bring-up. The native MXFP4 checkpoint (2.8T total params,
# ~1.4TB weights) does not fit one 8xB200 node, so TP8 shards attention/dense
# and PP2 splits layers. Plain TP (NOT TEP): ep 1, no expert parallelism —
# the 896 routed experts are TP-sharded within each pipeline stage. Node
# count = tp*pp/gpus_per_node = 8*2/8 = 2. Aggregated (prefill num-worker 1 +
# decode num-worker 0, RECIPES.md section 5) — the single worker serves both
# phases, so no P/D KV transfer. Dedicated kimi-k3 vLLM bring-up image with
# VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1 and the kimi_k3 tool-call/reasoning
# parsers.
# Recipe: benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yaml
kimik3-fp4-b200-dynamo-vllm-agentic:
image: vllm/vllm-openai:kimi-k3
model: moonshotai/Kimi-K3
model-prefix: kimik3
runner: cluster:b200-dgxc
precision: fp4
# framework stays dynamo-vllm for launcher routing, but this variant serves
# DIRECTLY with vllm serve (srt-slurm PR #278 frontend.type: vllm + the
# InferenceX multinode patch) — no dynamo frontend/worker/router involved.
framework: dynamo-vllm
multinode: true
disagg: false
scenarios:
agentic-coding:
- search-space:
- spec-decoding: none
conc-list: [1, 2, 4, 8, 16, 32]
prefill:
num-worker: 1
tp: 8
pp: 2
ep: 1
dp-attn: false
additional-settings:
- "CONFIG_FILE=recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yaml"
# The aggregate worker also performs decode; keep the decode worker
# count at zero so result aggregation counts the 16 GPUs only once.
decode:
num-worker: 0
tp: 8
pp: 2
ep: 1
dp-attn: false
16 changes: 16 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5159,3 +5159,19 @@
- "Bring-up validated in run 30326393603: all 12 configs green, zero ServerDisconnectedError after the keep-alive fix. GPU KV resolves to 42.23 GiB / 3,249,215 tokens, i.e. ~3.1 max-length requests, against MAX_NUM_SEQS = 2*CONC."
- "Measured behaviour: below conc 8 the GPU-resident and DRAM arms are within run-to-run noise (1-5%). At conc 16 and 24 the GPU-resident arm thrashes -- prefix cache hit rate 2.7%, TTFT p50 86s and 191s, 49.6 and 54.9 output tok/s -- because the working set exceeds GPU KV and prefixes are recomputed. The DRAM arm holds TTFT p50 0.85s and 6.2s for 245.0 and 260.6 output tok/s (4-5x), with the CPU tier serving a 62% external prefix cache hit rate at conc 24. The high-conc GPU-resident points are retained deliberately as the honest baseline that makes the offload gain legible."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2371

- config-keys:
- kimik3-fp4-b200-dynamo-vllm-agentic
description:
- "Add Kimi-K3 MXFP4 B200 aggregated multinode Dynamo-vLLM agentic-coding bring-up (new model on B200; first kimik3 benchmark config)"
- "Aggregated TP8 x PP2 across 2 B200 nodes (16 GPUs), plain TP (NOT TEP: ep 1, no enable-expert-parallel) — the native MXFP4 checkpoint (2.8T total params, ~1.4TB weights) does not fit one 8xB200 node, so TP8 shards attention/dense and PP2 splits the 93 layers. Aggregated mode (prefill num-worker 1 + decode num-worker 0, RECIPES.md section 5): one worker serves prefill and decode, no P/D KV transfer"
- "Dedicated bring-up image vllm/vllm-openai:kimi-k3 with VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1 (fuses the K3 LatentMoE tail path), --load-format fastsafetensors, --moe-backend auto, --gpu-memory-utilization 0.90 (0.95 OOMs: the flashinfer trtllm MXFP4 MoE kernel allocates a ~1.6 GiB runtime workspace outside vLLM's pool on the first forward), --no-enable-flashinfer-autotune, --trust-remote-code, --enable-auto-tool-choice, --tool-call-parser kimi_k3, --reasoning-parser kimi_k3 (agentic experiment Variant D: the OpenAI-frontend parser flags are legitimate here because serving is DIRECT vllm serve, not the dynamo worker entrypoint that rejects them)"
- "No explicit max-model-len (vLLM derives the native 1M window from the model config; K3's KDA layers keep per-token KV small — only the 24 gated-MLA layers hold cache), prefix caching on for trajectory reuse, max-num-batched-tokens 8192 so a single long prefill cannot OOM a pipeline stage; conc 1/8/16/32"
- "DIRECT vLLM serving via srt-slurm PR #278 (kylliang/direct-aggregate-vllm, frontend.type: vllm): vllm serve owns the OpenAI port itself, removing the dynamo frontend/worker/router entirely (dynamo install: false) and with it the kimi_k3 tiktoken tokenizer gap that 404'd every request on dynamo <=1.2.1. PR #278 validates single-node only, so the launcher clones the functionstackx/srt-slurm-nv klaud/direct-vllm-multinode branch, which carries PR #278 plus the multi-node extension (vLLM-native --master-addr/--nnodes/--node-rank serve, headless non-leader ranks) for the 2-node TP8xPP2 topology"
- "Model pre-staged at /lustre/fsw/models/Kimi-K3 (moonshotai/Kimi-K3); launcher launch_b200-dgxc.sh gains the kimik3/fp4 model-path mapping, pins the agentic srt-slurm base to upstream NVIDIA/srt-slurm v1.0.36 (validated in #2302/#2341; replaces the cquil11/srt-slurm-nv fork branch whose older srtctl schema rejects newer recipe fields such as benchmark.aiperf_server_metrics), overlays the kimi-k3 agentic recipes onto the clone, and adds the agentic default_mounts (/aiperf_mmap_cache, /hf_hub_cache) already used by the GB200/GB300 agentic paths"
- "aiperf conv-aware routing disabled (AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING=0, same opt-out as the GB300 aggregate AgentX recipes): aiperf's nvext.session_control is a removed POC field this dynamo build 400-rejects at warmup (fifth sweep attempt: tokenizer registered, engine served, all 5 warmup requests 400'd); with a single aggregate worker there is no P/D routing to bind"
- "In-container vLLM patch via setup_script kimi-k3-container-deps.sh: the kimi-k3 image's first decode step crashes in the KDA hybrid-state postprocess (vllm/v1/worker/gpu/model_states/mamba_hybrid.py postprocess_state, IndexError: index_fill_(): Expected dtype int64 for index — torch requires an int64 index but the runner passes the int32 idx_mapping; sixth sweep attempt, first warmup request 500s then the model 503s). The patch coerces the index with .long(), is idempotent, and refuses to run if the image layout changed"
- "Agentic experiment Variant G (of the #2359 direct-vllm Variant D): identical GPU-resident config plus VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0. Any positive value must be a multiple of Kimi-K3's KDA-hybrid scheduler_block_size 3145728 (the GB recipes' 32768 is hard-rejected at engine init, verified earlier on this PR family), so 0 is the only setting below one 3.1M-token scheduler block"
- "Agentic experiment Variant K (probe, of the #2374 Variant G): REMOVES the kimi-k3-container-deps.sh in-container patch (setup_script, script file, and launcher copy) to verify whether the mamba_hybrid index_fill_ dtype patch is still required by the current vllm/vllm-openai:kimi-k3 image — expected to fail at the first decode step if it is"
- "Recipe: benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yaml on the cluster:b200-dgxc pool"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2391
36 changes: 34 additions & 2 deletions runners/launch_b200-dgxc.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,10 @@ elif [[ $MODEL_PREFIX == "minimaxm3" && $PRECISION == "fp4" ]]; then
# NVFP4 checkpoint, pre-staged on the b200-dgxc scratch tree.
export MODEL_PATH="/scratch/fsw/models/MiniMax-M3-NVFP4"
export SRT_SLURM_MODEL_PREFIX="minimax-m3-nvfp4"
elif [[ $MODEL_PREFIX == "kimik3" && $PRECISION == "fp4" ]]; then
# Native MXFP4 checkpoint, pre-staged on the SRE-managed Lustre tree.
export MODEL_PATH="/lustre/fsw/models/Kimi-K3"
export SRT_SLURM_MODEL_PREFIX="kimik3"
else
echo "Unsupported model prefix/precision: $MODEL_PREFIX/$PRECISION"
echo "Available models under /lustre/fsw/models:"
Expand Down Expand Up @@ -104,9 +108,20 @@ if [[ "$IS_MULTINODE" == "true" ]]; then
fi

# TODO(CJQ): make first class upon srt-slurm upstream refactor
if [[ "$IS_AGENTIC" == "1" ]]; then
git clone --branch cam/sa-submission-q2-2026 --single-branch https://github.com/cquil11/srt-slurm-nv.git "$SRT_REPO_DIR"
if [[ "$IS_AGENTIC" == "1" && $MODEL_PREFIX == "kimik3" ]]; then
# Direct-vLLM agentic experiment (Variant D): srt-slurm PR #278
# (kylliang/direct-aggregate-vllm) adds frontend.type: vllm — `vllm
# serve` owns the OpenAI port itself, no Dynamo layer. The fork branch
# carries PR #278 plus the multi-node extension (vLLM-native
# --master-addr/--nnodes/--node-rank serve + headless non-leader
# ranks) so the 2-node TP8xPP2 topology can run.
git clone --branch klaud/direct-vllm-multinode --single-branch https://github.com/functionstackx/srt-slurm-nv.git "$SRT_REPO_DIR" || exit 1
cd "$SRT_REPO_DIR" || exit 1
if [[ $MODEL_PREFIX == "kimik3" ]]; then
mkdir -p recipes/vllm/kimi-k3/agentic || exit 1
cp -rT "$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic" \
recipes/vllm/kimi-k3/agentic || exit 1
fi
elif [[ $FRAMEWORK == "dynamo-vllm" && $MODEL_PREFIX == "dsv4" ]]; then
git clone https://github.com/NVIDIA/srt-slurm.git "$SRT_REPO_DIR"
cd "$SRT_REPO_DIR" || exit 1
Expand Down Expand Up @@ -207,6 +222,22 @@ if [[ "$IS_MULTINODE" == "true" ]]; then
export OSL="$OSL"
export EVAL_ONLY="${EVAL_ONLY:-false}"

# Agentic runs bind-mount two persistent caches into every worker
# container (Lustre, shared across nodes): aiperf's content-addressed
# dataset mmap cache and the HF hub cache holding the trace dataset
# download. The container-side paths are referenced by the agentic
# recipes' benchmark.env (AIPERF_DATASET_MMAP_CACHE_DIR=/aiperf_mmap_cache,
# HF_HUB_CACHE=/hf_hub_cache).
DEFAULT_MOUNTS_BLOCK=""
if [[ "$IS_AGENTIC" == "1" ]]; then
HF_HUB_CACHE_HOST_PATH="/lustre/fsw/gharunners/hf-hub-cache"
mkdir -p "$AIPERF_MMAP_CACHE_HOST_PATH" "$HF_HUB_CACHE_HOST_PATH"
chmod 777 "$AIPERF_MMAP_CACHE_HOST_PATH" "$HF_HUB_CACHE_HOST_PATH" 2>/dev/null || true
DEFAULT_MOUNTS_BLOCK="default_mounts:
${AIPERF_MMAP_CACHE_HOST_PATH}: /aiperf_mmap_cache
${HF_HUB_CACHE_HOST_PATH}: /hf_hub_cache"
fi

# Create srtslurm.yaml for srtctl (used by both frameworks)
SRTCTL_ROOT="${GITHUB_WORKSPACE}/${SRT_REPO_DIR}"
echo "Creating srtslurm.yaml configuration..."
Expand Down Expand Up @@ -234,6 +265,7 @@ containers:
"${IMAGE}": "${SQUASH_FILE}"
nginx-sqsh: "${NGINX_SQUASH_FILE}"
use_exclusive_sbatch_directive: true
${DEFAULT_MOUNTS_BLOCK}
EOF

echo "Generated srtslurm.yaml:"
Expand Down
Loading