Skip to content

Run Kimi K2.6 NVFP4 Dynamo vLLM on b200-new / 在 b200-new 上运行 Kimi K2.6 NVFP4 Dynamo vLLM #2438

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
RohitNagraj wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Run Kimi K2.6 NVFP4 Dynamo vLLM on b200-new / 在 b200-new 上运行 Kimi K2.6 NVFP4 Dynamo vLLM #2438

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d6e0efe
chore: route Kimi K2.6 sweep to b200-new
RohitNagraj Jul 31, 2026
6f0341e
chore: update perf changelog PR link
RohitNagraj Jul 31, 2026
2550525
fix: configure b200-new launch environment
RohitNagraj Jul 31, 2026
7eb3acb
fix: stabilize b200-new image staging
RohitNagraj Jul 31, 2026
938b380
fix: use batch_1 for b200-new
RohitNagraj Jul 31, 2026
57faf6d
fix: use batch_2 for b200-new
RohitNagraj Jul 31, 2026
fc26091
revert: restore batch_1 for b200-new
RohitNagraj Jul 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 0 additions & 112 deletions ...node/srt-slurm-recipes/vllm/kimi-k2.6/b200-fp4/8k1k/disagg-b200-1p1d-dep4-dep8-c1024.yaml
View file
Open in desktop

This file was deleted.

15 changes: 1 addition & 14 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4981,7 +4981,7 @@ kimik2.6-fp4-b200-dynamo-vllm:
image: vllm/vllm-openai:v0.25.1
model: nvidia/Kimi-K2.6-NVFP4
model-prefix: kimik2.6
runner: b200-multinode
runner: b200-new
precision: fp4
framework: dynamo-vllm
router: { name: dynamo-router, version: "1.3.0.dev20260721" }
Expand Down Expand Up @@ -5045,19 +5045,6 @@ kimik2.6-fp4-b200-dynamo-vllm:
tp: 4
ep: 1
dp-attn: false
- conc-list: [1024]
prefill:
num-worker: 1
tp: 1
ep: 4
dp-attn: true
additional-settings:
- "CONFIG_FILE=recipes/vllm/kimi-k2.6/b200-fp4/8k1k/disagg-b200-1p1d-dep4-dep8-c1024.yaml"
decode:
num-worker: 1
tp: 1
ep: 8
dp-attn: true
- conc-list: [2048]
prefill:
num-worker: 1
Expand Down
10 changes: 10 additions & 0 deletions configs/runners.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,16 @@ labels:
- b200-dgxc-slurm_7
- b200-dgxc-slurm_8
- b200-dgxc-slurm_9
b200-new:
- b200-nscale-slurm_0
- b200-nscale-slurm_1
- b200-nscale-slurm_2
- b200-nscale-slurm_3
- b200-nscale-slurm_4
- b200-nscale-slurm_5
- b200-nscale-slurm_6
- b200-nscale-slurm_7
- b200-nscale-slurm_8
mi300x:
- mi300x-amds_00
- mi300x-amds_01
Expand Down
8 changes: 8 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5355,3 +5355,11 @@
- "Apply the accuracy-gated Kimi-K2.5 MXFP4 settings: tuned AITER MXFP4 MoE, fused shared experts, FP8 KV cache, block size 16, 16384 batched tokens, 512 sequences, async scheduling, gpu-memory-utilization 0.85 (headroom for CUDA-graph capture on MI355X), and the AITER BF16 GEMM path"
- "Extend the TP4 and TP8 8k1k concurrency sweep from 64 to 128 (1k1k deprecated per #2263)"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2213

- config-keys:
- kimik2.6-fp4-b200-dynamo-vllm
description:
- "Run the Kimi K2.6 NVFP4 B200 Dynamo vLLM disaggregated configuration on the b200-new runner."
- "Remove the 1P/1D DEP4-to-DEP8 concurrency-1024 configuration."
- "Image: vllm/vllm-openai:v0.25.1"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2438
Comment on lines 5357 to +5365

@claude claude Bot Jul 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The new perf-changelog.yaml entry for kimik2.6-fp4-b200-dynamo-vllm ends with an unfilled placeholder link: pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX instead of the real PR number. Please replace XXX with 2438 to match every other entry in this append-only changelog.

Extended reasoning...

The newly appended block in perf-changelog.yaml (lines 5357-5365) documents the routing change to b200-new, but its pr-link field was never filled in with the actual PR number and instead ships with the literal placeholder https://github.com/SemiAnalysisAI/InferenceX/pull/XXX.

This is verifiable directly against the committed content rather than the rendered PR diff. Running git log --oneline -- perf-changelog.yaml shows commit d6e0efe ("chore: route Kimi K2.6 sweep to b200-new") as the tip of history for this file, with a subject line that matches this PR's title exactly and — unlike the surrounding merged commits ((#2213), (#2360), (#2312)) — carries no merge-PR suffix, confirming it is this PR's own commit rather than a later, unrelated change. Both git show HEAD:perf-changelog.yaml and the on-disk working tree confirm the last line of the new block reads pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX.

One verifier raised a plausible-sounding objection: the preloaded PR diff shown in review context displays pull/2438 on that line, and standard guidance says to prefer the diff over a possibly-diverged working tree. However, that guidance exists to handle the case where the checkout is ahead of the PR (later commits touched the file). That doesn't apply here — HEAD is the PR's commit, there is no later commit to discount, and the diff-displayed 2438 does not match what git actually recorded as committed. In other words, the "2438" appears to be a harness-level auto-substitution for display purposes, not the real content of the change. Trusting git show/working-tree content (the actual bytes that would land in the repo) over the rendered diff is correct in this specific case, since they conflict and only one can be what actually gets merged.

Step-by-step proof:

  1. git log --oneline -- perf-changelog.yaml → top commit is d6e0efe, no (#NNNN) suffix, subject matches this PR's title.
  2. git show HEAD:perf-changelog.yaml | tail -9 → last line of the new kimik2.6-fp4-b200-dynamo-vllm block is pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX.
  3. tail -9 perf-changelog.yaml (working tree) → identical .../pull/XXX line, confirming it is not a stale artifact of an intermediate git state.
  4. Every other entry in the file (e.g. the immediately preceding one ending .../pull/2213, and others at .../pull/2360, .../pull/2312) has a real, resolved PR number — this is the only entry left as a placeholder.

Impact is purely cosmetic: perf-changelog.yaml is an append-only audit/documentation log, and this field is not consumed by any routing, sweep, or runtime logic — nothing breaks functionally. But it leaves a dead link (.../pull/XXX 404s) in an otherwise-accurate historical record, which is a real, if minor, correctness gap. The fix is a one-line change: replace XXX with 2438 on line 5365.

8 changes: 4 additions & 4 deletions runners/launch_b200-dgxc.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
#!/usr/bin/bash

# System-specific configuration for B200 DGXC Slurm cluster
SLURM_PARTITION="gpu-2"
SLURM_ACCOUNT="benchmark"
SLURM_PARTITION="${SLURM_PARTITION:-gpu-2}"
SLURM_ACCOUNT="${SLURM_ACCOUNT:-benchmark}"

set -x

Expand Down Expand Up @@ -53,7 +53,7 @@ elif [[ $MODEL_PREFIX == "kimik2.5" && $PRECISION == "fp4" ]]; then
export MODEL_PATH="/lustre/fsw/models/Kimi-K2.5-NVFP4"
export SRT_SLURM_MODEL_PREFIX="kimik2.5-fp4"
elif [[ $MODEL_PREFIX == "kimik2.6" && $PRECISION == "fp4" ]]; then
export MODEL_PATH="/lustre/fsw/models/Kimi-K2.6-NVFP4"
export MODEL_PATH="${MODEL_PATH:-/lustre/fsw/models/Kimi-K2.6-NVFP4}"
export SRT_SLURM_MODEL_PREFIX="kimi-k2.6-nvfp4"
elif [[ $MODEL_PREFIX == "minimaxm2.5" && $PRECISION == "fp8" ]]; then
export MODEL_PATH="/lustre/fsw/models/MiniMax-M2.5"
Expand Down Expand Up @@ -213,7 +213,7 @@ if [[ "$IS_MULTINODE" == "true" ]]; then
local lock_file="${lock_dir}/${image_key}.lock"

(
flock -w 600 9 || { echo "Failed to acquire lock for $squash_file" >&2; exit 1; }
flock -w "${B200_SQUASH_LOCK_TIMEOUT:-600}" 9 || { echo "Failed to acquire lock for $squash_file" >&2; exit 1; }
if unsquashfs -l "$squash_file" > /dev/null 2>&1; then
echo "Squash file already exists and is valid, skipping import: $squash_file"
else
Expand Down
14 changes: 14 additions & 0 deletions runners/launch_b200-nscale-slurm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#!/usr/bin/bash

export SLURM_PARTITION="batch_1"
export SLURM_ACCOUNT="benchmark"
export MODEL_PATH="/scratch/models/Kimi-K2.6-NVFP4"

export B200_SQUASH_DIR="/data/home/sa-shared/containers"
export B200_SQUASH_LOCK_TIMEOUT="3600"
mkdir -p "$B200_SQUASH_DIR" || {
echo "Unable to prepare nscale runner storage" >&2
exit 1
}

exec bash "$(dirname "${BASH_SOURCE[0]}")/launch_b200-dgxc.sh" "$@"
Loading