Kimi K3 B200 Day 0 AgentX Pipeline Parallelism (rebased onto main; non-DSpark as it doesn't work with Pipeline yet, offloading & TP16 and DEP8PP2 to be done in followup PR) - #2400
Conversation
) Same content as #2391, replayed onto current main as a single commit. #2391 branched from de6ba0b and had accumulated a merge commit plus a conflict against main, leaving it CONFLICTING/DIRTY. Aggregated TP8 x PP2 across 2 B200 nodes (16 GPUs) for the native MXFP4 checkpoint (2.8T total params, ~1.4TB weights), which does not fit one 8xB200 node: TP8 shards attention/dense and PP2 splits the 93 layers. Plain TP, not TEP - ep 1, no expert parallelism, so the 896 routed experts are TP-sharded inside each pipeline stage. Direct vLLM serving via frontend.type: vllm, no dynamo frontend/worker/router. Conflict resolution: main removed the duplicate perf-changelog entry for #2386 in 69af6a4 (revert #2396, preserving the original #2371 entry). That branch predates the revert and re-added the duplicate, so this keeps main's state and appends only the new #2391 entry. All five other files are byte-identical to #2391. Validated by sweep run 30385435921 (conclusion: success). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Beyond the inline findings, I checked whether narrowing the IS_AGENTIC branch to && $MODEL_PREFIX == "kimik3" in runners/launch_b200-dgxc.sh regresses other agentic multinode jobs on the b200-dgxc cluster — it doesn't, since every other agentic config in nvidia-master.yaml on that runner (dsv4, qwen3.5, kimik2.5) is multinode: false and never enters this branch today.
Extended reasoning...
This is a rebase of already-validated #2391 (sweep run 30385435921 succeeded on this exact config), with the only intentional content difference being a perf-changelog merge-conflict resolution. Both inline nits are comment/config artifacts with no observed functional impact under the validated run. I additionally traced the restructured IS_AGENTIC conditional in the launcher script against every other agentic config on the b200-dgxc cluster to rule out a regression to existing models, and confirmed none currently take the multinode+agentic path, so the narrowed condition is safe today (though worth revisiting if a second multinode agentic model is added to this cluster later).
| health_check: | ||
| interval_seconds: 10 | ||
| max_attempts: 1440 |
There was a problem hiding this comment.
🟡 This recipe sets health_check.max_attempts: 1440 (a deliberate 4h window for the ~1.4TB MXFP4 weight load), but runners/launch_b200-dgxc.sh unconditionally runs a sed that rewrites any max_attempts: N line to 720 before calling srtctl apply on every multinode config. This silently halves the intended window to 2h, making the 1440 value dead/misleading config (harmless today since 720×10s=2h still comfortably exceeds VLLM_ENGINE_READY_TIMEOUT_S=3600, and the validation sweep succeeded under the clamped window).
Extended reasoning...
What happens: The new recipe agg-b200-tp8pp2-agentic.yaml declares health_check.interval_seconds: 10 and max_attempts: 1440, i.e. a 4-hour health-check budget, with an inline comment explicitly framing this as generous headroom for loading ~1.4TB of MXFP4 weights off shared Lustre. However, runners/launch_b200-dgxc.sh contains this pre-existing line (unchanged by this PR, but now exercised by this PR's new multinode recipe):
sed -i 's/^ max_attempts: [0-9]*/ max_attempts: 720/' "${CONFIG_FILE%%:*}"\n```
This sed runs unconditionally for every multinode job right before `srtctl apply`. It matches any line beginning with exactly two spaces followed by `max_attempts: <digits>` — which matches this recipe's ` max_attempts: 1440` line at line 35 — and rewrites it to `720` regardless of what the recipe author set.
**Why existing code doesn't prevent it:** The sed was added earlier (per its own comment) to *bump up* a smaller 360-attempt default to 720 for large models like DSR1-FP8. It was written as an unconditional overwrite rather than a floor/max, so it works correctly as a bump only when the recipe's own value is below 720; it silently clamps downward when a recipe (like this one, loading the largest checkpoint in the repo) deliberately declares something larger.
**Step-by-step proof:**
1. Recipe file declares (lines 33-35):
```yaml\n health_check:\n interval_seconds: 10\n max_attempts: 1440\n ```
2. `launch_b200-dgxc.sh` is invoked for this config with `IS_MULTINODE=true`.
3. Before `srtctl apply`, the script runs `sed -i 's/^ max_attempts: [0-9]*/ max_attempts: 720/' "${CONFIG_FILE%%:*}"`.
4. The regex `^ max_attempts: [0-9]*` matches ` max_attempts: 1440` (two-space indent, digits after the colon), rewriting the line in place to ` max_attempts: 720`.
5. `srtctl apply` is then called on the *mutated* file, so the effective health-check window becomes `720 * 10s = 7200s` (2h), not the author's intended `1440 * 10s = 14400s` (4h).
**Impact:** Today this is benign — the recipe's own `VLLM_ENGINE_READY_TIMEOUT_S=3600` (1h) is the binding constraint on engine readiness and sits well under even the clamped 2h health-check window, and the PR's linked validation run (30385435921) completed successfully with the sed already in effect. So no concrete failure results from merging as-is. But the `1440` value is dead config: it communicates author intent ("I need up to 4h for this 1.4TB load") that is silently discarded, and a future reader/maintainer could reasonably believe the recipe controls its own timeout when it does not. If a future checkpoint or FS slowdown pushes weight-load time past 2h, the author's actual safety margin will not be what the recipe file claims.
**Suggested fix:** Either drop the `1440` value in the recipe (since it's currently a no-op) and note that the effective cap is the launcher's 720, or change the launcher's sed to only raise the value (e.g. compare and take the max) rather than unconditionally overwriting it, so recipes can opt into a longer window than the 720 default without being silently clamped.
</details>
<!-- bhfn:1:nit -->| VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "0" | ||
| # No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: the GB200/GB300 AgentX value | ||
| # (32768) hard-fails engine init on Kimi-K3 — the KDA hybrid gives it a | ||
| # scheduler_block_size of 3145728 and the interval must be a multiple of | ||
| # it ("VLLM_PREFIX_CACHE_RETENTION_INTERVAL (32768) must be non-negative | ||
| # and a multiple of scheduler_block_size (3145728)"). Default retention | ||
| # served fine in earlier runs. |
There was a problem hiding this comment.
🟡 The comment block on lines 73-78 of the new recipe file is stale and contradicts the line directly above it: line 72 sets VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "0", but the leftover comment opens with 'No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: ...' and closes with 'Default retention served fine in earlier runs' — text describing a mutually-exclusive earlier variant where the var was left unset. This is comment-only with zero functional impact, but should be deleted so a future maintainer doesn't mistakenly believe the var is unset.
Extended reasoning...
What the bug is
agg-b200-tp8pp2-agentic.yaml sets VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "0" on line 72, with a correct explanatory comment on lines 67-71 describing why 0 is the only valid value below the 3.1M-token KDA-hybrid scheduler block (any positive value must be a multiple of scheduler_block_size = 3145728, and the GB200/GB300 AgentX value 32768 hard-fails engine init).
Immediately below that, lines 73-78 are a second comment block that begins "No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: the GB200/GB300 AgentX value (32768) hard-fails engine init on Kimi-K3..." and ends "Default retention served fine in earlier runs." This block describes the opposite decision — leaving the variable unset entirely and falling back to the vLLM default retention interval.
Why this is contradictory
Both blocks agree on the underlying fact (32768 is rejected, values must be multiples of 3145728), but they describe two mutually exclusive configuration choices:
- Lines 67-72: "we pin the var to 0."
- Lines 73-78: "we do NOT set the var at all (
No VLLM_PREFIX_CACHE_RETENTION_INTERVAL), relying on the default."
Only one of these is actually reflected in the YAML — the var IS set, to "0" (line 72). The second block is dead, leftover text from an earlier recipe variant (the perf-changelog entry documents at least two prior experiment variants, D and G, that iterated on this exact setting) that was never deleted when the approach changed from "leave unset" to "pin to 0".
Why nothing catches this
This is a plain YAML comment inside aggregated_environment, so it has no effect on parsing or runtime behavior — VLLM_PREFIX_CACHE_RETENTION_INTERVAL resolves to "0" regardless of what the comment says. No linter or CI check inspects comment prose for internal consistency, so this only surfaces via manual code review.
Impact
Zero functional/runtime impact — the engine will boot with the interval correctly pinned to 0 either way. The risk is purely to future maintainability: an engineer skimming this file later (e.g., while porting the recipe to a new GPU SKU or troubleshooting a prefix-cache issue) could read the second block, conclude the variable is unset, and either add a duplicate/conflicting entry or misdiagnose an unrelated issue based on a false premise about the current config.
Proof / walkthrough
- Read line 72:
VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "0"— the key is present and set. - Read lines 73-78 immediately after: the comment text opens "No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: the GB200/GB300 AgentX value (32768) hard-fails engine init..." — grammatically and semantically this asserts the variable is absent/unset.
- The same block closes with "Default retention served fine in earlier runs," which only makes sense as a rationale for not setting the variable (i.e., relying on the built-in default) — directly at odds with the pinned "0" one line above.
- Since YAML has exactly one
VLLM_PREFIX_CACHE_RETENTION_INTERVALkey-value pair on line 72, and comments are inert, the true, executed behavior is "the var is 0." The second comment block is therefore stale narration that no longer matches the file it sits in.
Suggested fix
Delete the stale second comment block (lines 73-78) entirely; the accurate rationale is already fully captured by lines 67-71.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30407055748 |
3 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30407055748 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30407055748 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30407055748 |
|
@functionstackx the AgentX/AIPerf harness has been updated, please merge origin/main into your branch and refresh your submission. Additional tuning may be necessary depending on the config. I apologize for any inconvenience. This is an automated message. |
2 participants
Summary
Rebase of #2391 onto current
main. Same content, replayed as a single clean commit. #2391 branched fromde6ba0b1, then took aMerge remote-tracking branch 'origin/main'commit, and has since goneCONFLICTING / DIRTYagainst its base. This PR is the clean equivalent. #2391 stays open.Aggregated TP8 × PP2 across 2 B200 nodes (16 GPUs) for Kimi-K3. The native MXFP4 checkpoint (2.8T total params, ~1.4 TB of weights) does not fit one 8×B200 node, so TP8 shards attention/dense and PP2 splits the 93 layers. Plain TP, not TEP —
ep: 1, no expert parallelism, so the 896 routed experts are TP-sharded inside each pipeline stage. Aggregated mode (prefill num-worker 1 + decode num-worker 0): one worker serves both phases, no P/D KV transfer.Direct vLLM serving via
frontend.type: vllm(srt-slurm #278 plus the multi-node extension on functionstackx/srt-slurm-nvklaud/direct-vllm-multinode) —vllm serveowns the OpenAI port, no dynamo frontend/worker/router.Content equivalence with #2391
MODELS.mdMODELS_zh.mdbenchmarks/.../agg-b200-tp8pp2-agentic.yamlconfigs/nvidia-master.yamlrunners/launch_b200-dgxc.shperf-changelog.yamlDiff stat matches #2391 exactly: 6 files, +237 / −4.
Conflict resolution (the one intentional difference)
mainremoved the duplicate perf-changelog entry for #2386 in69af6a4e(revert #2396, "preserving the original PR #2371 entry"). Branch #2391 predates that revert and re-adds the duplicate, which is what makes it conflict.This PR keeps
main's state — the #2386 duplicate stays removed — and appends only the newkimik3-fp4-b200-dynamo-vllm-agenticentry. Verified: the 14 dropped lines contain thepull/2386link and nothing else; the kept block contains onlypull/2391.perf-changelog.yamlparses (644 entries).Validation
Sweep run 30385435921 concluded success for this exact config (Variant K — without the in-container
mamba_hybridpatch), which answers #2391's probe question: theindex_fill_dtype patch is no longer required by the currentvllm/vllm-openai:kimi-k3image.中文说明
#2391 的 rebase 版本。 内容一致，在当前
main上重放为一个干净提交。#2391 基于de6ba0b1，其后合入过一次origin/main合并提交，目前对基线已处于CONFLICTING / DIRTY状态；本 PR 为其干净等价物，#2391 保持开启。TP8 × PP2、2 个 B200 节点（16 GPU） 聚合部署。原生 MXFP4 权重（2.8T 参数，约 1.4TB）无法放入单个 8×B200 节点，故 TP8 切分 attention/dense，PP2 切分 93 层。纯 TP，非 TEP（
ep: 1），896 个路由专家在每个流水段内按 TP 切分。冲突处理：
main已在69af6a4e（revert #2396）中移除 #2386 的重复 changelog 条目并保留 #2371 原条目；#2391 早于该 revert 因而重新加回该重复条目。本 PR 保持main的状态，仅追加新的 #2391 条目，其余五个文件与 #2391 完全一致。验证：run 30385435921 结果为 success。
🤖 Generated with Claude Code