[KimiK3][AgentX]: GB200 DSpark and Simple CPU KV offload - #2404
Conversation
Add srt-slurm TP, TEP, and DEP AgentX profiles plus GB200 runner routing and recipe overlay plumbing. 中文：新增 Kimi K3 GB200 首日 AgentX 支持，包括 srt-slurm TP、TEP、DEP 配置，以及 GB200 运行器路由和配方覆盖逻辑。
Record the draft pull request URL in the append-only benchmark changelog entry. 中文：在仅追加的基准测试变更日志条目中记录草稿 PR 链接。
中文：将 origin/main 合并到 GB200 Kimi K3 AgentX 分支。
Represent the throughput arm as TP1 x DP16 = EP16 in the master configuration, matching the vLLM recipe. 中文：将吞吐量配置修正为 TP1 × DP16 = EP16，与 vLLM 配方保持一致。
Use the pre-staged /mnt/numa1/models/Kimi-K3 checkpoint on every GB200 compute node while retaining shared-filesystem orchestration and caches. 中文：在每个 GB200 计算节点上使用预置的 /mnt/numa1/models/Kimi-K3 检查点，同时保留共享文件系统上的编排与缓存。
Keep the Day-0 changelog focused on the vLLM recipe performance points and use the model default context length. 中文：精简 Kimi K3 首日性能配置，仅保留与 vLLM 配方一致的性能测试点，并使用模型默认上下文长度。
Move latency and balanced profiles to 16-GPU TP/TEP, use TP2 x DP8 for EP16 throughput, and extend the throughput concurrency sweep through 256. 中文：根据 GB200 HBM 容量调整 Kimi K3 AgentX 扫描。将延迟与均衡配置扩展为 16 GPU TP/TEP，吞吐配置改为 TP2 × DP8（EP16），并将吞吐并发扫描扩展至 256。
The Dynamo vLLM worker entrypoint does not accept vllm serve tool-call or reasoning parser flags; agentic request parsing belongs to the Dynamo frontend. 中文：移除 Dynamo vLLM worker 不支持的 Kimi K3 工具调用与推理解析参数；AgentX 请求解析由 Dynamo frontend 负责。
Remove literal shell placeholders from the vLLM worker environment so Gloo and NCCL use the interfaces selected by the srt-slurm launcher. 中文：移除 vLLM worker 环境中的字面量 shell 占位符，由 srt-slurm launcher 为 Gloo 与 NCCL 选择网络接口。
Patch the pinned agentic srt-slurm checkout to launch one vLLM process per GB200 node, with two local DP ranks and TP2 per rank. Export every per-node metrics endpoint to AIPerf. 中文：修补 AgentX 固定版本的 srt-slurm，使每个 GB200 节点启动一个 vLLM 进程，每个进程承载两个本地 DP rank，并为每个 rank 使用 TP2；同时将所有节点的 metrics endpoint 传给 AIPerf。
Use TP4 x DP4 for the throughput-oriented EP16 profile after TP2 x DP8 exceeded GB200 HBM during MXFP4 weight loading. 中文：TP2 × DP8 在 MXFP4 权重加载阶段超过 GB200 HBM 容量，因此将吞吐优先的 EP16 配置调整为 TP4 × DP4。
Pin a tested srt-slurm revision that natively groups TP GPUs within each DP rank and exports every DP metrics endpoint. Remove the vendored patch and reserve enough HBM for one native 1M-context request. 中文：固定经过测试的 srt-slurm 版本，原生支持每个 DP rank 内的 TP GPU 分组，并导出全部 DP metrics endpoint。删除仓库内补丁，同时为原生 1M 上下文请求预留足够 HBM。
Record the tested TP4 x DP4 = EP16 throughput profile. 中文：记录经过测试的 TP4 x DP4 = EP16 吞吐配置。
Use NVIDIA’s Kimi K3 runtime with native tokenizer, parser, FP8 KV-cache, and FlashInfer support. Keep the model-default 1M context and remove the vLLM-mutating setup script.\n\n中文：改用 NVIDIA 官方 Kimi K3 Dynamo 运行时，启用原生 tokenizer、解析器、FP8 KV 缓存和 FlashInfer 支持。保留模型默认的 1M 上下文，并移除会修改 vLLM 文件的环境配置脚本。
Match identity verification and result metadata to the Dynamo 1.3.0 package reported by the official Kimi K3 runtime image.\n\n中文：将身份校验与结果元数据对齐到官方 Kimi K3 运行时镜像实际报告的 Dynamo 1.3.0 包版本。
Use the official Kimi K3 Dynamo 0.92 GPU memory utilization so native 1M-context MLA prefills retain transient workspace without changing max model length.\n\n中文：采用官方 Kimi K3 Dynamo 配方的 0.92 GPU 显存利用率，为原生 1M 上下文 MLA 预填充保留临时工作区，且不修改最大模型长度。
Apply vLLM EP_SIZE = TP_SIZE × DP_SIZE in the master metadata for the DP1 latency profile.\n\n中文：在 DP1 延迟配置的主配置元数据中应用 vLLM 的 EP_SIZE = TP_SIZE × DP_SIZE 规则。
中文：将 main 最新变更同步到 Kimi K3 GB200 支持分支。
Copy per-concurrency AgentX aggregate JSON files from the compute-visible shared workspace back into the GitHub workspace, and cover success and missing-result behavior with regression tests. 中文：将各并发度的 AgentX 聚合 JSON 从计算节点可见的共享工作区复制回 GitHub 工作区，并通过回归测试覆盖成功复制和结果缺失场景。
Enable AIPerf burst phase starts only for the Kimi K3 GB200 saturation recipes so warmup and profiling reach the requested concurrency within the benchmark window. Inter-turn replay timing remains unchanged. 中文：仅在 Kimi K3 GB200 饱和度测试配置中启用 AIPerf 阶段起始突发，使预热和正式测试能在基准测试窗口内达到目标并发度，同时保持轮次间回放时序不变。
Merge the latest main branch while preserving the Kimi K3 changelog entry at the append-only tail. 中文：合并最新 main 分支，并将 Kimi K3 变更日志条目保留在只追加文件的末尾。
Bring in the merged AgentX v1.0 AIPerf default and current benchmark updates while preserving the Kimi K3 changelog entry at the tail.\n\n中文：合并 AgentX v1.0 AIPerf 默认行为及最新基准测试更新，并保留位于末尾的 Kimi K3 变更日志条目。
Allow the full-context DEP saturation sweep up to 3600 seconds to drain accelerated warmup requests. The grace period exits early when the servers become idle. 中文：将 Kimi K3 DEP 全上下文饱和度测试的加速预热排空上限延长至 3600 秒；服务空闲后会提前结束等待。
中文：将最新 main 分支合并到 Kimi K3 GB200 支持分支。 # Conflicts: # perf-changelog.yaml
Restore the legacy AgentX AIPerf integration with the barrier-aware idle cap, and retain the GB200 Kimi K3 performance changelog entry at the append-only tail. 中文：同步最新 main，恢复带屏障感知空闲上限的旧版 AgentX AIPerf 集成，并将 GB200 Kimi K3 性能变更记录保留在追加式文件末尾。
Add gb200-nv_3 to the GB200 cluster runner pool so the Kimi K3 sweep can use the capacity provisioned for this bring-up. 中文：将 gb200-nv_3 注册到 GB200 集群运行器池，使 Kimi K3 扫描能够使用本次适配新增的容量。
中文：将最新 main 合并到 Kimi K3 GB200 KV 卸载分支，并按追加式规则保留性能变更记录。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30494165510 |
Incorporate the latest main-branch AgentX, runner, recipe, and AIPerf updates while preserving this PR performance changelog entry at the file tail.\n\n中文：合并 main 分支最新的 AgentX、运行器、配方及 AIPerf 更新，并将本 PR 的性能变更日志条目保留在文件末尾。
Raise the GB200 DEP and SimpleCPU-offload recipes to the official 0.95 GPU memory utilization so MRV2 can allocate the 16.9 GiB required for one full 1M-token request. 中文：将 GB200 DEP 与 SimpleCPU 卸载配置的 GPU 显存利用率提高到官方建议的 0.95，使 MRV2 能为单个完整 100 万 token 请求分配所需的 16.9 GiB KV 缓存。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30509515683 |
Tune GB200 DEP GPU memory utilization to 0.94: the measured 0.95 setting cleared the full-context KV admission gate but exhausted CUDA memory during MRV2 graph capture. 中文：将 GB200 DEP 的 GPU 显存利用率调至 0.94；实测 0.95 虽能通过完整上下文的 KV 缓存准入检查，但会在 MRV2 CUDA Graph 捕获阶段耗尽显存。
Keep MRV2 PIECEWISE CUDA graphs for GB200 DEP while disabling the FULL capture phase that OOMs after full-context KV allocation. This follows existing GB200 AgentX recipe practice and avoids eager execution. 中文：GB200 DEP 保留 MRV2 PIECEWISE CUDA Graph，同时禁用在完整上下文 KV 缓存分配后触发显存溢出的 FULL 捕获阶段。该设置沿用仓库现有 GB200 AgentX 配置，并避免退回纯 eager 执行。
|
@cquil11 the AgentX/AIPerf harness has been updated, please merge origin/main into your branch and refresh your submission. Additional tuning may be necessary depending on the config. I apologize for any inconvenience. This is an automated message. |
Use the supported FlashInfer TRT-LLM MoE backend for Kimi K3 DSpark DEP after DeepGEMM grid barriers timed out under MRV2 with one-sided all-to-all. Preserve the default 1M context, DEP16 topology, DSpark, and the one-sided FlashInfer all-to-all path. 中文：Kimi K3 DSpark DEP 在 MRV2 与单边 FlashInfer all-to-all 组合下触发 DeepGEMM 网格屏障超时，因此改用受支持的 FlashInfer TRT-LLM MoE 后端；保持默认 1M 上下文、DEP16 拓扑、DSpark 和单边 all-to-all 不变。
Use the standard safetensors loader for GB200 DEP recipes so FlashInfer MoE and DSpark do not exhaust transient HBM in the fastsafetensors GPU-staging path. 中文：GB200 DEP 配置改用标准 safetensors 加载器，避免 FlashInfer MoE 与 DSpark 在 fastsafetensors GPU 暂存阶段耗尽瞬时 HBM。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30514232639 |
Limit regular DEP captures to 64 sequences per engine and offload captures to 96, matching the highest configured global concurrencies under DP4 while retaining the default 1M context. 中文：按 DP4 下配置的最高全局并发量，将常规 DEP 的每引擎 CUDA Graph 捕获上限设为 64，将卸载配置设为 96；保留默认 1M 上下文，并为 FlashInfer MoE 运行时工作区释放显存。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30516773401 |
Align all Kimi K3 GB200 recipe identities and the master-config router label with the 1.3.0 version reported by the pinned K3-enabled source checkout. 中文：将所有 Kimi K3 GB200 配方身份信息及主配置中的路由版本统一为固定 K3 源码构建实际报告的 1.3.0。
Capture even DSpark sequence counts through the c384 DP4 steady state. This preserves every configured steady-state batch while releasing the HBM needed by the FlashInfer MXFP4 MoE workspace. 中文：按偶数序列数捕获 DSpark CUDA Graph，覆盖 c384/DP4 稳态批次；保留所有已配置稳态点，同时释放 FlashInfer MXFP4 MoE 工作区所需显存。
中文：将最新 main 合并到 Kimi K3 GB200 分支，解决 perf-changelog.yaml 冲突，并保留本 PR 条目在文件末尾。
Allow the full-context high-concurrency AgentX warmup and one-hour profile to complete by using the GB200 batch partition maximum. 中文：将 Kimi K3 GB200 Simple CPU KV 卸载配方的 Slurm 时限延长至 12 小时，使全上下文高并发 AgentX 预热和一小时性能采集能够完整结束。
Enable vLLM prompt-token details on every GB200 Kimi K3 AgentX profile so AIPerf can record per-request cached prompt tokens and aggregate cache-read rates. 中文：为所有 GB200 Kimi K3 AgentX 性能配置启用 vLLM prompt-token details，使 AIPerf 能记录每个请求的缓存提示词 token 数及汇总缓存读取率。
Merge the current main branch into the Kimi K3 GB200 AgentX branch and preserve its performance-changelog entry at the tail. 中文：将最新 main 分支合并到 Kimi K3 GB200 AgentX 分支，并将本 PR 的性能变更记录保留在 perf-changelog.yaml 末尾。
Pin the same AIPerf warmup-handoff revision used by the Qwen B200 and B300 AgentX PRs. Add a recipe-scoped 300-second per-trace idle-gap cap for all Kimi K3 GB200 profiles while retaining full model context and cached-token reporting. 中文：将 AIPerf 固定到 Qwen B200/B300 AgentX PR 使用的同一预热交接版本。为所有 Kimi K3 GB200 性能配置添加仅配方生效的 300 秒单轨迹空闲间隔上限，同时保留完整模型上下文与缓存 token 上报。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30521128887 |
Remove the native vLLM OpenAI frontend flag that Dynamo rejects. Dynamo already emits prompt_tokens_details.cached_tokens from RequestOutput for each response. 中文：移除 Dynamo 不支持的原生 vLLM OpenAI 前端参数。Dynamo 已会从每个 RequestOutput 生成 prompt_tokens_details.cached_tokens。
Resolve the append-only performance changelog by restoring the GB200 entry at the new tail. 中文：将最新 main 合并到 GB200 AgentX 分支，并按追加规则把 GB200 性能变更记录恢复到文件末尾。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30585243021 |
Pin the same AIPerf warmup-handoff revision used by the B200 and B300 Qwen AgentX PRs so baseline parents survive into profiling. 中文：将 AIPerf 固定到 B200/B300 Qwen AgentX PR 使用的同一预热交接版本，确保基线父请求能够正确延续到性能分析阶段。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30586775245 |
Allow the Kimi K3 TP recipe to collect a larger live sample before aborting while preserving the existing 10% post-run validity threshold. 中文：为 Kimi K3 TP 配置分离实时中止阈值与最终校验阈值，允许低并发轨迹先积累更充分的实时样本，同时保持现有 10% 运行后有效性门槛。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30594831393 |
2 participants
Summary
vllm/vllm-openai:kimi-k3image for every Kimi K3 GB200 AgentX profile.v1.4.0-kimi-k3-dev.1preview from its immutable release commit throughdynamo.install: true.SimpleCPUOffloadConnectoron the TP4 × DP4 / EP16 throughput topology at concurrency 128, 192, 256, and 384.References
Validation
vllm/vllm-openai:kimi-k3, all labeled for speculative decoding.srtctl dry-runon the pinned Kimi K3 srt-slurm revision.python -m pytest utils/matrix_logic/ -v: 224 passed.python -m pytest runners/test_slurm_utils.py -q: 2 passed.perf-changelog.yamlhas one pending Kimi K3 GB200 record, passes the changelog validator, and ends with a newline.The previous 18/18 green full sweep validated the pre-DSpark Dynamo-runtime-image revision and is retained only as historical baseline evidence.