MiniMax-M3 MXFP8 full sweep config for GB200#1734
Conversation
Add minimaxm3-fp8-gb200-dynamo-vllm to nvidia-master.yaml with 6
topologies covering the full concurrency range:
- TP4/TP8 (low latency, conc 4-64)
- TP4+EP4 agg + 1P+1D disagg (mid curve, conc 64-512)
- DEP4/DEP8 (high throughput, conc 256-2048)
All recipe YAMLs included under minimax-m3-gb200-fp8/{1k1k,8k1k}/.
|
Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers. If additional help is needed, PR authors can reach out to core maintainers over Slack. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 12a1da5. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27452229867 |
Adopt the NVIDIA Dynamo vLLM runtime image (nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.3.0-minimax-m3-dev.1), the canonical M3 runtime from ai-dynamo/dynamo release/1.3.0-minimax-m3-dev.1. Changes mirrored from that release's recipes/minimax-m3/vllm/disagg/MXFP8/deploy.yaml: - dynamo.install: false — the runtime image bundles dynamo 1.3.0, so the prior 1.2.0 wheel install is dropped (srtctl defaults install=true) - attention-backend: FLASH_ATTN on every prefill/decode/agg engine Benchmark-specific knobs kept over the reference's serving defaults: language-model-only (text-only), no-enable-prefix-caching (random data), scenario-trimmed max-model-len.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27455776567 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27455867510 |
enroot's docker:// URI needs `#` to separate the registry host from
the image path; `nvcr.io/...` was parsed as a Docker Hub repo and 401'd
against registry-1.docker.io. Matches the existing nvcr.io# convention
in nvidia-master.yaml. Recipe container fields kept byte-identical to
the master image: field (srtslurm.yaml maps "${IMAGE}" -> squashfile).
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27456172038 |
Replace the mostly-aggregated GB200 sweep (5 agg + 1 disagg) with a fully disaggregated sweep that splits prefill/decode over NixlConnector, mirroring the minimaxm2.5-fp8-gb200 reference. Every worker = one 4-GPU node since the 444 GB MXFP8 checkpoint can't fit in fewer. Topologies (1k1k): 1P1D TP4 (low-lat), 1P1D TP4+EP4 (mid), 1P2D TP4+EP4 (decode-scaled), 2P1D TP4+EP4 (prefill-scaled), 1P1D DEP4 (max-tput), spanning conc 4-2048. - add 4 disagg recipes; remove 8 orphaned agg recipes (1k1k + 8k1k) - rewire nvidia-master.yaml search-space to the 5 disagg entries - perf-changelog: describe disagg sweep; fix stale Image line (vllm/vllm-openai:minimax-m3 -> nvcr.io#.../vllm-runtime:1.3.0-minimax-m3-dev.1) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27478602476 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27478698552 |
… transfer Run 27478698552 failed: every disagg worker crashed at NixlConnector init with "NIXL is not available" (RuntimeError, vllm .../nixl/worker.py:248). The ai-dynamo vllm-runtime:1.3.0-minimax-m3-dev.1 image ships dynamo but NOT the nixl bindings (cupy missing too), so kv_connector=NixlConnector cannot initialize and the engine core never becomes healthy. Revert to the pre-ed63c1e0 runtime path that pulls NIXL in via the dynamo wheel (same as the working minimaxm2.5-gb200 disagg recipes): - image/container: vllm/vllm-openai:minimax-m3 (the m3_release build all other m3 entries already use) - dynamo.install=true + wheel 1.2.0.dev20260526 (nixl is a dynamo dep) - keep attention-backend FLASH_ATTN (added in the image-switch commit) Also enable NVLink (MNNVL) KV transfer so NIXL doesn't fall back to TCP, mirroring the deepseek-v4 gb200 disagg recipes — on every prefill/decode env block: UCX_TLS=cuda_copy,cuda_ipc,tcp UCX_CUDA_IPC_ENABLE_MNNVL=y UCX_MEMTYPE_CACHE=n / UCX_MEMTYPE_REG_WHOLE=n NCCL_CUMEM_ENABLE=1 (cuMem-allocate buffers so they are IPC-exportable) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27479440281 |
The narrow DEP8-max sweep showed no GB200 advantage over B200 because both cap at an 8-GPU NVLink island. Exploit NVL72's rack-scale NVLink with wide expert parallelism spanning multiple nodes, mirroring the deepseek-v4 "megamoe" ladder (DEP = data-parallel attention + expert-parallel): - 1P1D TP4 (2n) low-latency, conc 4-64 - 1P1D DEP8 (4n) mid, EP8/16-experts-per-rank, conc 128-512 - 1P1D DEP8->DEP16 (6n) wide decode (EP16), conc 512-2048 - 2P1D DEP8->DEP16 (8n) prefill-scaled, conc 2048-4096 - 4P1D DEP8->DEP16 (12n) max throughput, conc 4096-8192 M3 has 128 routed experts (top-4), so EP8/EP16 shard cleanly. EP16 across 16 GPU / 4 nodes is the regime B200 physically can't reach. Attention: FLASH_ATTN -> FLASHINFER (trtllm-gen) on all GB200 recipes to exploit Blackwell. Requires the :minimax-m3 image rebuilt from m3_release HEAD 022448dd (vllm-project/vllm#45381), which gates trtllm-gen page>=128. Also add GB200 perf/NVLink-KV knobs from the deepseek-v4 reference: numa-bind (Grace) and enable-sleep-mode (cuMem allocator so the KV cache is IPC-exportable over the MNNVL fabric), alongside the existing UCX MNNVL env. Replaces the four narrow EP4 recipes; keeps 1P1D TP4 for low latency. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27486649944 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27486767586 |
1k1k TP4 low-conc tuning: stream-interval 1 (was 128 decode / 32 prefill), cudagraph cap 128 (was 512), conc range extended to 1-64 (was 4-64) to match B200 coverage. 8k1k sweep: 5 disagg recipes mirroring the 1k1k megamoe ladder (TP4, DEP8, DEP8→DEP16, 2P1D, 4P1D) with max-model-len 9472 (74×128 blocks = ISL+OSL+256 headroom). Concurrencies shifted ~4x lower for 8x heavier prefill: TP4 1-16, DEP8 32-128, DEP8→DEP16 128-512, 2P1D 512-1024, 4P1D 1024-2048. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27490108355 |
1 participant
Summary
minimaxm3-fp8-gb200-dynamo-vllmto nvidia-master.yaml with 6 topologies: TP4, TP8, TP4+EP4, 1P+1D disagg, DEP4, DEP8/mnt/lustre01/models/MiniMax-M3-MXFP8extra_mountandHF_HOMEfrom recipesminimax-m3-gb200-fp8/{1k1k,8k1k}/Test plan
Note
Medium Risk
Changes affect production multinode benchmark launches (12-node racks, secrets in CI env, and shared squash import behavior); misconfiguration could waste cluster time or break parallel GB200 jobs, but scope is benchmark/infra rather than product auth or data paths.
Overview
Adds
minimaxm3-fp8-gb200-dynamo-vllmtonvidia-master.yaml: a multinode, fully disaggregated Dynamo+vLLM benchmark matrix for MiniMax-M3 MXFP8 on GB200 (vllm/vllm-openai:minimax-m3), covering 1k1k and 8k1k with five P/D topologies from 1P1D TP4 through 4P1D DEP8→DEP16 (up to 12 nodes) and concurrency ladders matched to each layout.Introduces 10 Slurm recipe YAMLs under
benchmarks/multi_node/srt-slurm-recipes/vllm/minimax-m3-gb200-fp8/(NixlConnector, FLASHINFER, block-size 128, Dynamo wheel install, UCX/NVLink KV transfer env). Documents the sweep inperf-changelog.yaml.launch_gb200-nv.shgains minimaxm3/fp8 staging (/mnt/lustre01/models/MiniMax-M3-MXFP8), reuses the watchtower/shared-FS paths and recipe staging used for minimaxm2.5, clones/copies the new recipe tree, and replaces bareenroot importwith a locked, atomicimport_squash(also for minimaxm3 runs) to avoid corrupting shared squash images under parallel matrix jobs.benchmark-multinode-tmpl.ymlsetsHF_TOKENfrom a repo secret so Slurm workers can pull large Hub snapshots without anonymous rate limits (fallback for day-zero style downloads).Reviewed by Cursor Bugbot for commit c806aa2. Bugbot is set up for automated code reviews on this repo. Configure here.