MiniMax-M3 MXFP8 full sweep config for GB300#1735
Conversation
Add minimaxm3-fp8-gb300-dynamo-vllm to nvidia-master.yaml with 7
topologies covering the full concurrency range:
- TP4/TP8 (low latency, conc 4-64)
- TP4+EP4 agg + 1P+1D disagg 2-node + 1P+1D collocated (mid, conc 64-512)
- DEP4/DEP8 (high throughput, conc 256-2048)
All recipe YAMLs included under minimax-m3-gb300-fp8/{1k1k,8k1k}/.
GB300 recipes include srun_options mem=0 (CW DefMemPerCPU cgroup fix)
and omit safetensors-load-strategy prefetch (host-memory limit).
|
Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers. If additional help is needed, PR authors can reach out to core maintainers over Slack. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27452223695 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27452273567 |
srun_options.mem=0 only grants a step the job's existing allocation; on gb300-cw (DefMemPerCPU=4096, no DefCpuPerGPU) the job itself was only allocated 4 GB/node and workers were cgroup-OOM-killed during engine init (run 27452273567: oom_kill in StepId=7409.7 on slurm-gb300-133-193, worker RLIMIT showed 4194304 KB). The canary passed because it landed on gb300-nv, which doesn't enforce the cap. Mirrors the sbatch_directives block of the DSV4 agentic recipes.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27452976271 |
…h_model lock race With the mem fix in place, run 27452976271 cleared the OOM but hit a new failure: both nodes of the TP8-2n job called dynamo fetch_model within 200ms (191 @ :23.637, 193 @ :23.833), 191 took the per-blob .lock on the shared /mnt/vast/hf-home cache and held it verifying the 444 GB snapshot, 193 retried ~6.4s and died 'Lock acquisition failed' (dynamo's rust hub doesn't wait like Python hf_hub). The launcher already pre-stages and verifies the snapshot offline before submit, so the workers never need to fetch. Setting HF_HUB_OFFLINE=1 in every worker env block makes dynamo serve cache-only and skip the download lock entirely, so co-fetching workers no longer collide. Applied to all agg + disagg (prefill/decode) env blocks across the 11 recipes.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27453434847 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27453693856 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27453693856 |
The previous pin 062a5de9 (set by #1571 "chore: agentx v0.3") was the cjq/agentx-v0.3 tip on 2026-06-02, but that branch was later rebased/ force-pushed (now at ff2b646c) which orphaned 062a5de9; GitHub has since garbage-collected it. It is now unfetchable ("upload-pack: not our ref") and absent from every CI runner cache, so actions/checkout fails on any cold runner with "Unable to find current revision in submodule path utils/aiperf" (e.g. the newly-added gb300-cw runner-4, run 27453693856). Re-pin to the current cjq/agentx-v0.3 tip — the branch .gitmodules already declares, which is live/fetchable and contains the prior aiperf history as an ancestor. This makes the pin and the declared branch consistent again.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27453693856 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27457134583 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27457134583 |
Replace the aggregated M3 GB300 topologies with disaggregated-only, and enable NixlConnector KV transfer over multi-node NVLink on every disagg recipe. On gb300-cw the cross-node prefill->decode KV handoff was silently falling back to RDMA/TCP (~268 MB/s, ~1400 tiny descriptors for M3 MSA cache) — the disagg ceiling. Setting UCX_CUDA_IPC_ENABLE_MNNVL=y plus --enable-cumem-allocator (VMM-registers KV so NIXL uses cuda_ipc across the NVL fabric) lifts it to ~1.4-1.7 GB/s and gives +17% / +23% / +49% out tok/s/gpu at conc 64 / 128 / 256 (jobs 7490 base vs 7493 MNNVL, 1P1D TP4EP4). This is a GB300-only win: B300 8-GPU IB islands cannot move KV over multi-node NVLink. Sweep (1k1k), all MNNVL: - 1P1D TP4+EP4 collocated 1n (8 GPU), conc 8-256 - low/mid latency - 1P1D TP4+EP4 split 2n (8 GPU), conc 64-512 - mid throughput - 1P + DP16+EP wide decode 5n (20 GPU), conc 512-2048 - max throughput (decode keeps scaling on NVL where 1P1D saturates: ~1213 vs ~810 out tok/s/gpu @ conc 1024) Removes all agg-gb300 recipes (1k1k + 8k1k); applies MNNVL to the 8k1k disagg recipe too for consistency.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27479316691 |
The collocated-1n topology (disagg-gb300-1p1d-tp4ep4-1n) declared gpus_per_node: 8, but gb300-cw nodes have 4 GPUs — sbatch rejects it with "Requested node configuration is not available" even on a fully idle cluster (confirmed: fails standalone with 28 nodes free; the split-2n and wide-decode at gpus_per_node 4 schedule fine). It was an 8-GPU-node template artifact that never reached sbatch before. Remove it (1k1k + 8k1k) and let the split-2n cover the low-latency end (conc extended down to 8). Add the 8k1k (isl 8192) scenario mirroring 1k1k with the two valid disagg shapes (split-2n + wide DP16 decode), MNNVL KV transfer on both, seq params retuned for long context (max-model-len 9472) and lower concurrency.
| ep: 4 | ||
| dp-attn: false | ||
| additional-settings: | ||
| - "CONFIG_FILE=recipes/vllm/minimax-m3-gb300-fp8/1k1k/disagg-gb300-1p1d-tp4ep4-2n.yaml" |
There was a problem hiding this comment.
1k1k conc-list recipe mismatch
Medium Severity
The 1k1k disagg conc-list includes 8, 16, and 32, but the linked recipe’s benchmark.concurrencies only runs 64 through 512. GB300 multinode sweeps execute recipe concurrencies, so those lower points never run while matrix metadata still advertises them.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 7fd8904. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27479506192 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27480086800 |
…sagg config Adds a 17-node (full-rack) disagg topology to the M3 GB300 sweep (1k1k + 8k1k) from on-cluster tuning (gb300-cw): - PREFILL is the binding bottleneck, not decode width or KV transfer: a single prefill worker left ~3967 reqs queued and starved 64 decode GPUs. Balancing to 5 prefill : 12 decode (TP4) cleared the backlog and lifted throughput +57% (535 -> 843 out tok/s/gpu @ conc 2048). - TP-only decode (ep1, no expert parallelism) per the Qwen3.5-397B-A17B recipes (closest M3 analog); M3 wide-EP/DP-attention all-to-all was slower and DP32 < DP16 per-GPU. - Kept the existing 1p1d (low/mid latency) and dep16dec (wide-decode) topologies so CI measures the full Pareto rather than replacing them. NixlConnector KV transfer stays on multi-node NVLink (MNNVL + cumem); note KV transfer was verified NOT to bottleneck throughput (doubling its bandwidth via num_threads changed end-to-end tok/s/gpu by ~0). recipe yamls line up 1:1 with the nvidia-master.yaml CONFIG_FILE references.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27485224567 |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 88e99ce. Configure here.
| type: "sa-bench" | ||
| isl: 1024 | ||
| osl: 1024 | ||
| concurrencies: "64x128x256x512" |
There was a problem hiding this comment.
Recipe omits low concurrencies
Medium Severity
The 1k1k 1P+1D sweep declares conc-list values 8, 16, and 32 in nvidia-master.yaml, but the recipe’s benchmark.concurrencies only runs 64 through 512. Multinode jobs use the recipe list via srtctl, not matrix CONC_LIST, so those low-concurrency points never execute.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 88e99ce. Configure here.
1 participant
Summary
minimaxm3-fp8-gb300-dynamo-vllmto nvidia-master.yaml with 7 topologies: TP4, TP8, TP4+EP4, 1P+1D disagg (2-node), 1P+1D collocated (1-node), DEP4, DEP8sbatch_directives: mem: "0" / cpus-per-task: "72"plussrun_options: mem: "0"(CW DefMemPerCPU=4096 cgroup fix — step-level mem=0 alone only grants what the job allocation already holds) and omit safetensors prefetch (host-memory limit)minimax-m3-gb300-fp8/{1k1k,8k1k}/Test plan
--mem=0 --cpus-per-task=72).lock(Lock acquisition failed); workers now run cache-only against the launcher-pre-staged snapshot — needs a re-run on gb300-cw to confirmNote
Medium Risk
Introduces a read-only HF token into multinode CI and shared NFS cache coordination for a very large model; changes are benchmark/infra-only, not production serving paths.
Overview
Adds MiniMax-M3 MXFP8 GB300 benchmarking on gb300-cw via dynamo-vllm: a new
minimaxm3-fp8-gb300-dynamo-vllmentry innvidia-master.yamland six Slurm recipes underminimax-m3-gb300-fp8/for 1k1k and 8k1k disagg shapes (1P+1D TP4+EP4, 1P + wide DP16 decode, 5P + 12D rack-scale). Recipes target CW cgroup limits (mem: "0",cpus-per-task: 72), omit safetensors prefetch, use NixlConnector with MNNVL +enable-cumem-allocator, and run workers HF_HUB_OFFLINE against a shared cache mount.launch_gb300-cw.shgains a minimaxm3/fp8 path:hf:model id, recipe sync from this repo,__M3_HF_HOME__substitution to/mnt/vast/hf-home, lock cleanup, and launcher-side Hub pre-download beforesrtctl apply.benchmark-multinode-tmpl.ymlsetsHF_TOKENfrom a GitHub secret so parallel Slurm workers can pull the ~444 GB snapshot without anonymous rate limits.perf-changelog.yamldocuments the new config key.Reviewed by Cursor Bugbot for commit 88e99ce. Bugbot is set up for automated code reviews on this repo. Configure here.