[AMD] dsv4-fp4-mi355x-atom: enable DPA TBO at high concurrency, update image to atom0.1.4

[seungrokj]

Summary

• Update image to rocm/atom:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom0.1.4_20260612
• Enable --enable-tbo (Token-Bucket Overlap) on top of DPA+TP8 at high concurrency:
  • ISL=1024 / OSL=1024 → TBO at CONC ≥ 1024
  • ISL=8192 / OSL=1024 → TBO at CONC ≥ 256
• Update ISL=8192 search-space: TP8-only conc=4–64, DPA conc=128–1024 (previously conc=1–64 and DPA conc=64–512)
• Add perf-changelog entry

Motivation

Based on Pareto frontier analysis from ATOM runs 27367309656 and 27030375093, DPA TBO dominates the Pareto frontier at high concurrency for both ISL settings, delivering the best throughput/GPU without sacrificing interactivity vs plain DPA.

Test plan

• Verify bash -n syntax check passes on dsv4_fp4_mi355x_atom.sh
• Confirm TBO is enabled for correct ISL/CONC combinations in CI run

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes ATOM launch flags (TBO, cudagraph capture, prefix caching, max concurrent seqs) for a large production model benchmark path; the CI sweep is temporarily collapsed to a single concurrency point, which limits regression coverage until the commented grid is re-enabled.

Overview
Updates DeepSeek-V4 FP4 on MI355X (ATOM) benchmarking to match Pareto-tuned high-concurrency serving and a newer ROCm ATOM image.

CI config (dsv4-fp4-mi355x-atom): Bumps the container to atom0.1.4_20260612, removes the prior Day-0 CONC=1 guard comment, and replaces the active fixed-seq-len sweep with a narrow setup—the full TP8 / DPA / DPA+TBO grids are left commented, and the only live point is ISL=8192, DPA, CONC=1024.

Benchmark script (dsv4_fp4_mi355x_atom.sh): On DPA+TP8, turns on --enable-tbo when ISL/OSl/concurrency cross the thresholds (1k/1k at CONC≥1024; 8k/1k at CONC≥256), with expanded --cudagraph-capture-sizes for the 1k/1k TBO case. The ATOM server now gets --no-enable_prefix_caching, --cudagraph-capture-sizes, and --max-num-seqs ${CONC}; eval-only runs wire context length via compute_eval_context_length (with --max-model-len left commented).

Documents the change in perf-changelog.yaml.

^{Reviewed by Cursor Bugbot for commit 7ffa976. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

• https://docs.sglang.io/cookbook/intro
• https://recipes.vllm.ai/

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

• https://github.com/vllm-project/recipes
• https://github.com/sgl-project/sglang/tree/main/docs_new

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

[github-actions]

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

• https://docs.sglang.io/cookbook/intro
• https://recipes.vllm.ai/

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

• https://github.com/vllm-project/recipes
• https://github.com/sgl-project/sglang/tree/main/docs_new

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

[github-actions]

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

• https://docs.sglang.io/cookbook/intro
• https://recipes.vllm.ai/

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

• https://github.com/vllm-project/recipes
• https://github.com/sgl-project/sglang/tree/main/docs_new

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27416363770
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27416363770

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27417198968
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27417198968

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27418122192
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27418122192

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27418122192
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27418122192

[cursor]

Search space not enabled

High Severity

The dsv4-fp4-mi355x-atom fixed-seq-len matrix no longer matches the PR: the ISL=1024 scenario is fully commented out, and ISL=8192 only runs a single DPA point at conc=1024 instead of the documented TP8 (conc 4–64) and DPA (conc 128–1024) sweeps.

 

^{Reviewed by Cursor Bugbot for commit c3b3289. Configure here.}

[cursor]

Eval context not wired

Medium Severity

This commit adds SERVE_MAX_MODEL_LEN / compute_eval_context_length handling for EVAL_ONLY, but the server launch leaves --max-model-len commented out, so the ATOM server never receives the computed context length.

 

^{Reviewed by Cursor Bugbot for commit c3b3289. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7ffa976. Configure here.}

[cursor]

CUDA graph sizes omit high concurrency

Medium Severity

For ISL=8192 and OSL=1024, --enable-tbo turns on when CONC≥256, but CUDAGRAPH_SIZES stays capped at 256 while the server uses --max-num-seqs equal to CONC (e.g. 1024 in the active YAML). The 1024/1024 TBO branch extends capture sizes through 1024; this path does not.

 

^{Reviewed by Cursor Bugbot for commit 7ffa976. Configure here.}

[functionstackx]

can u rebase it so the sweep triggers?