[AMD] [AGENTX] dsv4-fp4-mi355x-vllm-agentic: DEP tuning + LMCache refresh by seungrokj · Pull Request #2378 · SemiAnalysisAI/InferenceX

seungrokj · 2026-07-28T05:20:08Z

Summary

Tunes the DeepSeek-V4-Pro FP4 MI355X vLLM agentic recipe and refreshes its master-config entry.

benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm.sh

Merge --enable-expert-parallel into PARALLEL_ARGS (dropped the separate EP_ARGS).
Add MODE_ARGS: --enable-ep-weight-filter (EP>1), --prefill-schedule-interval 8 (DP-attn), and a DEP8 prefill budget (--max-num-batched-tokens $((MNBT*8)), --long-prefill-token-threshold 16384), else --max-num-batched-tokens 8192.
Serve flags: add --block-size 256, --max-model-len 1048576; switch cudagraph to FULL_DECODE_ONLY; --gpu-memory-utilization 0.8 → 0.95.
Enable VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4; drop the gist scheduler.py monkeypatch and the VLLM_ROCM_USE_AITER_MOE export.
LMCache: build v0.5.2 (was the PR-3853 SHA), --chunk-size 1024, undivided CPU pool.
Fix a TOTAL_CPU_DRAM_PARTITION_GB="$(TOTAL_CPU_DRAM_GB)" command-substitution typo (2 places) → "$TOTAL_CPU_DRAM_GB".

configs/amd-master.yaml (dsv4-fp4-mi355x-vllm-agentic)

Bump image to nightly-49f31d7c….
dram-utilization 0.60 → 0.80.
Switch the agentic search-space to a single DEP lmcache arm at conc 76 (previous arms commented out).

Test plan

Full-sweep CI green for dsv4-fp4-mi355x-vllm-agentic
LMCache MP server builds (v0.5.2, hipcc) and becomes healthy
Offload pool sizes correctly with the TOTAL_CPU_DRAM_GB fix

🤖 Generated with Claude Code

…resh Recipe: merge EP flag into PARALLEL_ARGS; add MODE_ARGS (ep-weight-filter, prefill-schedule-interval, DEP8 prefill budget --max-num-batched-tokens 8x / --long-prefill-token-threshold 16384); add --block-size 256, --max-model-len 1048576; switch cudagraph to FULL_DECODE_ONLY; gpu-mem-util 0.8 -> 0.95; enable VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4; drop the gist scheduler.py patch and the AITER_MOE export. LMCache: build v0.5.2 (was PR-3853 SHA), chunk-size 1024, undivided CPU pool. Fix TOTAL_CPU_DRAM_PARTITION_GB command substitution typo. amd-master.yaml: bump image, dram-utilization 0.60 -> 0.80, switch agentic search-space to DEP lmcache conc 76. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

github-actions · 2026-07-28T05:20:17Z

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

… master-config Align the agentic search-space kv-offload-backend version with the recipe (build tag v0.5.2, was the PR-3853 SHA 9229067). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

claude · 2026-07-28T05:29:08Z

 dsv4-fp4-mi355x-vllm-agentic:
-  image: vllm/vllm-openai-rocm:nightly-09663abde0f50944a8d5ea30120666024b503faa
+  image: vllm/vllm-openai-rocm:nightly-49f31d7cee425a6d38f8c5bc76877986daf832ed
  model: deepseek-ai/DeepSeek-V4-Pro
  model-prefix: dsv4
  runner: cluster:mi355x-amds


🔴 This PR bumps the dsv4-fp4-mi355x-vllm-agentic image tag, raises dram-utilization (0.60→0.80), and rewrites the search-space in configs/amd-master.yaml, but no corresponding entry was appended to perf-changelog.yaml. Per AGENTS.md's 'Updating Docker images' section this entry is required — it's the mechanism that triggers the benchmark sweep for the changed config, so without it the PR's own stated test plan ('Full-sweep CI green') will not execute.

Extended reasoning...

AGENTS.md's "Updating Docker images" section (around line 152) states explicitly: "Update the image tag in the relevant configs/*-master.yaml and/or benchmarks/*.sh, update any related env vars / config params, and append a perf-changelog.yaml entry (required - triggers benchmarks)." AGENTS.md also documents the mechanism itself (line ~78): perf-changelog.yaml is read chronologically and drives which configs get benchmarked in a sweep; new entries must be appended to the end.

This PR does exactly the class of change that section calls out: it bumps the dsv4-fp4-mi355x-vllm-agentic image (nightly-09663abde... → nightly-49f31d7c...), raises dram-utilization from 0.60 to 0.80, and replaces the search-space with a single new DEP+lmcache arm at conc: 76 (configs/amd-master.yaml:1518-1522). These are precisely "image tag" and "related config params" changes.

The diff only touches two files: benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm.sh and configs/amd-master.yaml. perf-changelog.yaml is not modified anywhere in this PR, and the tail of that file has no entry referencing this PR or dsv4-fp4-mi355x-vllm-agentic — every other recent recipe-bump PR in the history (e.g. #2327, #2345, #2349, #2341, #2305) appended a corresponding changelog entry for the same kind of change. There is an older entry referencing dsv4-fp4-mi355x-vllm-agentic, but it belongs to the recipe's original creation (PR #2109) — since the changelog is append-only and read chronologically, that stale entry does not re-trigger a sweep for this PR's new image/config changes.

Concrete consequence: per the documented trigger mechanism, the full-sweep CI will not pick up dsv4-fp4-mi355x-vllm-agentic for re-benchmarking as a result of this PR's changes, since nothing was appended to signal that this config needs a fresh run. That directly undermines the PR's own stated Test plan checkbox — "Full-sweep CI green for dsv4-fp4-mi355x-vllm-agentic" — which cannot be exercised through the normal trigger path without the changelog entry. The DEP tuning and LMCache refresh this PR is meant to validate would ship unvalidated by the sweep infrastructure.

Fix: append a perf-changelog.yaml entry at the end of the file for dsv4-fp4-mi355x-vllm-agentic, following the existing schema (config-keys / description / pr-link), describing the image bump, dram-utilization change, and search-space rewrite, so the sweep triggers correctly.

claude · 2026-07-28T05:29:08Z

+      #- { tp: 8, ep: 1, dp-attn: false, kv-offloading: none, conc-list: [1, 4, 8, 16, 32, 40, 48] }
+      #- { tp: 8, ep: 1, dp-attn: true, kv-offloading: none, conc-list: [64, 72], router: { name: vllm-router, version: "0.1.14" } }
+      #- { tp: 8, ep: 1, dp-attn: false, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "9229067cec0b3a63bb8a39368d101db7ac0bc3c1" }, conc-list: [32, 40, 48] }
+      - { tp: 8, ep: 1, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "9229067cec0b3a63bb8a39368d101db7ac0bc3c1" }, conc-list: [76] }


🟡 The active dsv4-fp4-mi355x-vllm-agentic search-space arm still tags kv-offload-backend.version as the old LMCache PR-3853 SHA (9229067cec...), but dsv4_fp4_mi355x_vllm.sh now does git checkout v0.5.2 instead of that SHA. This is a pre-existing-pattern metadata field that isn't consumed by the launch script, so nothing breaks, but every run of this config will record the wrong LMCache version in benchmark result tracking. Fix: update the yaml version to "v0.5.2".

Extended reasoning...

The dsv4-fp4-mi355x-vllm-agentic entry in configs/amd-master.yaml has a single active search-space arm (the other two are commented out) that uses kv-offload-backend: { name: lmcache, version: "9229067cec0b3a63bb8a39368d101db7ac0bc3c1" }. That SHA is the old LMCache PR-3853 commit that dsv4_fp4_mi355x_vllm.sh used to check out. This PR's diff to the launch script replaces that with git checkout v0.5.2 (dropping the # https://github.com/LMCache/LMCache/pull/3853 comment and the SHA), but the corresponding yaml metadata for the still-active arm was not updated to match.

The version string is pure provenance metadata, not something that drives behavior: the launch script hardcodes the git ref directly and never reads any LMCache-version env var (compare to VLLM_ROUTER_VERSION, which actually is read from the yaml-equivalent hardcoded value). The yaml's kv-offload-backend dict is serialized via generate_sweep_configs.py, flows through run-sweep.yml's toJson(...) into the KV_OFFLOAD_BACKEND_METADATA env var, and is recorded verbatim by utils/agentic/aggregation/process_agentic_result.py (_validate_kv_offload_env / optional_kv_offload_backend_metadata) into each result's kv_offload_backend field. The only validation performed on that metadata is that name == KV_OFFLOAD_BACKEND (i.e. "lmcache" == "lmcache") — the version is never cross-checked against what's actually installed, and validation.py only special-cases strings starting with image:, so the stale SHA passes through untouched.

Concretely: every future run of the active { tp: 8, ep: 1, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "9229067cec0b3a63bb8a39368d101db7ac0bc3c1" } } arm will build and install LMCache v0.5.2 on the box, but the emitted benchmark result will report kv_offload_backend.version = "9229067cec0b3a63bb8a39368d101db7ac0bc3c1". Anyone looking at result provenance/tracking later (e.g. comparing performance across LMCache versions, or auditing which commit produced a given number) will be misled into thinking the run used the old PR-3853 SHA rather than v0.5.2. The PR description itself confirms the intent to move off that SHA ("LMCache: build v0.5.2 (was the PR-3853 SHA)"), so this is a straightforward oversight rather than a deliberate choice — the two now-commented-out arms in the same file still carry the same stale SHA too, but since they're inactive they don't currently emit misleading results.

This does not cause any functional failure or crash — the sweep will run and pass CI, since nothing validates the version value. The fix is a one-line yaml change: version: "9229067cec0b3a63bb8a39368d101db7ac0bc3c1" → version: "v0.5.2" on the active arm (and optionally the two commented-out arms, for consistency, though those aren't running).

github-actions · 2026-07-28T05:30:45Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30331521829
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30331521829

… rank --max-num-seqs is applied per scheduler. Under DP-attention each of the TP DP ranks runs its own scheduler (router spreads sessions across them), so use 2*CONC/TP (aggregate 2*CONC); pure-TP keeps 2*CONC. Mirrors the B300 recipe. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

….85 for lmcache The LMCache MP server (kv_buffer_device=cuda) holds ~32 GiB GPU staging memory per device before vLLM starts (only ~256/288 GiB free at init), so --gpu-memory-utilization 0.95 requested 273.6 GiB and every DP worker failed init_device with "Free memory ... less than desired GPU memory utilization". Derate the lmcache arm to 0.85 (244.8 GiB, ~11 GiB headroom); other arms keep 0.95. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

functionstackx

would u like to add mtp plz

seungrokj · 2026-07-28T07:20:56Z

would u like to add mtp plz

yes will quickly check if this setting works, then moving to spec dec.

github-actions · 2026-07-28T07:59:15Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30333375922
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30333375922

…m to GPU-resident c48 - dsv4_fp4_mi355x_vllm.sh: export EVAL_FRAMEWORK=lm-eval so run_eval never falls back to the agentic swebench default. - amd-master.yaml: switch the active search-space arm to TP8/DPA GPU-resident (kv-offloading: none) conc 48; comment out the lmcache dram c76 arm. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

github-actions · 2026-07-28T09:45:00Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30347718162
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30347718162

…-tokens to MNBT*2 MNBT*8 (65536) drove a bf16 einsum/bmm of M=65536 at warmup that segfaulted inside rocBLASLt/TensileLite (hipblasLtMatmulAlgoGetHeuristic -> loadPlaceholderLibrary). Drop the DEP8 prefill budget to MNBT*2 (16384), matching the validated B300 recipe, to avoid the oversized GEMM shape. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

… for all arms The GPU-resident (kv-none) c48 arm also hit the startup OOM: these MI355X nodes have a stable ~32 GiB/GPU carveout (only ~256/288 GiB free at init, independent of the KV-offload backend), so 0.95 (273.6 GiB) fails every DP worker. Collapse the lmcache-only derate into a flat GPU_MEM_UTIL=0.85 (244.8 GiB, ~11 GiB headroom) covering both the lmcache and GPU-resident arms. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

cquil11 · 2026-07-30T04:29:50Z

@seungrokj the AgentX/AIPerf harness has been updated, please merge origin/main into your branch and refresh your submission. Additional tuning may be necessary depending on the config. I apologize for any inconvenience.

This is an automated message.

seungrokj · 2026-07-30T07:31:18Z

closing in favor of MTP #2381

seungrokj requested a review from a team July 28, 2026 05:20

seungrokj requested review from 1am9trash, billishyahao, chunfangamd and yctseng0211 as code owners July 28, 2026 05:20

github-project-automation Bot added this to InferenceMAX Board Jul 28, 2026

seungrokj and others added 2 commits July 28, 2026 14:21

docs(perf-changelog): add dsv4-fp4-mi355x-vllm-agentic entry for #2378

e10d502

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Merge branch 'main' into amd/agentx_dsv4_vllm_0728

1b1a6f1

seungrokj added AMD full-sweep-fail-fast agentx AgentX benchmarks, recipes, and infrastructure labels Jul 28, 2026

[AMD] [AGENTX] dsv4-fp4-mi355x-vllm-agentic: pin lmcache to v0.5.2 in…

5237349

… master-config Align the agentic search-space kv-offload-backend version with the recipe (build tag v0.5.2, was the PR-3853 SHA 9229067). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

claude Bot reviewed Jul 28, 2026

View reviewed changes

SemiAnalysisAI deleted a comment from github-actions Bot Jul 28, 2026

functionstackx requested changes Jul 28, 2026

View reviewed changes

seungrokj removed the full-sweep-fail-fast label Jul 28, 2026

seungrokj and others added 2 commits July 28, 2026 20:08

seungrokj closed this Jul 30, 2026

github-project-automation Bot moved this to Done in InferenceMAX Board Jul 30, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[AMD] [AGENTX] dsv4-fp4-mi355x-vllm-agentic: DEP tuning + LMCache refresh - #2378

[AMD] [AGENTX] dsv4-fp4-mi355x-vllm-agentic: DEP tuning + LMCache refresh#2378
seungrokj wants to merge 9 commits into
mainfrom
amd/agentx_dsv4_vllm_0728

seungrokj commented Jul 28, 2026

Uh oh!

github-actions Bot commented Jul 28, 2026

Uh oh!

claude Bot Jul 28, 2026

Uh oh!

claude Bot Jul 28, 2026

Uh oh!

github-actions Bot commented Jul 28, 2026

Uh oh!

functionstackx left a comment

Uh oh!

seungrokj commented Jul 28, 2026

Uh oh!

github-actions Bot commented Jul 28, 2026

Uh oh!

github-actions Bot commented Jul 28, 2026

Uh oh!

cquil11 commented Jul 30, 2026

Uh oh!

seungrokj commented Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

seungrokj commented Jul 28, 2026

Summary

Test plan

Uh oh!

github-actions Bot commented Jul 28, 2026

Uh oh!

claude Bot Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 28, 2026

Uh oh!

functionstackx left a comment

Choose a reason for hiding this comment

Uh oh!

seungrokj commented Jul 28, 2026

Uh oh!

github-actions Bot commented Jul 28, 2026

Uh oh!

github-actions Bot commented Jul 28, 2026

Uh oh!

cquil11 commented Jul 30, 2026

Uh oh!

seungrokj commented Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants