Skip to content

[NV] Add H200 DeepSeek-V4-Pro AgentX recipes - #2364

Open
csahithi wants to merge 6 commits into
mainfrom
nv-dsv4-h200-sglang-agentx
Open

[NV] Add H200 DeepSeek-V4-Pro AgentX recipes#2364
csahithi wants to merge 6 commits into
mainfrom
nv-dsv4-h200-sglang-agentx

Conversation

@csahithi

@csahithi csahithi commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Add one 8xH200 aggregated TP8 (EP1/DP1) DeepSeek-V4-Pro FP8 Dynamo-SGLang AgentX recipe with EAGLE MTP and HiCache, sweeping concurrency [1,2,4,8,16] over a fixed serving topology. Uses the Marlin MoE backend and Dynamo header-based session affinity (X-Dynamo-Session-ID).

  • New recipe agg-h200-tp8-mtp-kvoffload.yaml + master key dsv4-fp8-h200-dynamo-sglang-agentic-agg (decode num-worker 0 so per-GPU accounting reflects the single aggregated TP8 worker).
  • benchmark_lib.sh: skip the legacy nvext conv-aware CLI routing when a recipe opts into the header path (AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID=true); the existing AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING opt-out and default are unchanged.
  • launch_h200-dgxc-slurm.sh: dsv4 fp8 model-path routing, srt-slurm v1.0.10 overlay for the agentic recipe, on-demand SGLang/nginx squash imports, and AgentX dataset / HF caches mounted into the multi-node agentic path.

@csahithi
dsv4 H200 agentic: add aggregated TP8 DeepSeek-V4-Pro Pareto sweep
59956b0 
Add one 8xH200 aggregated TP8 (EP1/DP1) DeepSeek-V4-Pro FP8 Dynamo-SGLang AgentX
recipe with EAGLE MTP and HiCache, sweeping concurrency [1,2,4,8,16] over a fixed
serving topology. Uses the Marlin MoE backend and Dynamo header-based session
affinity (X-Dynamo-Session-ID).

- New recipe agg-h200-tp8-mtp-kvoffload.yaml + master key
  dsv4-fp8-h200-dynamo-sglang-agentic-agg (decode num-worker 0 so per-GPU
  accounting reflects the single aggregated TP8 worker).
- benchmark_lib.sh: skip the legacy nvext conv-aware CLI routing when a recipe
  opts into the header path (AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID=true);
  the existing AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING opt-out and default are unchanged.
- launch_h200-dgxc-slurm.sh: dsv4 fp8 model-path routing, srt-slurm v1.0.10 overlay
  for the agentic recipe, on-demand SGLang/nginx squash imports, and AgentX dataset /
  HF caches mounted into the multi-node agentic path.
@csahithi
csahithi requested a review from a team July 27, 2026 22:23
@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

2 similar comments
@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30310586014
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30310586014

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30310920956
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30310920956

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30311003032
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30311003032

2 similar comments
@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30311003032
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30311003032

@github-actions

github-actions Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30311003032
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30311003032

@csahithi csahithi changed the title [WIP] [NV] Add H200 DeepSeek-V4-Pro AgentX recipes [NV] Add H200 DeepSeek-V4-Pro AgentX recipes Jul 28, 2026
@cquil11

cquil11 commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

/stage-results

@github-actions

github-actions Bot commented Jul 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

@cquil11 staged run 30311003032: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-07-27~r30311003032

This shared staging slot remains available until the next /stage-results request. Staging workflow

cquil11
cquil11 reviewed Jul 28, 2026
View reviewed changes
Comment thread runners/launch_h200-dgxc-slurm.sh Outdated
csahithi and others added 2 commits July 29, 2026 17:49
@csahithi
h200 dsv4 agentic: drop import_squash, use pre-staged cluster sqsh
61a9767 
The SGLang image is already staged on the H200 cluster at
/data/containers/*.sqsh, like the dsr1 multinode path which imports
nothing. Drop the import_squash helper and its now-unused NGINX_IMAGE;
the multinode path maps SQUASH_FILE into srtslurm.yaml directly.
@Ankur-singh
chore: merge origin/main and resolve perf changelog
5c3156c 
中文：合并 origin/main 并解决 perf-changelog 冲突
@Ankur-singh

Ankur-singh commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Revoking the standing `/reuse-sweep-run` authorization on this PR (removing the bare command comment from 2026-07-28).

A bare `/reuse-sweep-run` is standing rather than one-shot, and it has been silently swallowing sweeps here. The run at the current head, 30505397990, shows the gate emitting skip-pr-sweep: true with reuse-source-run-id empty — the sweep was suppressed outright, not even pinned to a source.

That matters because a real code change landed after the authorization: commit 61a9767f removed import_squash() and its two calls from runners/launch_h200-dgxc-slurm.sh, so the launcher no longer imports the sglang/nginx images and instead assumes pre-staged .sqsh files on the cluster (import_squash occurrences 3 → 0). That is launcher resolution logic on this config's exact path, and no sweep has ever exercised it. Separately, utils/aiperf moved ed2e040d → 381758a8, which changes AgentX pacing/idle semantics for the trace replay.

The last real evidence, 30311003032 (5/5 multi-node agentic), is at 193a0c96 and predates both.

Re-authorize with an explicit run ID once a fresh sweep lands.

@SemiAnalysisAI SemiAnalysisAI deleted a comment from csahithi Jul 30, 2026
@cquil11 ChatGPT Codex Connector

cquil11 commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

@csahithi the AgentX/AIPerf harness has been updated, please merge origin/main into your branch and refresh your submission. Additional tuning may be necessary depending on the config. I apologize for any inconvenience.

This is an automated message.

@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30506436017
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30506436017

@github-actions

github-actions Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30586537651
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30586537651

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cquil11 cquil11 cquil11 left review comments
@Ankur-singh Ankur-singh Awaiting requested review from Ankur-singh Ankur-singh is a code owner
@kedarpotdar-nv kedarpotdar-nv Awaiting requested review from kedarpotdar-nv kedarpotdar-nv is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

full-sweep-enabled NVIDIA

Projects

InferenceMAX Board
Status: No status

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@csahithi @cquil11 @Ankur-singh