Skip to content

[None][test] Add DeepSeek-V4-Pro perf sanity cases on GB300#16540

Open
chenfeiz0326 wants to merge 4 commits into
NVIDIA:mainfrom
chenfeiz0326:feat/dsv4-perf-sanity
Open

[None][test] Add DeepSeek-V4-Pro perf sanity cases on GB300#16540
chenfeiz0326 wants to merge 4 commits into
NVIDIA:mainfrom
chenfeiz0326:feat/dsv4-perf-sanity

Conversation

@chenfeiz0326

@chenfeiz0326 chenfeiz0326 commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

Add four end-to-end DeepSeek-V4-Pro (FP4, 1.6T MoE) disaggregated perf-sanity cases on GB300, covering the concurrency sweep:

Case Concurrency Shape (ctx × TP + gen × TP) Nodes / GPUs MoE backend (gen) MTP
con8 8 1×dep4 + 4×tep8 9 / 36 TRTLLM 3
con180 180 3×dep4 + 1×dep32 11 / 44 MEGAMOE_DEEPGEMM 3
con666 666 6×dep4 + 1×dep16 10 / 40 MEGAMOE_DEEPGEMM 3
con4301 4,301 12×dep4 + 1×dep8 14 / 56 MEGAMOE_DEEPGEMM 1

Each config runs in three test modes (aggr_upload-ctx_only-, disagg_upload-gen_only-, disagg_upload-e2e-*), so this PR ends up with 4 aggregated + 8 multi-node stages.

What's in this PR

New yaml schema keys (backward-compatible) — environment.ctx_worker_env_var and environment.gen_worker_env_var — merged with the shared environment.worker_env_var in test_perf_sanity.py and both submit scripts (jenkins/scripts/perf/{,local/}submit.py). Existing yamls with only worker_env_var are unaffected. Used by these configs to apply PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True on the ctx worker only (long-prompt path).

4 new disagg configs under tests/scripts/perf-sanity/disaggregated/gb300_deepseek-v4-pro-fp4_8k1k_*.yaml. Use `<dataset_file>` placeholder → random synthetic dataset at runtime (mirrors GLM-5 perf-sanity pattern).

Model registration — deepseek_v4_pro_fp4 → DeepSeek-V4-Pro in tests/integration/defs/perf/_model_paths.py.

In-repo MoE load-balancer yamls — tests/scripts/perf-sanity/disaggregated/deepseek-v4-pro-eplb/moe_load_balancer_{ctx_ep4_384, gen_ep{4,8,16,32}_slots{384,416}}.yaml (nine files, cherry-picked from ibc/bench-trtllm-disagg) so CI doesn't need external filesystem access. test_perf_sanity.py's generate_extra_llm_api_config now resolves relative moe_config.load_balancer paths against get_llm_root().

Test-db entries — 4 new l0_gb300_multi_nodes_perf_sanity_ctx{1,3,6,12}_node1_gpu4_gen*_node*_gpu*.yml (each holding one e2e + one gen_only), plus 4 new aggr_upload-ctx_only-* entries in l0_gb300_multi_gpus_perf_sanity.yml (split bumped 3 → 4).

Jenkins wiring — 4 new buildStageConfigs blocks in the GB300 disagg section of L0_Test.groovy (9, 10, 11, 14 nodes).

Test plan

  • Case 1 (con8, 9 nodes) e2e smoke test on OCI-AGA GB300 — verify install + benchmark end-to-end
  • Case 2 (con180, 11 nodes) e2e run
  • Case 3 (con666, 10 nodes) e2e run
  • Case 4 (con4301, 14 nodes) e2e run
  • Confirm PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True lands in CTX_WORKER_ENV_VARS but NOT in GEN_WORKER_ENV_VARS in generated slurm_launch.sh (validated locally against a DSv3.2 yaml to confirm backward compat)
  • gen_only + ctx_only mode runs (deferred to follow-up)

Notes

  • Local run scripts require absolute paths for --run-sh / --install-sh (relative paths break because the container CWD is the outputs dir, not the repo root).
  • GB300 build: slurm_install.sh currently hard-codes --cuda_architectures '100-real' (GB200). GB300 needs 103-real. Not fixed in this PR; recommend a follow-up that reads the target arch from yaml metadata.supported_gpus.

Summary by CodeRabbit

  • New Features

    • Added DeepSeek-V4-Pro FP4 performance sanity coverage for GB300 systems.
    • Added disaggregated context/generation benchmark configurations across 36-, 40-, 44-, and 56-GPU setups.
    • Added role-specific environment settings for context and generation workers.
    • Added support for DeepSeek-V4-Pro model discovery and MoE load-balancer configurations.
  • Bug Fixes

    • Improved model-path handling for relative load-balancer configuration paths.
    • Expanded GB300 post-merge test coverage and split scheduling.

@coderabbitai

coderabbitai Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Adds role-specific context/generation environment handling, a DeepSeek-V4-Pro model mapping, GB300 disaggregated PerfSanity workloads, post-merge scheduling entries, and fixed MoE load-balancer assignment configurations.

Changes

GB300 PerfSanity

Layer / File(s) Summary
Role-specific worker environments
jenkins/scripts/perf/*.py, tests/integration/defs/perf/test_perf_sanity.py
Environment configuration now produces separate context and generation worker values, applies them to ServerConfig, and uses them in disaggregated launch scripts. Relative MoE load-balancer paths are normalized against the model root.
Benchmark model and workload definitions
tests/integration/defs/perf/_model_paths.py, tests/scripts/perf-sanity/disaggregated/gb300_*
Adds the DeepSeek-V4-Pro model mapping and four GB300 benchmark definitions with topology, runtime, NIXL, speculative decoding, and role-specific worker settings.
Integration and post-merge scheduling
tests/integration/test_lists/test-db/l0_gb300_*, jenkins/L0_Test.groovy
Registers ctx-only and multi-node tests for 36, 40, 44, and 56 GPU layouts, and expands the GB300 post-merge stage split matrix.
MoE load-balancer assignment data
tests/scripts/perf-sanity/disaggregated/deepseek-v4-pro-eplb/*
Adds initial assignment mappings for context and generation EPLB configurations with 384- or 416-slot domains and zero update iterations.

Estimated code review effort: 3 (Moderate) | ~30 minutes

Sequence Diagram(s)

sequenceDiagram
  participant DisaggYAML
  participant PerfSanityTestConfig
  participant ServerConfig
  participant submit_py
  DisaggYAML->>PerfSanityTestConfig: provide shared and role-specific environment values
  PerfSanityTestConfig->>ServerConfig: assign context and generation environments
  submit_py->>submit_py: compose role-specific launch prefixes
  submit_py->>ServerConfig: generate disaggregated worker launch environments
Loading

Possibly related PRs

  • NVIDIA/TensorRT-LLM#10912: Both changes expand disaggregated PerfSanity post-merge stage configurations in jenkins/L0_Test.groovy.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed Docstring coverage is 80.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title is concise and accurately summarizes the main change: adding DeepSeek-V4-Pro perf sanity cases on GB300.
Description check ✅ Passed The description is mostly complete, with a clear summary, implementation details, test plan, and notes; the checklist section is the main omission.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@jenkins/L0_Test.groovy`:
- Around line 4963-4966: Update the stage-name performance checks in both the
x86 and SBSA Slurm job closures to recognize names containing “-PerfSanity-” as
performance tests alongside the existing “-Perf-” condition. Ensure the
resulting perfMode passed to runLLMTestlistOnSlurm is true for these stages so
the --perf workflow is preserved.

In `@jenkins/scripts/perf/submit.py`:
- Around line 538-541: Update the base_prefix construction to remove the
unnecessary f-string prefixes while preserving the generated shell variable
syntax. Convert the doubled braces around SLURM_LOCALID to literal braces so the
resulting command still contains ${SLURM_LOCALID} and satisfies Ruff F541.

In
`@tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_ctx12_node1_gpu4_gen1_node2_gpu8.yml`:
- Around line 1-2: Add the repository’s standard NVIDIA copyright header at the
beginning of the YAML file, before the version declaration. Preserve the
existing version and configuration key unchanged.

In
`@tests/scripts/perf-sanity/disaggregated/gb300_deepseek-v4-pro-fp4_8k1k_con8_ctx1_dep4_gen4_tep8_eplb0_mtp3_ccb-NIXL.yaml`:
- Around line 1-2: Add the repository-standard NVIDIA copyright header at the
beginning of each new YAML file:
tests/scripts/perf-sanity/disaggregated/gb300_deepseek-v4-pro-fp4_8k1k_con8_ctx1_dep4_gen4_tep8_eplb0_mtp3_ccb-NIXL.yaml
(lines 1-2),
tests/scripts/perf-sanity/disaggregated/gb300_deepseek-v4-pro-fp4_8k1k_con4301_ctx12_dep4_gen1_dep8_eplb384_mtp1_ccb-NIXL.yaml
(lines 1-2),
tests/scripts/perf-sanity/disaggregated/gb300_deepseek-v4-pro-fp4_8k1k_con666_ctx6_dep4_gen1_dep16_eplb384_mtp3_ccb-NIXL.yaml
(lines 1-2),
tests/scripts/perf-sanity/disaggregated/gb300_deepseek-v4-pro-fp4_8k1k_con180_ctx3_dep4_gen1_dep32_eplb384_mtp3_ccb-NIXL.yaml
(lines 1-2),
tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen4_node2_gpu8.yml
(lines 1-2),
tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_ctx6_node1_gpu4_gen1_node4_gpu16.yml
(lines 1-2), and
tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_ctx3_node1_gpu4_gen1_node8_gpu32.yml
(lines 1-2), while preserving each file’s existing YAML content.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8e14155b-d131-457e-95ad-9d4ac5ac36da

📥 Commits

Reviewing files that changed from the base of the PR and between 9fdfd10 and b41b8fa.

📒 Files selected for processing (23)
  • jenkins/L0_Test.groovy
  • jenkins/scripts/perf/local/submit.py
  • jenkins/scripts/perf/submit.py
  • tests/integration/defs/perf/_model_paths.py
  • tests/integration/defs/perf/test_perf_sanity.py
  • tests/integration/test_lists/test-db/l0_gb300_multi_gpus_perf_sanity.yml
  • tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_ctx12_node1_gpu4_gen1_node2_gpu8.yml
  • tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen4_node2_gpu8.yml
  • tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_ctx3_node1_gpu4_gen1_node8_gpu32.yml
  • tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_ctx6_node1_gpu4_gen1_node4_gpu16.yml
  • tests/scripts/perf-sanity/disaggregated/deepseek-v4-pro-eplb/moe_load_balancer_ctx_ep4_384.yaml
  • tests/scripts/perf-sanity/disaggregated/deepseek-v4-pro-eplb/moe_load_balancer_gen_ep16_slots384.yaml
  • tests/scripts/perf-sanity/disaggregated/deepseek-v4-pro-eplb/moe_load_balancer_gen_ep16_slots416.yaml
  • tests/scripts/perf-sanity/disaggregated/deepseek-v4-pro-eplb/moe_load_balancer_gen_ep32_slots384.yaml
  • tests/scripts/perf-sanity/disaggregated/deepseek-v4-pro-eplb/moe_load_balancer_gen_ep32_slots416.yaml
  • tests/scripts/perf-sanity/disaggregated/deepseek-v4-pro-eplb/moe_load_balancer_gen_ep4_slots384.yaml
  • tests/scripts/perf-sanity/disaggregated/deepseek-v4-pro-eplb/moe_load_balancer_gen_ep4_slots416.yaml
  • tests/scripts/perf-sanity/disaggregated/deepseek-v4-pro-eplb/moe_load_balancer_gen_ep8_slots384.yaml
  • tests/scripts/perf-sanity/disaggregated/deepseek-v4-pro-eplb/moe_load_balancer_gen_ep8_slots416.yaml
  • tests/scripts/perf-sanity/disaggregated/gb300_deepseek-v4-pro-fp4_8k1k_con180_ctx3_dep4_gen1_dep32_eplb384_mtp3_ccb-NIXL.yaml
  • tests/scripts/perf-sanity/disaggregated/gb300_deepseek-v4-pro-fp4_8k1k_con4301_ctx12_dep4_gen1_dep8_eplb384_mtp1_ccb-NIXL.yaml
  • tests/scripts/perf-sanity/disaggregated/gb300_deepseek-v4-pro-fp4_8k1k_con666_ctx6_dep4_gen1_dep16_eplb384_mtp3_ccb-NIXL.yaml
  • tests/scripts/perf-sanity/disaggregated/gb300_deepseek-v4-pro-fp4_8k1k_con8_ctx1_dep4_gen4_tep8_eplb0_mtp3_ccb-NIXL.yaml

Comment thread jenkins/L0_Test.groovy Outdated
Comment thread jenkins/scripts/perf/submit.py

@BowenFu BowenFu left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — additive perf-sanity test/CI (DeepSeek-V4-Pro on GB300); new ctx/gen worker_env_var yaml keys are backward-compatible; existing yamls unaffected.

@chenfeiz0326

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN4-NODE2-GPU8-Post-Merge-1,GB300-40_GPUs-10_Nodes-PyTorch-Disagg-PerfSanity-CTX6-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-1,GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1,GB300-56_GPUs-14_Nodes-PyTorch-Disagg-PerfSanity-CTX12-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

Adds four end-to-end disaggregated perf-sanity cases for DeepSeek-V4-Pro
(FP4, 1.6T MoE) on GB300, covering the concurrency sweep (single-user
latency at con=8, mid throughput at con=180 and con=666, max throughput
at con=4301).

New per-role env-var support
- environment.ctx_worker_env_var and environment.gen_worker_env_var yaml
  keys (both optional). Merged with the shared environment.worker_env_var
  at ServerConfig construction time in test_perf_sanity.py and in both
  submit scripts (jenkins/scripts/perf/{local/,}submit.py). Existing yamls
  with only worker_env_var are unaffected.
- Used by the new DSv4 configs to apply
  PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True on the ctx worker only
  (long-prompt path where expandable_segments avoids CUDA fragmentation).

New DSv4-Pro configs (all 8k1k, GB300, NIXL, dep4 ctx / MegaMoE gen)
- gb300_deepseek-v4-pro-fp4_8k1k_con8_ctx1_dep4_gen4_tep8_eplb0_mtp3 (1 ctx x 4GPU + 4 gen x 8GPU = 9 nodes / 36 GPUs)
- gb300_deepseek-v4-pro-fp4_8k1k_con180_ctx3_dep4_gen1_dep32_eplb384_mtp3 (3 ctx + 1 gen x 32GPU = 11 nodes / 44 GPUs)
- gb300_deepseek-v4-pro-fp4_8k1k_con666_ctx6_dep4_gen1_dep16_eplb384_mtp3 (6 ctx + 1 gen x 16GPU = 10 nodes / 40 GPUs)
- gb300_deepseek-v4-pro-fp4_8k1k_con4301_ctx12_dep4_gen1_dep8_eplb384_mtp1 (12 ctx + 1 gen x 8GPU = 14 nodes / 56 GPUs)

Each config is registered in three test modes (aggr_upload-ctx_only-*,
disagg_upload-gen_only-*, disagg_upload-e2e-*) via the aggregated test-db
(l0_gb300_multi_gpus_perf_sanity.yml, split bumped 3 -> 4) and four new
multi-node test-db yamls (one per shape).

Companion pieces
- tests/integration/defs/perf/_model_paths.py: add
  deepseek_v4_pro_fp4 -> DeepSeek-V4-Pro
- test_perf_sanity.py generate_extra_llm_api_config: resolve
  moe_config.load_balancer relative paths against get_llm_root(), so
  configs can reference the in-repo eplb yamls.
- tests/scripts/perf-sanity/disaggregated/deepseek-v4-pro-eplb/: nine
  offline moe-load-balancer yamls (ctx_ep4_384 + gen_ep{4,8,16,32}_slots{384,416})
  cherry-picked from bench-trtllm-disagg so CI does not need external
  filesystem access.
- jenkins/L0_Test.groovy: four new buildStageConfigs entries under the
  GB300 disagg section (9, 10, 11, 14 nodes), plus the aggregated split
  bump.

Signed-off-by: chenfeiz <chenfeiz@nvidia.com>
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
…erverConfig

The new DeepSeek-V4-Pro configs added in the previous commit reference the
in-repo eplb YAMLs via a repo-relative path string (moe_config.load_balancer:
tests/scripts/perf-sanity/disaggregated/deepseek-v4-pro-eplb/...yaml). The
TRT-LLM engine accepts either an inline dict or a path string via
MoeLoadBalancerConfig, and generate_extra_llm_api_config already resolves
the relative path to an absolute one before writing the extra-llm-api-config
yaml. But ServerConfig.__init__ was still treating the field as always-dict
and crashed at pytest collection:

  perf/test_perf_sanity.py:470: AttributeError:
    'str' object has no attribute 'get'

Add an isinstance(str) branch that zeros out the inline load_balancer_num_slots
and load_balancer_layer_updates_per_iter — those metrics live inside the
referenced YAML and aren't scraped by perf-sanity's DB fields anyway.

Signed-off-by: chenfeiz <chenfeiz@nvidia.com>
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
- Drop disagg_upload-gen_only-* entries from the four new GB300 test-db
  yamls; keep only the disagg_upload-e2e-* entry. gen_only variants are
  deferred to a follow-up PR so this PR ships the single mode the
  concurrency-sweep table calls for.
- Bump each new buildStageConfigs testCount from 2 → 1 in L0_Test.groovy
  now that each test-db yaml holds exactly one active test.
- Drop the extraneous f-prefix on the base_prefix string in
  jenkins/scripts/perf/submit.py's disagg env-var block (Ruff F541).

Signed-off-by: chenfeiz <chenfeiz@nvidia.com>
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
Fold the two-line string concat into a single line so pre-commit's
line-length / auto-fold formatter passes.

Signed-off-by: chenfeiz <chenfeiz@nvidia.com>
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
@chenfeiz0326
chenfeiz0326 force-pushed the feat/dsv4-perf-sanity branch from a529cf9 to f846061 Compare July 17, 2026 15:41
@chenfeiz0326

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --stage-list "GB300-36_GPUs-9_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN4-NODE2-GPU8-Post-Merge-1,GB300-40_GPUs-10_Nodes-PyTorch-Disagg-PerfSanity-CTX6-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-1,GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1,GB300-56_GPUs-14_Nodes-PyTorch-Disagg-PerfSanity-CTX12-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #60001 [ run ] triggered by Bot. Commit: f846061 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #60001 [ run ] completed with state FAILURE. Commit: f846061
/LLM/main/L0_MergeRequest_PR pipeline #48393 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@yufeiwu-nv
yufeiwu-nv removed their request for review July 17, 2026 23:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants