test(e2e): add multi-GPU MIG profile coverage - #9021
Conversation
There was a problem hiding this comment.
Pull request overview
This PR expands GPU managed-experience E2E coverage for NVIDIA MIG on Ubuntu 24.04 by adding a multi-GPU scenario and tightening the shared MIG validators to assert exact, VM-wide expected results (rather than allowing partial/one-GPU partitioning to pass).
Changes:
- Updated MIG validators to require exact per-GPU MIG-mode enablement and exact MIG-instance counts.
- Updated existing single-GPU MIG scenarios to use the new exact-count validators.
- Added a new Ubuntu 24.04 multi-GPU MIG scenario.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| e2e/validators.go | Makes MIG validation stricter by checking per-GPU status and exact instance counts. |
| e2e/scenario_gpu_managed_experience_test.go | Updates existing MIG tests to use strict validators and adds a new multi-GPU MIG scenario. |
Comments suppressed due to low confidence (1)
e2e/scenario_gpu_managed_experience_test.go:656
- The PR description calls out validating MIG 2g.10gb instances, but this scenario is currently asserting "MIG 2g.20gb". If the intent is ND96asr_v4 (A100 40GB), the MIG2g profile corresponds to 2g.10gb, so the expected profile string here should match that.
ValidateMIGInstancesCreated(ctx, s, "MIG 2g.20gb", totalMIGInstances)
Windows Unit Test Results 3 files 11 suites 49s ⏱️ Results for commit 4d63844. ♻️ This comment has been updated with latest results. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
e2e/validators.go:2627
- The failure message passed to execScriptOnVMForScenarioValidateExitCode is misleading now that the script only queries MIG mode (it will still exit 0 when MIG is "Disabled"). This message should describe the command failure (querying MIG mode), while the subsequent assertions handle the enabled/disabled validation.
command := []string{
"set -ex",
"sudo nvidia-smi --query-gpu=mig.mode.current --format=csv,noheader",
}
execResult := execScriptOnVMForScenarioValidateExitCode(ctx, s, strings.Join(command, "\n"), 0, "MIG mode is not enabled")
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
e2e/scenario_gpu_managed_experience_test.go:627
- The new multi-GPU A100 MIG scenario doesn’t set
Location, while the other A100/MIG scenarios in this file pin towestus2. Leaving it unset can cause this scenario to run in a region without capacity forStandard_NC48ads_A100_v4, increasing flakiness and time spent searching for capacity.
Description: "Tests that a MIG profile is applied to every GPU on an Ubuntu 24.04 multi-GPU VM",
K8sSystemPoolSKU: "Standard_D2s_v3",
Tags: Tags{
GPU: true,
},
What this PR does / why we need it:
Adds an Ubuntu 24.04 E2E scenario that applies the
MIG2gprofile to a two-GPUStandard_NC48ads_A100_v4VM and validates the full VM-wide result:MIG 2g.20gbinstances are creatednvidia.com/gpu=6The shared MIG validators now assert exact GPU and MIG-instance counts, so the test cannot pass after partitioning only one card. Existing single-GPU MIG scenarios were updated to use the same exact-count validation.
Which issue(s) this PR fixes:
Fixes #
Testing:
cd e2e && go test ./... -run "^$" -count=1Test_Ubuntu2404_NvidiaDevicePluginRunning_MIG_MultiGPUpassed with 2 GPUs, 6MIG 2g.20gbinstances, and a schedulable GPU workloadTest_Ubuntu2404_NvidiaDevicePluginRunning_MIG_MultiGPUpassed in 1487.40sgit diff --check