Skip to content

test(e2e): add multi-GPU MIG profile coverage - #9021

Open
karenychen wants to merge 6 commits into
mainfrom
codex/e2e-multi-gpu-mig-profile
Open

test(e2e): add multi-GPU MIG profile coverage#9021
karenychen wants to merge 6 commits into
mainfrom
codex/e2e-multi-gpu-mig-profile

Conversation

@karenychen

@karenychen karenychen commented Jul 23, 2026

Copy link
Copy Markdown

What this PR does / why we need it:

Adds an Ubuntu 24.04 E2E scenario that applies the MIG2g profile to a two-GPU Standard_NC48ads_A100_v4 VM and validates the full VM-wide result:

  • the NVIDIA device plugin service is running
  • MIG mode is enabled on both A100 GPUs
  • 6 MIG 2g.20gb instances are created
  • the node advertises nvidia.com/gpu=6
  • a MIG workload can be scheduled and run successfully

The shared MIG validators now assert exact GPU and MIG-instance counts, so the test cannot pass after partitioning only one card. Existing single-GPU MIG scenarios were updated to use the same exact-count validation.

Which issue(s) this PR fixes:

Fixes #

Testing:

  • cd e2e && go test ./... -run "^$" -count=1
  • Local E2E: Test_Ubuntu2404_NvidiaDevicePluginRunning_MIG_MultiGPU passed with 2 GPUs, 6 MIG 2g.20gb instances, and a schedulable GPU workload
  • PR GPU E2E build 173548093: Test_Ubuntu2404_NvidiaDevicePluginRunning_MIG_MultiGPU passed in 1487.40s
  • git diff --check

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR expands GPU managed-experience E2E coverage for NVIDIA MIG on Ubuntu 24.04 by adding a multi-GPU scenario and tightening the shared MIG validators to assert exact, VM-wide expected results (rather than allowing partial/one-GPU partitioning to pass).

Changes:

  • Updated MIG validators to require exact per-GPU MIG-mode enablement and exact MIG-instance counts.
  • Updated existing single-GPU MIG scenarios to use the new exact-count validators.
  • Added a new Ubuntu 24.04 multi-GPU MIG scenario.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
e2e/validators.go Makes MIG validation stricter by checking per-GPU status and exact instance counts.
e2e/scenario_gpu_managed_experience_test.go Updates existing MIG tests to use strict validators and adds a new multi-GPU MIG scenario.
Comments suppressed due to low confidence (1)

e2e/scenario_gpu_managed_experience_test.go:656

  • The PR description calls out validating MIG 2g.10gb instances, but this scenario is currently asserting "MIG 2g.20gb". If the intent is ND96asr_v4 (A100 40GB), the MIG2g profile corresponds to 2g.10gb, so the expected profile string here should match that.
				ValidateMIGInstancesCreated(ctx, s, "MIG 2g.20gb", totalMIGInstances)

Comment thread e2e/scenario_gpu_managed_experience_test.go
Comment thread e2e/validators.go Outdated
Copilot AI review requested due to automatic review settings July 28, 2026 18:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

@github-actions

github-actions Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Windows Unit Test Results

  3 files   11 suites   49s ⏱️
381 tests 381 ✅ 0 💤 0 ❌
384 runs  384 ✅ 0 💤 0 ❌

Results for commit 4d63844.

♻️ This comment has been updated with latest results.

Copilot AI review requested due to automatic review settings July 29, 2026 21:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

e2e/validators.go:2627

  • The failure message passed to execScriptOnVMForScenarioValidateExitCode is misleading now that the script only queries MIG mode (it will still exit 0 when MIG is "Disabled"). This message should describe the command failure (querying MIG mode), while the subsequent assertions handle the enabled/disabled validation.
	command := []string{
		"set -ex",
		"sudo nvidia-smi --query-gpu=mig.mode.current --format=csv,noheader",
	}
	execResult := execScriptOnVMForScenarioValidateExitCode(ctx, s, strings.Join(command, "\n"), 0, "MIG mode is not enabled")

Copilot AI review requested due to automatic review settings July 30, 2026 17:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

e2e/scenario_gpu_managed_experience_test.go:627

  • The new multi-GPU A100 MIG scenario doesn’t set Location, while the other A100/MIG scenarios in this file pin to westus2. Leaving it unset can cause this scenario to run in a region without capacity for Standard_NC48ads_A100_v4, increasing flakiness and time spent searching for capacity.
		Description:      "Tests that a MIG profile is applied to every GPU on an Ubuntu 24.04 multi-GPU VM",
		K8sSystemPoolSKU: "Standard_D2s_v3",
		Tags: Tags{
			GPU: true,
		},

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants