Skip to content

ml/azure-ai-ml: audit and improve e2etest skip/live_test_only marker documentation#48232

Draft
lavakumarrepala with Copilot wants to merge 3 commits into
mainfrom
copilot/audit-e2e-test-modules
Draft

ml/azure-ai-ml: audit and improve e2etest skip/live_test_only marker documentation#48232
lavakumarrepala with Copilot wants to merge 3 commits into
mainfrom
copilot/audit-e2e-test-modules

Conversation

Copilot AI commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

All @pytest.mark.skip and @pytest.mark.live_test_only markers across ~35 e2etest directories under sdk/ml/azure-ai-ml/tests/**/e2etests were audited. Bare markers and vague reasons have been replaced with actionable documentation.

Changes

Bare @pytest.mark.skip() → documented reasons

All @pytest.mark.skip() calls without a reason= now include one, covering Docker daemon requirements (local endpoint tests), infrastructure unavailability, and API removal.

Vague reasons replaced with root causes

  • "investigate later", "Skip for not ready." → specific explanation of the blocking condition
  • "Flaky test" → root cause (HTTP timeout under load, no retry logic)
  • "request header size being too large" → prerequisite context (materialization store must be enabled on sdk_vnext_cli_fs)

ADO work item references standardized

Internal Azure DevOps IDs prefixed with ADO for clarity:

# Before
@pytest.mark.skip(reason="Task 1791832: Inefficient...")
# After
@pytest.mark.skip(reason="TODO (ADO 1791832): The full-model-list operation is slow enough to cause pipeline timeouts...")

Recurring skip patterns — improved across all instances

Pattern Files affected Root cause added
"parallel e2e recording issue" 8+ markers in dsl/pipeline Non-deterministic parallel component step ordering in test-proxy recordings
"Spark version Upgrade" 6+ markers Spark version incompatibility in test environment
TODO (2370129) dsl + pipeline "Cannot find pipeline run" recording ID mismatch
TODO (2177353) dsl + pipeline + control_flow Non-deterministic component version ordering between recording and playback

Specific test skips clarified

  • test_promote_model: _prepare_to_copy was removed from the SDK; test must be rewritten to use MLClient.models.share()
  • Monitoring tests (test_monitor_schedule.py): iris-endpoint resources were deleted from the test workspace; tests need a replacement endpoint
  • test_model_archive_restore_container: uses client.models.list() on every assertion, causing pipeline timeouts (ADO 1791832)

Not addressed (out of scope)

11 @pytest.mark.live_test_only tests across 5 files remain unchanged. These have stale recordings that no longer match current sanitizer patterns. Re-enabling them requires:

  1. A live Azure ML workspace with AZURE_TEST_RUN_LIVE=true
  2. Re-running the tests to generate new recordings
  3. Pushing recordings to Azure/azure-sdk-assets (tag: python/ml/azure-ai-ml_d3ebe00cb3)
  4. Removing the live_test_only marker

Affected files: test_connections.py, test_datastore.py, test_environment.py, test_component.py, test_dsl_pipeline_samples.py.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI and others added 2 commits July 23, 2026 17:00
…odules

Co-authored-by: lavakumarrepala <221403938+lavakumarrepala@users.noreply.github.com>
… final batch

Co-authored-by: lavakumarrepala <221403938+lavakumarrepala@users.noreply.github.com>
Copilot AI changed the title [WIP] Audit and update e2e tests for live workspace recordings ml/azure-ai-ml: audit and improve e2etest skip/live_test_only marker documentation Jul 23, 2026
Copilot AI requested a review from lavakumarrepala July 23, 2026 17:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants