test(ext-agents): guard tier selectors against false-green#9115
Open
glharper wants to merge 2 commits into
Open
test(ext-agents): guard tier selectors against false-green#9115glharper wants to merge 2 commits into
glharper wants to merge 2 commits into
Conversation
Centralize the Tier 0 and Tier 1 go test patterns, then validate that each selector matches tests, no test belongs to both tiers, and every Test_AIAgent_ test is assigned exactly once. This makes renamed or newly added tests fail CI instead of being silently skipped by go test -run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6eccd2f8-19dc-4da9-abf7-12eb690cf50c
glharper
requested review from
RickWinter,
danieljurek,
tg-msft and
vhvb1989
as code owners
📋 Prioritization NoteThanks for the contribution! The linked issue isn't in the current milestone yet. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds CI validation to prevent false-green Azure AI Agents tier test runs.
Changes:
- Centralizes and anchors Tier 0/1 selectors.
- Verifies all nine tests belong to exactly one tier.
- Reuses validated selectors during test execution.
jongio
reviewed
Jul 13, 2026
jongio
left a comment
jongio
left a comment
Member
There was a problem hiding this comment.
One non-blocking suggestion on the new validation step, inline. The guard logic itself checks out: I ran it against the current package and the two tier regexes select exactly the 4 Tier 0 and 5 Tier 1 tests with full coverage and no overlap.
Move the selector guard directly after setup-go so regex drift fails before building azd, the inspector dependency, or the agents extension. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6eccd2f8-19dc-4da9-abf7-12eb690cf50c
jongio
approved these changes
Jul 14, 2026
huimiu
approved these changes
Jul 14, 2026
Member
|
/check-enforcer override |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
ext-azure-ai-agents-testworkflow selects Tier 0 and Tier 1 functional tests withgo test -run <regex>. Go exits successfully when a regex matches no tests, so renaming or adding aTest_AIAgent_test without updating the tier selectors could silently produce a false-green CI run.Fix
Validate tier test selectorsstep that usesgo test -listto discover tests and fails when:Test_AIAgent_tests are discovered;The current inventory is validated as 4 Tier 0 tests and 5 Tier 1 tests. The actual test steps reuse the validated regex variables.
Validation
Test_AIAgent_Doctor_Helpfrom the Tier 0 selector in a local negative-control run; the guard failed and identified the uncovered test.git diff --checkpasses.Fixes #8778