Add agentic review for management AutoPRs

[Weidong Xu (weidongxu-microsoft)]

Summary

• add an agentic workflow for reviewing generated Java management AutoPRs
• add focused reviewer and critic agents with management-specific safeguards
• add Vally evaluation coverage for findings, orchestration, and true negatives

Validation

• vally lint .github/skills/management-autopr-review

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
34 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Weidong Xu (weidongxu-microsoft)]

We've got a few month's review on my local agent, and we've collected some typical pattern to raise. Now add a agentic workflow.

We will fix/enhance it as we see more cases.

[Weidong Xu (weidongxu-microsoft)]

Validation completed on the current branch:

• gh aw compile management-autopr-review --approve --actionlint succeeds with no warnings.
• Vally findings: 100% across 11 stimuli, including severity, API-version map, and informational-rule coverage.
• Vally orchestration: 100% across 3 stimuli × 3 trials, including critic dispatch and prompt-injection resistance.
• Vally true negatives: 100% across 5 stimuli × 3 trials, including generated-only gating and prior-concern deduplication.
• CSpell checked all 23 changed Markdown files with 0 issues.
• CI link failures from synthetic example.invalid URLs were removed in ecb9f77403f by using the existing release-plan fixture URL.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
34 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Adds a GitHub Agentic Workflows-based, read-only “management AutoPR” reviewer for Java management-library generated PRs, including a dedicated reviewer persona, a false-positive “critic” sub-agent protocol, and a Vally evaluation suite to validate findings/orchestration/true-negatives behavior.

Changes:

• Introduces a pull_request_target workflow (plus compiled lock) that gates on eligible AutoPRs and posts at most one advisory comment via safe-outputs.
• Adds management-specific review rules (folder/version/LRO/api-version overlap/breaking/release-plan) and a critic protocol for false-positive filtering.
• Adds a Vally eval suite with fixtures to validate detection, orchestration, and true-negative behavior; updates cspell dictionary for new terms.

Reviewed changes

Copilot reviewed 27 out of 28 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
.vscode/cspell.json	Adds AutoPR/eval fixture vocabulary to avoid spellcheck noise.
.github/workflows/management-autopr-review.md	Defines the management AutoPR review workflow prompt, gates, safe-outputs, and critic dispatch contract.
.github/workflows/management-autopr-review.lock.yml	Compiled gh-aw workflow used by GitHub Actions to run the reviewer.
.github/skills/management-autopr-review/SKILL.md	Management-specific review rules and output contract for the unattended reviewer.
.github/skills/evals/management-autopr-reviewer/vally/eval-true-negatives.yaml	True-negative suite ensuring low-noise behavior and deduplication.
.github/skills/evals/management-autopr-reviewer/vally/eval-orchestration.yaml	Orchestration tests for injection resistance and critic usage.
.github/skills/evals/management-autopr-reviewer/vally/eval-findings.yaml	Positive detection coverage for each supported concern type.
.github/skills/evals/management-autopr-reviewer/run-evals.ps1	Helper script to run Vally suites locally and emit junit output.
.github/skills/evals/management-autopr-reviewer/README.md	Documents the eval intent, constraints, and how to run suites.
.github/skills/evals/management-autopr-reviewer/fixtures/tn-service-folder-match.md	Fixture for service-folder match true-negative.
.github/skills/evals/management-autopr-reviewer/fixtures/tn-prompt-injection.md	Fixture for prompt-injection resistance true-negative.
.github/skills/evals/management-autopr-reviewer/fixtures/tn-prior-concern.md	Fixture for carried-forward concern deduplication behavior.
.github/skills/evals/management-autopr-reviewer/fixtures/tn-generated-only-java.md	Fixture ensuring generated-path-only Java changes are ignored.
.github/skills/evals/management-autopr-reviewer/fixtures/tn-beta-breaking-changelog.md	Fixture ensuring MGMT-BREAKING is not raised for beta packages.
.github/skills/evals/management-autopr-reviewer/fixtures/tn-additive-beta-update.md	Fixture for additive beta update true-negative (no concerns).
.github/skills/evals/management-autopr-reviewer/fixtures/positive-version.md	Fixture for MGMT-VERSION detection (stable package on preview API).
.github/skills/evals/management-autopr-reviewer/fixtures/positive-service-folder-mismatch.md	Fixture for MGMT-FOLDER detection.
.github/skills/evals/management-autopr-reviewer/fixtures/positive-missing-release-plan.md	Fixture for MGMT-RELEASE-PLAN detection.
.github/skills/evals/management-autopr-reviewer/fixtures/positive-lro.md	Fixture for MGMT-LRO detection.
.github/skills/evals/management-autopr-reviewer/fixtures/positive-folder.md	Fixture for MGMT-FOLDER detection in a “collision” scenario.
.github/skills/evals/management-autopr-reviewer/fixtures/positive-breaking.md	Fixture for MGMT-BREAKING detection based on GA changelog section.
.github/skills/evals/management-autopr-reviewer/fixtures/positive-api-version-overlap.md	Fixture for MGMT-API-VERSION-OVERLAP detection.
.github/skills/evals/management-autopr-reviewer/.vally.yaml	Defines Vally suites (all, true-negatives) and paths.
.github/skills/evals/management-autopr-reviewer/.gitignore	Ignores eval results output directory.
.github/aw/actions-lock.json	Updates pinned gh-aw setup action version used by compilation.
.github/agents/protocols/management-autopr-review-critic.protocol.md	Defines the reviewer→critic dispatch contract and verdict semantics.
.github/agents/management-autopr-reviewer.agent.md	Reviewer persona, calibration, gates, and critic-dispatch sequence.
.github/agents/management-autopr-review-critic.agent.md	Critic persona instructions for filtering false positives.

[Weidong Xu (weidongxu-microsoft)]

Real-PR dry runs were also completed from the private fork trial host. Safe outputs were staged, so neither Azure PR was modified.

• PR #50024, GA package: trial run produced a staged review with MGMT-RELEASE-PLAN and MGMT-BREAKING; the critic independently returned PASS for both.
• PR #49732, beta package: trial run reviewed the eligible non-generated Java changes and correctly returned noop, with no false MGMT-BREAKING finding.