Python POC for GitHub-based API Reviews by tjprescott · Pull Request #47203 · Azure/azure-sdk-for-python

tjprescott · 2026-05-28T16:55:26Z

Summary

Adds tooling for GitHub-native API reviews for the Python SDK, enabling API surface tracking and review via GitHub PRs and CI. Based on this proposal: Proposal: API Reviews via GitHub (azure-sdk-tools#15789).

What's Included

1. CI Consistency Check (`.github/workflows/api-consistency.yml`)

A GitHub Actions workflow that enforces API.md consistency on every PR that touches sdk/**:

find_affected.js — diffs the PR against the base branch to find changed packages
regenerate.js — runs azpysdk apistub --md --extract-metadata for each affected package
find_mismatches.js — compares the regenerated API.md to the committed version and validates select metadata fields
Fails with actionable instructions if drift is detected

2. API Review PR Creator (`create_api_review_pr.js`)

A CLI tool to create dedicated API review PRs:

node scripts/api_md_workflow/create_api_review_pr.js \
  --package-name azure-foo-bar \
  --base azure-foo-bar_1.0.0 \
  --target owner:branch

Generates API.md for both the base tag and target branch
Pushes two branches and opens a draft PR where the diff is the API diff
Reviewers use GitHub's native review tools (comments, suggestions, approvals)

3. Language Adapter Pattern (`scripts/api_md_workflow/`)

Core orchestration is language-agnostic. A per-repo adapter (adapters/python.js) implements the language-specific logic (isPackageDir, findPackageDir, readVersion, generateApiForPackage). This allows the same framework to be ported to other Azure SDK language repos.

4. Shared Utility Layer (`.github/shared/`)

Common ESM modules for subprocess execution, logging, path manipulation, and GitHub API interactions — shared across workflow scripts.

Examples

Auto Reviews

Tag vs main: [API Review] azure-keyvault-keys 4.12.0b3 (base 4.11.1) #47357
Tag vs tag: [API Review] azure-ai-projects 2.2.0 (base 2.1.0) #47370
Tag vs main: [API Review] azure-storage-blob 12.31.0b1 (base 12.29.0) #47373

PR Review

[API Review] azure-cosmos 4.16.0b3 (base 4.14.0) #47375

Copilot

Pull request overview

Adds a new repo script (scripts/generate_api_text.py) to generate a package-level API.md by building a wheel, running apiview-stub-generator (apistub) to produce the token JSON, and converting that JSON to markdown via Export-APIViewMarkdown.ps1. This is repo tooling intended to streamline API review artifact generation.

Changes:

Introduces scripts/generate_api_text.py to locate a package under sdk/*/, build a wheel, run apistub, and emit API.md into the package directory.
Adds logic to install eng/apiview_reqs.txt and (optionally) upgrade apiview-stub-generator from the Azure SDK package index.
Uses eng/common/scripts/Export-APIViewMarkdown.ps1 to convert the generated *_python.json token file into markdown.

Copilot

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 9 comments.

Comments suppressed due to low confidence (1)

eng/tools/azure-sdk-tools/azpysdk/apistub.py:158

--dest-dir now writes outputs directly into the destination directory for every targeted package. When multiple packages are targeted in a single azpysdk apistub invocation, later packages can overwrite earlier ones (e.g., generic api.md / API.metadata.yml), causing incorrect/stale results.

            dest_dir = getattr(args, "dest_dir", None)
            if dest_dir:
                out_token_path = os.path.abspath(dest_dir)
                os.makedirs(out_token_path, exist_ok=True)
            else:
                out_token_path = os.path.abspath(staging_directory)

…le versions

Copilot

Pull request overview

Copilot reviewed 30 out of 30 changed files in this pull request and generated 6 comments.

Comments suppressed due to low confidence (1)

scripts/api_md_workflow/README.md:112

Same issue later in the README: the printed regeneration command uses --dest-dir . and omits --install-deps, which won’t reproduce the CI behavior and will write files to the wrong directory when run from the repo root. This should point --dest-dir at the package directory and include --install-deps.

4. `regenerate.js` rebuilds `api.md` for those packages.
5. `find_mismatches.js` records any `api.md` drift, including missing or untracked `api.md` files.
6. If drift is found, the workflow fails and prints the affected packages plus the `azpysdk apistub --md --extract-metadata <package-name> --dest-dir .` command to regenerate each `api.md` file locally from the repository root.

Copilot

Pull request overview

Copilot reviewed 30 out of 30 changed files in this pull request and generated 4 comments.

+    const packageDir = apiFile.replace(/\/(api\.md|api\.metadata\.yml)$/, "");
+    const packageName = path.basename(packageDir);
+    lines.push(`- ${packageDir}`);
+    lines.push(`  API file: ${apiFile}`);
+    lines.push(`  Regenerate: azpysdk apistub --md --extract-metadata ${packageName} --dest-dir .`);
+  }


+- Detects affected package directories from the PR diff.
+- Regenerates `api.md` for those packages.
+- Fails if the generated files differ from the committed files.
+- Fails if an affected package does not have a committed `api.md`.
+- Prints the mismatched or missing packages and the `azpysdk apistub --md --extract-metadata <package-name> --dest-dir .` command needed to regenerate each `api.md` file from the repository root.
+


+1. A PR changes files under `sdk/**`.
+2. `consistency.yml` runs.
+3. `find_affected.js` determines which packages were touched.
+4. `regenerate.js` rebuilds `api.md` for those packages.
+5. `find_mismatches.js` records any `api.md` drift, including missing or untracked `api.md` files.
+6. If drift is found, the workflow fails and prints the affected packages plus the `azpysdk apistub --md --extract-metadata <package-name> --dest-dir .` command to regenerate each `api.md` file locally from the repository root.


+function isPackageDir(repoRoot, packageDirRelative) {
+  const candidate = path.join(repoRoot, packageDirRelative);
+  if (!fs.existsSync(candidate) || !fs.statSync(candidate).isDirectory()) {
+    return false;
+  }
+
+  return fs.existsSync(path.join(candidate, "pyproject.toml")) || fs.existsSync(path.join(candidate, "setup.py"));
+}


mikeharder · 2026-06-06T03:35:46Z

we will need to either add a package-lock.json file, or add the necessary deps to another package.json already in the repo

mikeharder · 2026-06-06T03:39:49Z

+const path = require("path");
+const { spawnSync } = require("child_process");
+
+function runNode(scriptRelativePath, workspace, core) {


use a helper from exec.js, or we can add another helper if needed. we have a helper to run an arbitary file, or a helper to run "npm", but not a helper to run "node" itself (though we could add one).

Add generate_api_text.py script.

fd2ab3d

tjprescott requested a review from kashifkhan May 28, 2026 16:55

tjprescott requested a review from scbedd as a code owner May 28, 2026 16:55

Copilot AI review requested due to automatic review settings May 28, 2026 16:55

tjprescott requested a review from danieljurek as a code owner May 28, 2026 16:55

Copilot started reviewing on behalf of tjprescott May 28, 2026 16:55 View session

Copilot AI reviewed May 28, 2026

View reviewed changes

tjprescott force-pushed the GenerateAPITextScript branch 3 times, most recently from 7c45b15 to b9dc383 Compare May 29, 2026 19:55

Add PR creation script.

ebb3870

tjprescott force-pushed the GenerateAPITextScript branch from b9dc383 to ebb3870 Compare May 29, 2026 20:09

tjprescott mentioned this pull request May 29, 2026

[API Review] azure-ai-projects 2.1.0 (base 2.1.0) #47239

Closed

Review feedback.

2d24fa9

tjprescott closed this May 29, 2026

tjprescott reopened this May 29, 2026

tjprescott mentioned this pull request Jun 1, 2026

Proposal: API Reviews via GitHub Azure/azure-sdk-tools#15789

Draft

tjprescott changed the title ~~Add generate_api_text.py script~~ Python POC for GitHub-based API Reviews Jun 1, 2026

Add scripts for syncing api.md

e4b2876

tjprescott force-pushed the GenerateAPITextScript branch from f320bdb to e4b2876 Compare June 2, 2026 17:59

tjprescott requested a review from a team as a code owner June 2, 2026 17:59

tjprescott added 3 commits June 2, 2026 13:13

Refactor create_api_review_pr from Python to JS in a module way.

dd1b565

Remove update process and just have a consistency check.

688e16f

Add minor change to azure-template to test pipeline.

323d4f3

tjprescott requested review from benbp and weshaggard as code owners June 2, 2026 22:02

tjprescott added 2 commits June 2, 2026 15:11

Add API.md to test mismatch.

93184af

Apply actual api.md

7680ff6

mikeharder reviewed Jun 2, 2026

View reviewed changes

Comment thread .github/workflows/consistency.yml Outdated

mikeharder reviewed Jun 2, 2026

View reviewed changes

Comment thread .github/workflows/consistency.yml Outdated

tjprescott requested a review from Copilot June 3, 2026 21:13

Copilot started reviewing on behalf of tjprescott June 3, 2026 21:13 View session

Copilot AI reviewed Jun 3, 2026

View reviewed changes

tjprescott force-pushed the GenerateAPITextScript branch from 580208a to 4b6f18e Compare June 3, 2026 21:27

CI fixes.

9d8137c

tjprescott force-pushed the GenerateAPITextScript branch from 4b6f18e to 9d8137c Compare June 3, 2026 21:50

tjprescott added 8 commits June 3, 2026 14:55

Add skill for generating review PR.

760fc63

Update review generation logic.

1517d58

fix(python adapter): add shell:true on Windows for spawnSync

258a36a

fix(python adapter): pass --dest-dir so API.md lands in package dir

89e6c98

fix(python adapter): skip _generated dirs in readVersion to avoid sta…

62f81e3

…le versions

Make script idempotent and reuse branches with the same API hash.

92e0c92

Code review feedback.

bb253fa

Updates.

317bd4c

tjprescott force-pushed the GenerateAPITextScript branch 3 times, most recently from 6d81d65 to 818a5ee Compare June 5, 2026 21:08

tjprescott requested a review from Copilot June 5, 2026 22:27

Copilot started reviewing on behalf of tjprescott June 5, 2026 22:27 View session

tjprescott force-pushed the GenerateAPITextScript branch from 818a5ee to 778edda Compare June 5, 2026 22:32

Copilot AI reviewed Jun 5, 2026

View reviewed changes

Make review PR creation only work for updates, not new packages.

a82a7ee

tjprescott force-pushed the GenerateAPITextScript branch from 778edda to a82a7ee Compare June 5, 2026 22:37

CI fixes.

8d07492

tjprescott force-pushed the GenerateAPITextScript branch from 804d5e8 to 8d07492 Compare June 5, 2026 22:50

tjprescott requested a review from Copilot June 5, 2026 22:54

Copilot started reviewing on behalf of tjprescott June 5, 2026 22:54 View session

Copilot AI reviewed Jun 5, 2026

View reviewed changes

mikeharder reviewed Jun 6, 2026

View reviewed changes

Conversation

tjprescott commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's Included

1. CI Consistency Check (.github/workflows/api-consistency.yml)

2. API Review PR Creator (create_api_review_pr.js)

3. Language Adapter Pattern (scripts/api_md_workflow/)

4. Shared Utility Layer (.github/shared/)

Examples

Auto Reviews

PR Review

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

mikeharder Jun 6, 2026

Choose a reason for hiding this comment

Uh oh!

mikeharder Jun 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tjprescott commented May 28, 2026 •

edited

Loading

1. CI Consistency Check (`.github/workflows/api-consistency.yml`)

2. API Review PR Creator (`create_api_review_pr.js`)

3. Language Adapter Pattern (`scripts/api_md_workflow/`)

4. Shared Utility Layer (`.github/shared/`)