Skip to content

feat(ci): PR size (non-test churn) pre-commit hook + size-labeling workflow - #445

Open
arekay-nv wants to merge 3 commits into
mlcommons:mainfrom
arekay-nv:pr-size-check
Open

feat(ci): PR size (non-test churn) pre-commit hook + size-labeling workflow#445
arekay-nv wants to merge 3 commits into
mlcommons:mainfrom
arekay-nv:pr-size-check

Conversation

@arekay-nv

@arekay-nv arekay-nv commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

Adds a PR-size classifier used two ways, implementing the "CI job for LOC count / label large PRs" item from the Endpoints PR Review Policy:

  • pre-commit hook pr-size-check — prints the one-line size class on every commit (verbose: true). Run python3 scripts/pr_size_check.py --verbose for the threshold grid and per-class review requirements.
  • CI workflow .github/workflows/pr-size-label.yml — labels each PR size/normal | size/large | size/very-large from the same logic.

Size = non-test churn (additions + deletions and changed-file count) excluding tests/**, lockfiles, vendored sources, and generated files. Class = the higher of the lines/files dimensions. Advisory only — nothing blocks.

Per its own tool, this PR classifies as Normal.

Type of change

  • New feature (tooling / CI)

Testing

  • Tests added — tests/unit/scripts/test_pr_size_check.py (21 cases): classify boundaries, numstat parsing, verbose gating, temp-git measurement, $GITHUB_OUTPUT, the CI two-ref --head path, top-level *.lock exclusion, and base-ref auto-detection (candidate scan + GITHUB_BASE_REF).
  • All new tests pass locally (21 passed)
  • Manual testing — verified per-commit hook output and the label logic against a temp repo

Checklist

  • Code follows project style (ruff / ruff-format / license header)
  • Pre-commit hooks pass — mypy skipped locally (known macOS sched_setaffinity env issue on unrelated files; green on Linux CI)
  • Documentation updated — base-branch resolution / forked-PR behavior (below)

Notes for reviewers

  • The labeling workflow uses pull_request_target so GITHUB_TOKEN can write labels on fork PRs (a plain pull_request token is read-only for forks). It never checks out or runs PR code — only the trusted base-repo copy of scripts/pr_size_check.py over the fetched PR diff.
  • Because pull_request_target runs the workflow from the base branch, this very PR won't be auto-labeled; labeling takes effect for PRs opened after this merges.
  • Label provisioning is a one-off manual step (labels are no longer auto-created). The per-PR workflow only needs pull-requests: write to attach labels, whereas creating label definitions needs issues: write; keeping them separate keeps the per-PR token minimally scoped. Create the labels once with:
    gh label create size/normal     -c 0E8A16 -d "PR Review Policy: <=500 non-test lines & <=20 files" --force
    gh label create size/large      -c FBCA04 -d "PR Review Policy: 501-1500 lines or 21-50 files"     --force
    gh label create size/very-large -c B60205 -d "PR Review Policy: >1500 lines or >50 files"          --force
    Applying is best-effort: a missing label or transient API error emits a ::warning:: and never fails the PR, and stale size/* labels are removed only after the new label is successfully applied (so a failed add never leaves a PR unlabeled).

Base-branch resolution & forked PRs

The size class is computed against a base branch. When --base / PR_BASE_REF are unset, resolve_base_ref() tries, in order: GITHUB_BASE_REForigin/mainmainupstream/main → any other <remote>/main.

  • CI (labeling workflow): always measures against the exact pull_request.base.sha, so the applied label is correct regardless of local remotes.
  • Local pre-commit in a forked clone: in the common fork layout — origin = your fork, upstream = mlcommons/endpointsorigin/main resolves before upstream/main. If your fork's origin/main is stale, the local advisory size can count already-merged upstream commits and over-classify. This affects only the local hint, never the CI label. To pin the base locally, either keep your fork's main synced with upstream, or set the base explicitly:
    PR_BASE_REF=upstream/main pre-commit run pr-size-check   # or, ad hoc:
    python3 scripts/pr_size_check.py --base upstream/main --verbose

…orkflow

Classify a PR by non-test churn per the Endpoints PR Review Policy: additions
+ deletions and changed-file count outside tests/**, lockfiles, vendored
sources, and generated files, mapped to Normal / Large / Very large (class =
higher of the two dimensions). Advisory only -- always exits 0.

- scripts/pr_size_check.py: pure classify()/parse_numstat() + git measurement.
  Terse one-line verdict by default; -v/--verbose adds the threshold grid and
  per-class review requirements; --github-output emits size_class/churn/files.
- pre-commit hook `pr-size-check`: quiet on a normal commit; the breakdown
  surfaces under `pre-commit run pr-size-check --verbose`.
- .github/workflows/pr-size-label.yml: labels PRs size/normal|large|very-large
  from the same logic. Uses pull_request_target so fork PRs can be labeled,
  running only the trusted base-repo script over the fetched PR diff (never
  checks out or executes PR code).
- tests/unit/scripts/test_pr_size_check.py: boundary, verbose-gating, and
  temp-git measurement coverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@codecov-commenter

codecov-commenter commented Aug 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@01b080e). Learn more about missing BASE report.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #445   +/-   ##
=======================================
  Coverage        ?   80.99%           
=======================================
  Files           ?      150           
  Lines           ?    20211           
  Branches        ?        0           
=======================================
  Hits            ?    16369           
  Misses          ?     3842           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

arekay-nv and others added 2 commits August 14, 2026 14:27
Follow-up hardening on the pr-size-check feature:

- workflow: drop the inline `gh label create` calls (creating label
  definitions needs `issues: write`; the job only has `pull-requests: write`).
  Label application is now warn-only and never fails the advisory check, and
  stale `size/*` labels are stripped only after the new label applies, so a
  failed add can't leave a PR unlabeled. Labels are provisioned once
  out-of-band via `gh label create`.
- pre-commit: run the advisory hook with `python3` (not `python`) and set
  `verbose: true` so the one-line verdict shows on every commit.
- size script: exclude root-level `*.lock` (was `**/*.lock`, which git's
  default pathspec matching never matches against a top-level `uv.lock`).
- tests: cover the CI two-ref `--head` path, top-level `*.lock` exclusion,
  and base-ref auto-detection (candidate scan + `GITHUB_BASE_REF`).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@arekay-nv
arekay-nv marked this pull request as ready for review August 14, 2026 23:30
@arekay-nv
arekay-nv requested a review from a team August 14, 2026 23:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants