fix(tooling): docs-drift stops flagging docs for tests-only changes - #4091
Merged
Conversation
`affected-docs.mjs` derived changed-package roots from every file under `packages/`, including tests. A test observes behaviour rather than defining it, so it cannot make an implementation-accuracy doc stale — yet every tests-only PR lit up its packages' whole doc set. Three in a row did it: #4064 and #4078 each flagged 6 `packages/services` docs for a diff containing no production code, and one before them the same way. This is the one place the mapper's deliberate over-inclusion actively hurt. The script's own header says over-inclusion is preferred to misses, and for real code that is right — the periodic full audit is the backstop. But a category that is ALWAYS false is different: a reader who learns the comment is noise stops reading it, and then it fails on the PR where it is correct. Precision matters here for the same reason a flaky test is worse than no test. - Test files are dropped before the package roots are derived: `*.test.*` / `*.spec.*` at any depth (so `.integration.test.ts` and `.conformance.test.ts` are covered) plus anything under `__tests__` / `__mocks__` / `__fixtures__`. - The narrowing is NOT silent. The excluded count rides the summary line and `testFilesSkipped` in `--json` — a tool that quietly narrows its own scope reads as "nothing to see here" when it means "I did not look". - `--self-test` pins the matcher, following `check-error-code-casing` / `check-route-envelope`, and the drift workflow runs it before computing the mapping so a regression fails loudly instead of emptying the comment. Verified against real history rather than by inspection: - `46e86bad` (#4078, tests only) → 0 docs, was 6, reporting `1 test file(s) excluded` - `4965bfac` (#4059, engine.ts + formula) → the same 11 docs as before, `1` skipped - `--all` unchanged at 178 docs The self-test earns its place too: swapping the anchored match for a naive `path.includes('test')` fails 7 cases, among them `control-flow.zod.ts` and `latest.ts` being swallowed as tests — the direction that would turn this optimisation into a miss. Tooling only; empty changeset since nothing releases. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QoA8AV99Ss1RRkLLAYmDzq
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
CI on the first push of this PR showed the gap: `Flag docs affected by code changes` never ran, because the workflow triggers on `paths: ['packages/**']` and this PR touches only `scripts/` and `.github/`. So the self-test step added one commit ago was added but never exercised, and editing `affected-docs.mjs` was the one change its own guard could structurally never see — the tool is unguarded exactly when it is being modified. Adds the mapper's own paths to the trigger. A PR touching only these gets the benign "No hand-written docs reference the 0 changed package(s). ✅" branch the comment step already has, which is the correct answer, and the self-test now runs on the change most likely to break it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QoA8AV99Ss1RRkLLAYmDzq
Contributor
📓 Docs Drift CheckNo hand-written docs reference the 0 changed package(s). ✅ |
os-zhuang
marked this pull request as ready for review
July 30, 2026 09:41
os-zhuang
added a commit
that referenced
this pull request
Jul 30, 2026
…xclusion (#4136) #4091 dropped test files before deriving the changed-package roots, because a test cannot make an implementation-accuracy doc stale. `packages/*/CHANGELOG.md` is the obvious next candidate by the same argument — release notes define behaviour no more than a test does — and extending the exclusion there is a provable no-op. Measured before writing anything down: - `chore: version packages` is the only PR class that mass-touches those files, and it runs no GitHub Actions at all. `changesets/action` opens it with the repo's GITHUB_TOKEN, and GitHub does not create workflow runs from GITHUB_TOKEN-authored events. #3910 carries exactly one check run, from Vercel's own app — `ci.yml` matches that PR unconditionally and still never fired. So the drift gate never sees a release PR to be noisy on. - Even if it did, `changeset version` writes `package.json` beside every `CHANGELOG.md` it appends to (45 against 46 on page 1 of #3910's diff), so dropping the CHANGELOGs would leave the derived package-root set identical. The bump is not unverified, just verified later: `ci.yml` and `lint.yml` both run on `push: main`, and `release.yml` gates publish on a green build — worth stating so the absent PR checks are not read as a hole. Recorded in the README beside the exclusion it neighbours, so the next reader does not spend a PR rediscovering it. Claude-Session: https://claude.ai/code/session_01QoA8AV99Ss1RRkLLAYmDzq Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tooling only, no package changes. Found while working the #4045 line — three of its PRs tripped it in a row.
The false positive
affected-docs.mjsderives changed-package roots from every file underpackages/, tests included. So a PR that changes only a test file flags every hand-written doc that names those packages:packages/servicesdocspackages/servicesdocsA test observes behaviour rather than defining it, so it cannot make an implementation-accuracy doc stale. Every one of those was a guaranteed-false finding.
Why precision matters here specifically
The script's own header says over-inclusion is preferred to misses, and for real code changes that is right — the periodic full audit is the backstop. But a category that is always false is a different thing: a reader who learns the comment is noise stops reading it, and then it fails on the PR where it is correct. Same reason a flaky test is worse than no test.
The change
*.test.*/*.spec.*at any depth (covering.integration.test.tsand.conformance.test.ts), plus anything under__tests__/__mocks__/__fixtures__.testFilesSkippedin--json. A tool that quietly narrows its own scope reads as "nothing to see here" when it means "I did not look".--self-test, following thecheck-error-code-casing/check-route-envelopeconvention, run by the workflow before it computes the mapping — so a regression fails loudly instead of quietly emptying the comment.Verified against real history, not by inspection
Ran the patched mapper over the actual commits:
46e86bad(#4078)1 test file(s) excluded4965bfac(#4059)engine.ts+formula1skipped--allThe second row is the one that matters — it proves implementation detection is intact, not just that the noise is gone.
The self-test earns its place
Replacing the anchored match with a naive
path.includes('test')— the plausible wrong way to write this — fails 7 cases, including the direction that would turn the optimisation into a miss:latest.tscontainstestas a substring, which is exactly the sneaky case an unanchored check swallows.Correction, and a second gap CI surfaced
An earlier revision of this description said this PR's own drift comment "should now report 0 docs, which is itself a live demonstration of the fix". That was wrong. CI on the first push showed no drift comment at all: the workflow triggers on
paths: ['packages/**'], and this PR touches onlyscripts/and.github/, soFlag docs affected by code changesnever ran.Which surfaced a real gap worth fixing rather than working around — the self-test step had been added but was structurally unreachable by the change most likely to break it. Editing
affected-docs.mjswas the one change its own guard could never see.The second commit adds
scripts/docs-audit/**and this workflow file to the trigger. A PR touching only those now runs the self-test and gets the comment step's existingNo hand-written docs reference the 0 changed package(s). ✅branch, which is the correct answer. This PR exercises it.Not done here
A package's
CHANGELOG.mdis arguably the same class — changeset releases touch everypackages/*/CHANGELOG.mdand would light up everything — but those PRs are bot-generated and nobody reads the comment on them, so I left the scope to the observed problem rather than widening it unasked.There is no test harness for root
scripts/(pnpm testisturbo run test, workspace packages only), which is why this uses the repo's--self-testconvention rather than a vitest file.🤖 Generated with Claude Code
https://claude.ai/code/session_01QoA8AV99Ss1RRkLLAYmDzq