docs(docs-audit): record that the periodic full-audit backstop is not running - #9428
Merged
Merged
Conversation
… running Part 4 of the docs-audit README described a cron routine running the docs-accuracy audit on a monthly/per-release cadence. Measured 2026-08-18: no such schedule exists. No GitHub Actions workflow runs the audit (the 11 scheduled workflows are all unrelated), no Routine on the agent-seat account runs it, the `schedule` skill the section named as the creation mechanism is gone, and repo history holds only three hand-initiated audit PRs with no cadence between them. This matters beyond the README: the #9192 option-A ruling (keep the docs-drift scope precise) rests on this backstop as its coverage leg. Recording the measurement rather than standing the routine up — a periodic LLM audit spends budget on a cadence, which is the maintainer's call. Also records a second defect the measurement surfaced: the old text scoped the routine to "the change-scoped doc list since the last audit" while the cost note called part 4 the "periodic full backstop". Only the latter is a backstop — a change-scoped list is derived by the same anchor heuristic whose misses the backstop exists to catch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XqDQYVU5smx29ts9pAErja
Contributor
📓 Docs Drift CheckNothing in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 0 changed package(s)), so this run has no opinion about the docs. |
…not copied The first commit said 11 scheduled workflows and included `cut-rc`. That list was copied from an earlier triage comment rather than derived. On origin/main `cut-rc.yml` has no `schedule:` trigger — its only `schedule:` match is a comment stating it deliberately has none, which an unanchored grep counts. Ten workflows carry a real `schedule:` trigger. The conclusion is unchanged (none of them runs the docs audit), but a list consumed as authoritative without deriving it is the exact failure this card exists to measure, so the section now carries the derivation command instead of the copied list. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XqDQYVU5smx29ts9pAErja
…o it is no backstop The obvious cheap substitute for a missing full-audit backstop is to widen the audit's scope back to the coarse packageMentionDocs set. Measured across the 8 most recent packages/**-touching commits on main, that does not work: the coarse set is wider but not a superset — 3 to 7 pages present in the precise docs set are absent from packageMentionDocs in every one of the 8. That is the #9192 finding restated. The two sets miss in different directions, because the coarse one is a dependency-graph proxy and pages documenting a change through the SDK surface never name the implementing package. Swapping to it trades one incomplete set for another and loses pages the precise scope gets right. Only --all is a backstop. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XqDQYVU5smx29ts9pAErja
This was referenced Aug 18, 2026
os-steve
marked this pull request as ready for review
August 18, 2026 01:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #9231
Answer: the backstop does not exist
This card was a verification first. The answer is no — there is no live schedule running the docs-accuracy
--allfull audit, and there never was one.scripts/docs-audit/README.md§4 described a cron routine on a monthly / per-release cadence. Measured 2026-08-18, four independent ways, all negative:schedule:trigger are all unrelated.list_triggers)18 * * * *. No docs-audit Routine, enabled or disabled.scheduleskill §4 pointed at is not in.claude/skills/.Two notes on method, because the card was explicitly about not reading a conclusion off a field that cannot carry it:
cut-rc, inherited from an earlier triage comment on the card. Onmain,cut-rc.ymlhas noschedule:trigger; its onlyschedule:match is a comment saying it deliberately has none, which an unanchored grep counts. Ten workflows carry a real one. The conclusion is unchanged, but consuming a stated list as authoritative is precisely the failure this card measures, so the README now carries the derivation command rather than the list alone.matrix-aggregate-experiment.ymlis registered in Actions with no file onmaintoday, and a scheduled workflow auto-disabled after 60 days of inactivity leaves its file byte-identical. Neither view alone answers "is it running".list_triggersenumerates every Routine the account owns, and a cron Routine is never hidden by theinclude_completedfilter, so its absence is a real reading rather than a listing artifact.Second finding: change-scoped is not a backstop
Independent of whether it ran. §4 scoped the routine to "the change-scoped doc list since the last audit", while the cost note in the same file calls part 4 the "periodic full backstop". Those are different runs and only the second is a backstop — a change-scoped list is derived by the very anchor heuristic whose misses the backstop exists to catch, so scoping it that way re-inherits the blind spot it is meant to cover.
So the leg was doubly unsound: absent, and mis-specified even as written. For the record,
--allitself is sound as a coverage mechanism — it enumerates all 178 hand-written docs, verified against the audit workflow's own inline scope viacheck-audit-scope.mjs.Third finding: the named fallback is not a backstop either
The card names a cheap fallback — point the audit's args wiring at
packageMentionDocs. It was not implemented (see below), and it also would not work. Measured across the 8 most recentpackages/**-touching commits onmain, the coarse set is wider but not a superset:docspackageMentionDocsdocsbut NOT inpackageMentionDocsa4331227b48032c92719539b4b240d5b2d4c42d899071In every one of the 8, between 3 and 7 pages the precise scope correctly finds are absent from the coarse set —
api/error-catalog.mdx,automation/approvals.mdx,api/plugin-endpoints.mdx,data-modeling/relationships.mdxamong them. That is #9192's own finding restated: the two sets miss in different directions, because the coarse one is a dependency-graph proxy and pages documenting a change through the SDK surface never name the implementing package.So the fallback trades one incomplete set for another and loses pages the current scope gets right. Only
--allis a backstop. This is a fact the PM should have before revisiting the ruling, not a recommendation.What this changes
Documentation only. §4 now states the measured fact, carries the four checks so the next person re-reads them instead of re-deriving them, and keeps the intended design for whoever stands the routine up. The cost note no longer names a backstop that is not running.
⛔ Deliberately not done, both being the maintainer's / PM's call:
packageMentionDocs) is not implemented. Choosing it means revisiting the The docs-drift advisory lists pages by package dependency, so it is wrong in BOTH directions — measured: 2 of 3 listed pages irrelevant, and the 2 pages that actually document the changed surface were not listed #9192 option-A ruling, whose second leg this PR shows is absent.scripts/docs-audit/affected-docs.mjswas not touched — it is held by #9294.Verification
Gate union re-run on the final commit
673885abd, working tree clean:Gate family derived from the changed path with
node scripts/pm/dispatch-gates.mjs scripts/docs-audit/README.md, which namedcheck-affected-docs.mjsvia thescripts/docs-audit/**trigger indocs-drift-check.yml.No changeset: this is an internal tooling README, publishing nothing.
Generated by Claude Code
Generated by Claude Code