Skip to content

docs(docs-audit): record that the periodic full-audit backstop is not running - #9428

Merged
os-steve merged 3 commits into
mainfrom
claude/issue-9231-all-audit-backstop
Aug 18, 2026
Merged

os-steve merged 3 commits into
mainfrom
claude/issue-9231-all-audit-backstop

Conversation

@os-steve

@os-steve os-steve commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

Fixes #9231

Answer: the backstop does not exist

This card was a verification first. The answer is no — there is no live schedule running the docs-accuracy --all full audit, and there never was one.

scripts/docs-audit/README.md §4 described a cron routine on a monthly / per-release cadence. Measured 2026-08-18, four independent ways, all negative:

check result
GitHub Actions workflows No workflow runs this audit at all. 31 workflows registered on the repo; the 10 carrying a schedule: trigger are all unrelated.
Routines (list_triggers) The only cron Routine on the agent-seat account is the hourly triage seat, 18 * * * *. No docs-audit Routine, enabled or disabled.
The named creation mechanism The schedule skill §4 pointed at is not in .claude/skills/.
Trace of any periodic run Exactly three docs-accuracy audit PRs in repo history (#3243, #4219, #4312), each hand-initiated against a named issue family, most recent 2026-07-31. No cadence.

Two notes on method, because the card was explicitly about not reading a conclusion off a field that cannot carry it:

  • The scheduled-workflow list was derived, not copied — and copying it was wrong. The first push of this PR said 11 and included cut-rc, inherited from an earlier triage comment on the card. On main, cut-rc.yml has no schedule: trigger; its only schedule: match is a comment saying it deliberately has none, which an unanchored grep counts. Ten workflows carry a real one. The conclusion is unchanged, but consuming a stated list as authoritative is precisely the failure this card measures, so the README now carries the derivation command rather than the list alone.
  • The workflow list was read from the Actions API, not from disk. The two disagree in both directions: matrix-aggregate-experiment.yml is registered in Actions with no file on main today, and a scheduled workflow auto-disabled after 60 days of inactivity leaves its file byte-identical. Neither view alone answers "is it running".
  • The Routines surface is readable by an agent. The earlier triage note on this card recorded that Routine state is "readable only from the maintainer's Routines UI — no agent-facing surface can confirm it". That is falsified: list_triggers enumerates every Routine the account owns, and a cron Routine is never hidden by the include_completed filter, so its absence is a real reading rather than a listing artifact.

Second finding: change-scoped is not a backstop

Independent of whether it ran. §4 scoped the routine to "the change-scoped doc list since the last audit", while the cost note in the same file calls part 4 the "periodic full backstop". Those are different runs and only the second is a backstop — a change-scoped list is derived by the very anchor heuristic whose misses the backstop exists to catch, so scoping it that way re-inherits the blind spot it is meant to cover.

So the leg was doubly unsound: absent, and mis-specified even as written. For the record, --all itself is sound as a coverage mechanism — it enumerates all 178 hand-written docs, verified against the audit workflow's own inline scope via check-audit-scope.mjs.

Third finding: the named fallback is not a backstop either

The card names a cheap fallback — point the audit's args wiring at packageMentionDocs. It was not implemented (see below), and it also would not work. Measured across the 8 most recent packages/**-touching commits on main, the coarse set is wider but not a superset:

commit precise docs packageMentionDocs in docs but NOT in packageMentionDocs
a4331227b 5 21 3
48032c927 9 31 3
19539b4b2 41 119 6
40d5b2d4c 45 120 6
42d899071 47 120 7

In every one of the 8, between 3 and 7 pages the precise scope correctly finds are absent from the coarse set — api/error-catalog.mdx, automation/approvals.mdx, api/plugin-endpoints.mdx, data-modeling/relationships.mdx among them. That is #9192's own finding restated: the two sets miss in different directions, because the coarse one is a dependency-graph proxy and pages documenting a change through the SDK surface never name the implementing package.

So the fallback trades one incomplete set for another and loses pages the current scope gets right. Only --all is a backstop. This is a fact the PM should have before revisiting the ruling, not a recommendation.

What this changes

Documentation only. §4 now states the measured fact, carries the four checks so the next person re-reads them instead of re-deriving them, and keeps the intended design for whoever stands the routine up. The cost note no longer names a backstop that is not running.

⛔ Deliberately not done, both being the maintainer's / PM's call:

scripts/docs-audit/affected-docs.mjs was not touched — it is held by #9294.

Verification

Gate union re-run on the final commit 673885abd, working tree clean:

node scripts/docs-audit/check-affected-docs.mjs
  ✓ affected-docs self-test: 197 cases pass.
node scripts/docs-audit/check-audit-scope.mjs
  ✓ docs-accuracy-audit scope is in sync with content/docs/: 178 hand-written doc(s).
  ✓ release-owned pages are in scope and read-only: 9 page(s) under content/docs/releases/
node scripts/check-nul-bytes.mjs
  check-nul-bytes: OK (scanned 6122 text file(s) ... no raw ASCII control bytes)

Gate family derived from the changed path with node scripts/pm/dispatch-gates.mjs scripts/docs-audit/README.md, which named check-affected-docs.mjs via the scripts/docs-audit/** trigger in docs-drift-check.yml.

No changeset: this is an internal tooling README, publishing nothing.


Generated by Claude Code


Generated by Claude Code

… running

Part 4 of the docs-audit README described a cron routine running the
docs-accuracy audit on a monthly/per-release cadence. Measured 2026-08-18:
no such schedule exists. No GitHub Actions workflow runs the audit (the 11
scheduled workflows are all unrelated), no Routine on the agent-seat account
runs it, the `schedule` skill the section named as the creation mechanism is
gone, and repo history holds only three hand-initiated audit PRs with no
cadence between them.

This matters beyond the README: the #9192 option-A ruling (keep the docs-drift
scope precise) rests on this backstop as its coverage leg. Recording the
measurement rather than standing the routine up — a periodic LLM audit spends
budget on a cadence, which is the maintainer's call.

Also records a second defect the measurement surfaced: the old text scoped the
routine to "the change-scoped doc list since the last audit" while the cost
note called part 4 the "periodic full backstop". Only the latter is a backstop
— a change-scoped list is derived by the same anchor heuristic whose misses the
backstop exists to catch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XqDQYVU5smx29ts9pAErja
@claude claude Bot added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Aug 18, 2026
@github-actions github-actions Bot added size/s and removed skip-changeset PR has no user-facing published change; bypasses the changeset gate labels Aug 18, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

Nothing in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 0 changed package(s)), so this run has no opinion about the docs.

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Aug 18, 2026
@claude claude Bot added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Aug 18, 2026
claude added 2 commits August 18, 2026 01:03
…not copied

The first commit said 11 scheduled workflows and included `cut-rc`. That list
was copied from an earlier triage comment rather than derived. On origin/main
`cut-rc.yml` has no `schedule:` trigger — its only `schedule:` match is a
comment stating it deliberately has none, which an unanchored grep counts.

Ten workflows carry a real `schedule:` trigger. The conclusion is unchanged
(none of them runs the docs audit), but a list consumed as authoritative
without deriving it is the exact failure this card exists to measure, so the
section now carries the derivation command instead of the copied list.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XqDQYVU5smx29ts9pAErja
…o it is no backstop

The obvious cheap substitute for a missing full-audit backstop is to widen the
audit's scope back to the coarse packageMentionDocs set. Measured across the 8
most recent packages/**-touching commits on main, that does not work: the coarse
set is wider but not a superset — 3 to 7 pages present in the precise docs set
are absent from packageMentionDocs in every one of the 8.

That is the #9192 finding restated. The two sets miss in different directions,
because the coarse one is a dependency-graph proxy and pages documenting a
change through the SDK surface never name the implementing package. Swapping to
it trades one incomplete set for another and loses pages the precise scope gets
right. Only --all is a backstop.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XqDQYVU5smx29ts9pAErja
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/s skip-changeset PR has no user-facing published change; bypasses the changeset gate

Projects

None yet

2 participants