From b6a2452f2d16b7e998bbe3f1acc0d64e407bf771 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 18 Aug 2026 00:59:00 +0000 Subject: [PATCH 1/3] docs(docs-audit): record that the periodic full-audit backstop is not running MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Part 4 of the docs-audit README described a cron routine running the docs-accuracy audit on a monthly/per-release cadence. Measured 2026-08-18: no such schedule exists. No GitHub Actions workflow runs the audit (the 11 scheduled workflows are all unrelated), no Routine on the agent-seat account runs it, the `schedule` skill the section named as the creation mechanism is gone, and repo history holds only three hand-initiated audit PRs with no cadence between them. This matters beyond the README: the #9192 option-A ruling (keep the docs-drift scope precise) rests on this backstop as its coverage leg. Recording the measurement rather than standing the routine up — a periodic LLM audit spends budget on a cadence, which is the maintainer's call. Also records a second defect the measurement surfaced: the old text scoped the routine to "the change-scoped doc list since the last audit" while the cost note called part 4 the "periodic full backstop". Only the latter is a backstop — a change-scoped list is derived by the same anchor heuristic whose misses the backstop exists to catch. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01XqDQYVU5smx29ts9pAErja --- scripts/docs-audit/README.md | 59 +++++++++++++++++++++++++++++++----- 1 file changed, 51 insertions(+), 8 deletions(-) diff --git a/scripts/docs-audit/README.md b/scripts/docs-audit/README.md index 0b1ea22c748..ac3dfc31de8 100644 --- a/scripts/docs-audit/README.md +++ b/scripts/docs-audit/README.md @@ -352,17 +352,60 @@ Always follow a run with the docs build gate: pnpm --filter @objectstack/docs build # must compile all pages clean ``` -## 4. Scheduled routine — periodic backstop - -A cron routine (created via the `schedule` skill) runs on a cadence (default monthly / -per-release) to catch drift the CI gate missed. It computes the change-scoped doc list -since the last audit, runs the `docs-accuracy-audit` workflow on it, runs the build, and -opens a PR when there are fixes. See the routine prompt for the exact steps. +## 4. Scheduled routine — periodic backstop ⛔ NOT RUNNING + +**Measured 2026-08-18: no live schedule runs this audit.** This section previously +described the backstop in the present tense; it does not exist, so **it cannot be cited +as coverage for what the part-1 scope leaves out.** Recorded rather than fixed on the +spot: standing up a periodic LLM audit spends real budget on a cadence, which is the +maintainer's call. + +The intended design, for whoever stands it up: a cron routine on a cadence (default +monthly / per-release) catches drift the CI gate missed — it runs the +`docs-accuracy-audit` workflow, runs the build, and opens a PR when there are fixes. + +**Two independent defects in the old text, both worth keeping in view:** + +1. **It never ran.** Checked four ways, all negative — repeat these rather than + re-deriving them: + + - **GitHub Actions** — no workflow runs this audit at all. Of the 31 workflows + registered on the repo, the 11 carrying a `schedule:` trigger (`codeql`, + `coverage-nightly`, `cut-rc`, `engine-split-metric`, `prerelease-pin-watch`, + `publish-smoke`, `rerun-safety-nightly`, `scaffold-e2e`, `showcase-smoke`, `stale`, + `validate-deps`) are all unrelated. + ⚠️ **Read the registered workflow list and its run history, not the YAML on disk.** + The two disagree in *both* directions: a workflow can be registered in Actions with + no file on `main` (`matrix-aggregate-experiment.yml` is, today), and a scheduled + workflow that GitHub auto-disabled after 60 days of repo inactivity leaves its file + byte-identical. "The file is there" cannot answer "is it running" — the same + read-a-conclusion-off-a-field-that-cannot-carry-it failure this docs-audit subsystem + keeps paying for. + - **Routines** — the Claude Code Remote `list_triggers` tool lists every Routine the + agent-seat account owns, and this is the surface that answers the question (it was + once written off as unreadable by any agent; it is not). The only cron Routine is the + hourly triage seat, `18 * * * *`. There is no docs-audit Routine, enabled *or* + disabled. A cron Routine is never hidden by the `include_completed` filter, so the + absence is real rather than a listing artifact. + - **The creation mechanism this section named is gone** — there is no `schedule` skill + in `.claude/skills/`. + - **No trace of a periodic run.** Repo history holds exactly three docs-accuracy audit + PRs (#3243, #4219, #4312), every one hand-initiated against a named issue family, the + most recent 2026-07-31. Nothing on a cadence. + +2. **A change-scoped run is not a backstop.** The old text had the routine compute "the + change-scoped doc list since the last audit", while the cost note below calls part 4 the + "periodic **full** backstop". Those are two different runs and only the second is a + backstop — a change-scoped list is derived by the very anchor heuristic whose misses + the backstop exists to catch, so scoping it that way re-inherits the blind spot it is + meant to cover. A backstop has to run `--all`: all 178 hand-written docs (run + `check-audit-scope.mjs` for today's number rather than trusting this one). --- **Cost note:** a full audit is ~2 agents per doc — measured at ~2.8M output tokens / ~160 agents when the scope was 128 docs, and the hand-written set is 178 today (run `check-audit-scope.mjs` for the current number; don't trust a count written down here). -Always prefer the change-scoped list (`affected-docs.mjs`) over `--all` except for the -periodic full backstop. +Always prefer the change-scoped list (`affected-docs.mjs`) over `--all`; reach for `--all` +only for a deliberate full audit. (This sentence used to name the periodic full backstop as +the exception — see part 4: that backstop is not running.) From 2f48c0a9a6578dbe6b5b0e23fc2165e7b68f27bf Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 18 Aug 2026 01:03:05 +0000 Subject: [PATCH 2/3] docs(docs-audit): correct the scheduled-workflow list to 10, derived not copied MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first commit said 11 scheduled workflows and included `cut-rc`. That list was copied from an earlier triage comment rather than derived. On origin/main `cut-rc.yml` has no `schedule:` trigger — its only `schedule:` match is a comment stating it deliberately has none, which an unanchored grep counts. Ten workflows carry a real `schedule:` trigger. The conclusion is unchanged (none of them runs the docs audit), but a list consumed as authoritative without deriving it is the exact failure this card exists to measure, so the section now carries the derivation command instead of the copied list. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01XqDQYVU5smx29ts9pAErja --- scripts/docs-audit/README.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/scripts/docs-audit/README.md b/scripts/docs-audit/README.md index ac3dfc31de8..1bf3b8fb5df 100644 --- a/scripts/docs-audit/README.md +++ b/scripts/docs-audit/README.md @@ -370,10 +370,12 @@ monthly / per-release) catches drift the CI gate missed — it runs the re-deriving them: - **GitHub Actions** — no workflow runs this audit at all. Of the 31 workflows - registered on the repo, the 11 carrying a `schedule:` trigger (`codeql`, - `coverage-nightly`, `cut-rc`, `engine-split-metric`, `prerelease-pin-watch`, - `publish-smoke`, `rerun-safety-nightly`, `scaffold-e2e`, `showcase-smoke`, `stale`, - `validate-deps`) are all unrelated. + registered on the repo, the 10 carrying a `schedule:` trigger (`codeql`, + `coverage-nightly`, `engine-split-metric`, `prerelease-pin-watch`, `publish-smoke`, + `rerun-safety-nightly`, `scaffold-e2e`, `showcase-smoke`, `stale`, `validate-deps`) + are all unrelated. Derive that list with `grep -rl '^\s*schedule:' .github/workflows/` + rather than copying it: an unanchored grep for `schedule:` also matches `cut-rc.yml`, + whose only hit is a comment saying it deliberately has no schedule. ⚠️ **Read the registered workflow list and its run history, not the YAML on disk.** The two disagree in *both* directions: a workflow can be registered in Actions with no file on `main` (`matrix-aggregate-experiment.yml` is, today), and a scheduled From 673885abd0c2d1196868f38a2f92b0a245b870dd Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 18 Aug 2026 01:04:31 +0000 Subject: [PATCH 3/3] docs(docs-audit): record that packageMentionDocs is not a superset, so it is no backstop MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The obvious cheap substitute for a missing full-audit backstop is to widen the audit's scope back to the coarse packageMentionDocs set. Measured across the 8 most recent packages/**-touching commits on main, that does not work: the coarse set is wider but not a superset — 3 to 7 pages present in the precise docs set are absent from packageMentionDocs in every one of the 8. That is the #9192 finding restated. The two sets miss in different directions, because the coarse one is a dependency-graph proxy and pages documenting a change through the SDK surface never name the implementing package. Swapping to it trades one incomplete set for another and loses pages the precise scope gets right. Only --all is a backstop. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01XqDQYVU5smx29ts9pAErja --- scripts/docs-audit/README.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/scripts/docs-audit/README.md b/scripts/docs-audit/README.md index 1bf3b8fb5df..7d830f8c17c 100644 --- a/scripts/docs-audit/README.md +++ b/scripts/docs-audit/README.md @@ -403,6 +403,19 @@ monthly / per-release) catches drift the CI gate missed — it runs the meant to cover. A backstop has to run `--all`: all 178 hand-written docs (run `check-audit-scope.mjs` for today's number rather than trusting this one). +3. **And the obvious cheap substitute is not a backstop either.** When the backstop is + missing, the tempting one-line fix is to widen the audit's scope back to the coarse + `packageMentionDocs` set part 1 still emits. Measured across the 8 most recent + `packages/**`-touching commits on `main`: it is wider (21 pages vs 5 on `a4331227b`) + but it is **not a superset** — in every one of the 8, between 3 and 7 pages present in + the precise `docs` set are absent from `packageMentionDocs` + (`api/error-catalog.mdx`, `automation/approvals.mdx`, `api/plugin-endpoints.mdx`, + `data-modeling/relationships.mdx` among them). That is #9192's finding restated: the + two sets miss in *different* directions, because the coarse one is a dependency-graph + proxy and pages documenting a change through the SDK surface never name the + implementing package. Swapping to it trades one incomplete set for another and loses + pages the precise scope gets right. Only `--all` is a backstop. + --- **Cost note:** a full audit is ~2 agents per doc — measured at ~2.8M output tokens /