Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 66 additions & 8 deletions scripts/docs-audit/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -352,17 +352,75 @@ Always follow a run with the docs build gate:
pnpm --filter @objectstack/docs build # must compile all pages clean
```

## 4. Scheduled routine — periodic backstop

A cron routine (created via the `schedule` skill) runs on a cadence (default monthly /
per-release) to catch drift the CI gate missed. It computes the change-scoped doc list
since the last audit, runs the `docs-accuracy-audit` workflow on it, runs the build, and
opens a PR when there are fixes. See the routine prompt for the exact steps.
## 4. Scheduled routine — periodic backstop ⛔ NOT RUNNING

**Measured 2026-08-18: no live schedule runs this audit.** This section previously
described the backstop in the present tense; it does not exist, so **it cannot be cited
as coverage for what the part-1 scope leaves out.** Recorded rather than fixed on the
spot: standing up a periodic LLM audit spends real budget on a cadence, which is the
maintainer's call.

The intended design, for whoever stands it up: a cron routine on a cadence (default
monthly / per-release) catches drift the CI gate missed — it runs the
`docs-accuracy-audit` workflow, runs the build, and opens a PR when there are fixes.

**Two independent defects in the old text, both worth keeping in view:**

1. **It never ran.** Checked four ways, all negative — repeat these rather than
re-deriving them:

- **GitHub Actions** — no workflow runs this audit at all. Of the 31 workflows
registered on the repo, the 10 carrying a `schedule:` trigger (`codeql`,
`coverage-nightly`, `engine-split-metric`, `prerelease-pin-watch`, `publish-smoke`,
`rerun-safety-nightly`, `scaffold-e2e`, `showcase-smoke`, `stale`, `validate-deps`)
are all unrelated. Derive that list with `grep -rl '^\s*schedule:' .github/workflows/`
rather than copying it: an unanchored grep for `schedule:` also matches `cut-rc.yml`,
whose only hit is a comment saying it deliberately has no schedule.
⚠️ **Read the registered workflow list and its run history, not the YAML on disk.**
The two disagree in *both* directions: a workflow can be registered in Actions with
no file on `main` (`matrix-aggregate-experiment.yml` is, today), and a scheduled
workflow that GitHub auto-disabled after 60 days of repo inactivity leaves its file
byte-identical. "The file is there" cannot answer "is it running" — the same
read-a-conclusion-off-a-field-that-cannot-carry-it failure this docs-audit subsystem
keeps paying for.
- **Routines** — the Claude Code Remote `list_triggers` tool lists every Routine the
agent-seat account owns, and this is the surface that answers the question (it was
once written off as unreadable by any agent; it is not). The only cron Routine is the
hourly triage seat, `18 * * * *`. There is no docs-audit Routine, enabled *or*
disabled. A cron Routine is never hidden by the `include_completed` filter, so the
absence is real rather than a listing artifact.
- **The creation mechanism this section named is gone** — there is no `schedule` skill
in `.claude/skills/`.
- **No trace of a periodic run.** Repo history holds exactly three docs-accuracy audit
PRs (#3243, #4219, #4312), every one hand-initiated against a named issue family, the
most recent 2026-07-31. Nothing on a cadence.

2. **A change-scoped run is not a backstop.** The old text had the routine compute "the
change-scoped doc list since the last audit", while the cost note below calls part 4 the
"periodic **full** backstop". Those are two different runs and only the second is a
backstop — a change-scoped list is derived by the very anchor heuristic whose misses
the backstop exists to catch, so scoping it that way re-inherits the blind spot it is
meant to cover. A backstop has to run `--all`: all 178 hand-written docs (run
`check-audit-scope.mjs` for today's number rather than trusting this one).

3. **And the obvious cheap substitute is not a backstop either.** When the backstop is
missing, the tempting one-line fix is to widen the audit's scope back to the coarse
`packageMentionDocs` set part 1 still emits. Measured across the 8 most recent
`packages/**`-touching commits on `main`: it is wider (21 pages vs 5 on `a4331227b`)
but it is **not a superset** — in every one of the 8, between 3 and 7 pages present in
the precise `docs` set are absent from `packageMentionDocs`
(`api/error-catalog.mdx`, `automation/approvals.mdx`, `api/plugin-endpoints.mdx`,
`data-modeling/relationships.mdx` among them). That is #9192's finding restated: the
two sets miss in *different* directions, because the coarse one is a dependency-graph
proxy and pages documenting a change through the SDK surface never name the
implementing package. Swapping to it trades one incomplete set for another and loses
pages the precise scope gets right. Only `--all` is a backstop.

---

**Cost note:** a full audit is ~2 agents per doc — measured at ~2.8M output tokens /
~160 agents when the scope was 128 docs, and the hand-written set is 178 today (run
`check-audit-scope.mjs` for the current number; don't trust a count written down here).
Always prefer the change-scoped list (`affected-docs.mjs`) over `--all` except for the
periodic full backstop.
Always prefer the change-scoped list (`affected-docs.mjs`) over `--all`; reach for `--all`
only for a deliberate full audit. (This sentence used to name the periodic full backstop as
the exception — see part 4: that backstop is not running.)
Loading