Skip to content

docs: implementation-accuracy audit of the #4212-family affected docs - #4312

Merged
os-zhuang merged 5 commits into
mainfrom
claude/docs-accuracy-4212-followup
Jul 31, 2026
Merged

docs: implementation-accuracy audit of the #4212-family affected docs#4312
os-zhuang merged 5 commits into
mainfrom
claude/docs-accuracy-4212-followup

Conversation

@os-zhuang

@os-zhuang os-zhuang commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

What

Ran the repo's docs-accuracy-audit workflow scoped to the drift-check advisory from #4242 (its 14 docs) plus 4 more that a grep for the retired surfaces implicated. 158 evidence-backed fixes across 18 docs, each cited to file:line, then 29 adversarial-verifier repairs on top.

The findings are mostly the "copy this and it breaks" class:

Doc Fixes Examples
kernel/cluster.mdx 22 A documented eventBus.emit(...) that exists nowhere in packages/; a ctx.cluster PluginContext property that does not exist; defineStack({ cluster }) that stack.zod.ts silently strips; postgres/nats listed as shipped drivers when neither package exists; REDIS_URL for the real OS_CLUSTER_DRIVER; an at-least-once delivery promise no shipped driver provides
protocol/kernel/plugin-spec.mdx 20 plugin.manifest.ts as the manifest filename — grep-empty across packages/, apps/, examples/; the real one is objectstack.plugin.json. Also context.plugins.isInstalled() (no such member), compound semver ranges ('>=1.0.0 <2.0.0') that make the resolver throw, and init ordering described as registration order when bootstrap() runs a dependency topological sort
concepts/metadata-lifecycle.mdx 17
protocol/kernel/index.mdx 15
kernel/services-checklist.mdx 13
data-modeling/formulas.mdx 11
releases/v9.mdx 10
protocol/objectql/state-machine.mdx 8
plugins/index.mdx 7 os dev defaults to a project-anchored SQLite file, not an auto-registered memory driver
kernel/services.mdx, releases/implementation-status.mdx 6 each
deployment/vercel.mdx 5
protocol/objectql/query-syntax.mdx, permissions/authentication.mdx 4 each
kernel/events.mdx, data-modeling/external-datasources.mdx 3 each credentialsRef: 'secret:…' — the shipped binder only resolves sys_secret:<id>, so the sample as written fails closed at connect
deployment/migration-from-objectql.mdx, plugins/packages.mdx 2 each A ^3.1.0 pin that can never resolve to the current 16.1.0 line

Adversarial verification — all 18 docs

36/36 agents completed, 0 errors. Every doc carries a verifier pass that re-checked each applied fix against live code and repaired over-corrections. That phase earned its keep — the audit agents were confident and sometimes wrong:

  • enforceNetworkAccess invented for the real enforceNetworkRequest (plugin-permission-enforcer.ts:212);
  • a retirement callout widened to "None of it existed" over a list containing onEnable, which would have contradicted the live app-bundle contract AppPlugin.start() dispatches (app-plugin.ts:574);
  • cluster.lock.withLock(key, fn) attributed to service-job's cron scheduler, which actually does lock.acquire(...) + manual release (cron-job-adapter.ts:139-147);
  • a "(planned)" marker dropped from the webhook dispatcher, filing an already-shipping partitioned dispatcher under "Not started".

91 residual items the verifiers flagged as suspected-but-unproven are recorded in the run journal rather than guessed at in prose.

Verification

  • pnpm --filter @objectstack/docs build — exit 0, all pages compile.
  • node scripts/check-doc-authoring.mjs — 215 files clean.
  • node scripts/check-role-word.mjs — OK. The audit removed a role occurrence from kernel/events.mdx and one from services-checklist.mdx (4 → 3); since that check is a ratchet, an improvement fails the gate until recorded, so the baseline is ratcheted down in its own commit.
  • Prose/example changes only; no code paths touched.

Follow-ups the audit surfaced but did not fix

  • packages/core/README.md:64,94 still documents ctx.eventBus.on/.emit as part of PluginContext. No such runtime object exists — a code-side stale doc, not fixable here.
  • kernel/cluster.mdx's leader-elected strategy resolves only to packages/spec/src/kernel/cluster.zod.ts and its test — no runtime consumer of clusterScope / leaderStrategy. Another declared-but-unbuilt surface wanting its own ADR-0049 enforce-or-remove pass.
  • ClusterDriverSchema is a closed enum, so a driver registered under an arbitrary name via registerClusterDriver('etcd', …) can never be selected — driver: 'custom' resolves only a factory registered under the literal key 'custom'.
  • examples/app-showcase/src/system/datasources/showcase-external.datasource.ts still carries the broken credentialsRef: 'secret:warehouse/password' form in a commented-out demo — the likely origin of the doc error.

@github-actions github-actions Bot added documentation Improvements or additions to documentation tooling size/xs labels Jul 31, 2026
@os-zhuang
os-zhuang force-pushed the claude/docs-accuracy-4212-followup branch from 04ec8ce to 998bece Compare July 31, 2026 05:39
@vercel

vercel Bot commented Jul 31, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
objectstack Ignored Ignored Jul 31, 2026 6:19am

Request Review

@github-actions github-actions Bot added size/xl and removed size/xs labels Jul 31, 2026
claude added 3 commits July 31, 2026 05:43
WIP checkpoint — the adversarial verification pass is still running and will
add repairs on top of this commit. Recorded now so the work survives.

An audit agent read each doc, located the backing implementation under
packages/, and applied evidence-backed fixes in place (158 across 18 docs).
Every fix carries file:line evidence in the run's fix log. The heaviest:

- kernel/cluster.mdx (22): documented an `eventBus.emit(...)` API that exists
  nowhere in the codebase, a `ctx.cluster` PluginContext property that does
  not exist, `defineStack({ cluster })` that stack.zod.ts silently strips,
  fabricated `postgres`/`nats` drivers, the wrong env var (real one is
  OS_CLUSTER_DRIVER), and an at-least-once delivery promise neither shipped
  driver provides.
- protocol/kernel/plugin-spec.mdx (20): `plugin.manifest.ts` as the manifest
  filename — grep-empty across the implementation, the real one is
  objectstack.plugin.json; `context.plugins.isInstalled()` which does not
  exist; compound semver ranges ('>=1.0.0 <2.0.0') that make the resolver's
  anchored parse throw; init described as registration order when bootstrap()
  runs the dependency-topological order.
- data-modeling/external-datasources.mdx (3): credentialsRef samples used a
  `secret:` prefix the shipped SecretBinder cannot dereference (`sys_secret:`),
  so copying the sample fails closed at connect.
- deployment/migration-from-objectql.mdx (2): a `^3.1.0` pin that can never
  resolve to the current 16.1.0 line.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HrRNgrWaRtggzmrHpbomyh
Ran the repo's docs-accuracy-audit workflow scoped to the drift-check advisory
from #4242 (14 docs) plus 4 the retired-surface grep implicated. 158
evidence-backed fixes across 18 docs, each cited to file:line. The largest
classes:

- kernel/cluster.mdx (22): a documented `eventBus.emit(...)` that exists
  nowhere in packages/, a `ctx.cluster` PluginContext property that does not
  exist, `defineStack({ cluster })` that stack.zod.ts silently strips,
  `postgres`/`nats` listed as shipped drivers when neither package exists,
  `REDIS_URL` for the real `OS_CLUSTER_DRIVER`, and an at-least-once delivery
  promise no shipped driver provides.
- protocol/kernel/plugin-spec.mdx (20): `plugin.manifest.ts` as the manifest
  filename — grep-empty across packages/, apps/ and examples/; the real one is
  `objectstack.plugin.json`. Plus `context.plugins.isInstalled()` (no such
  member), compound semver ranges (`'>=1.0.0 <2.0.0'`) that make the resolver
  throw, and init ordering described as registration order when bootstrap()
  runs a dependency topological sort.
- data-modeling/external-datasources.mdx: `credentialsRef: 'secret:…'` — the
  shipped binder only resolves `sys_secret:<id>`, so the sample as written
  fails closed at connect.
- deployment/migration-from-objectql.mdx: a `^3.1.0` pin that can never
  resolve to the current 16.1.0 line.

VERIFICATION STATUS — 7 of 18 docs carry an adversarial-verifier pass; the
other 11 do not (the run hit the session limit mid-phase). The verifier earned
its keep on the 7 it reached: 8 repairs, including a fabricated
`enforceNetworkAccess` (real: `enforceNetworkRequest`,
plugin-permission-enforcer.ts:212) and an over-broad retirement claim that
would have contradicted the live app-bundle `onEnable`. The remaining 11 are
committed unverified and MUST NOT ship until their verifier pass runs — the PR
stays draft until then.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HrRNgrWaRtggzmrHpbomyh
The audit pass removed a "role" occurrence from kernel/events.mdx (now clean)
and one from kernel/services-checklist.mdx (4 → 3). check-role-word is a
ratchet, so an improvement fails the gate until the baseline records it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HrRNgrWaRtggzmrHpbomyh
@os-zhuang
os-zhuang force-pushed the claude/docs-accuracy-4212-followup branch from b9d2ea5 to 2383192 Compare July 31, 2026 05:43
claude added 2 commits July 31, 2026 05:51
…l/kernel/index)

Verifier pass for two more of the 18 audited docs completed: 27 of the audit's
fixes confirmed against code, 5 repaired. Checkpoint commit — the remaining
verifier passes are still running and will land separately.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HrRNgrWaRtggzmrHpbomyh
Completes the verifier phase for all 18 audited docs (36/36 agents, 0 errors).
Across the run: 158 audit fixes, 29 verifier repairs, 91 residual items
recorded rather than guessed at.

The repairs are why this phase exists — the audit agents were confident and
sometimes wrong. Representative catches:

- `enforceNetworkAccess` invented for the real `enforceNetworkRequest`
  (plugin-permission-enforcer.ts:212).
- A retirement callout widened to "None of it existed" over a list containing
  `onEnable`, which would have contradicted the live app-bundle contract
  AppPlugin.start() dispatches (app-plugin.ts:574).
- `cluster.lock.withLock(key, fn)` attributed to service-job's cron scheduler,
  which actually does `lock.acquire(...)` + manual release
  (cron-job-adapter.ts:139-147).
- A "(planned)" marker dropped from the webhook dispatcher, which filed an
  already-shipping partitioned dispatcher under "Not started".

Gates: check-doc-authoring 215 files clean; check-role-word OK (44 baselined);
docs build compiles all pages.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HrRNgrWaRtggzmrHpbomyh
@os-zhuang
os-zhuang marked this pull request as ready for review July 31, 2026 06:24
@os-zhuang
os-zhuang merged commit 9f38cdc into main Jul 31, 2026
16 checks passed
@os-zhuang
os-zhuang deleted the claude/docs-accuracy-4212-followup branch July 31, 2026 06:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/xl tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants