Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .changeset/strict-object-registered-types.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
---
'@objectstack/spec': minor
---

`strictObject` makes closing an authoring shape one call; `seed` and `doc` are the first two registered metadata types converted with it; and a new invariant test found two live protection-envelope bugs on `hook` and `datasource`.

**The helper.** The #4001 wiring was four parts per schema plus a drift test: a hand-transcribed `const X_KEYS = [...]` array, a `strictUnknownKeyError({ knownKeys: X_KEYS, … })` call, the `{ error }` argument, `.strict()`, and an "accepts every declared key" probe to catch the array drifting from the shape it describes. That was 34 key arrays and 16 probe files with most of the authorable surface still ahead — and the array was never necessary: `knownKeys` feeds only the edit-distance suggestion, and the shape object is right there at the call site. `strictObject({ surface, history, aliases?, guidance? }, shape)` derives it, which also removes the per-schema drift probe: a key list read from the shape cannot disagree with it. `aliases` and `guidance` stay hand-written and stay **optional** — they carry judgement rather than transcription, and treating curation as a precondition is part of why the ratchet moved slowly.

**A sharper targeting rule.** The five-directory triage answers "is this authorable?" but not "is this parsed?" — and after #4410 that second question decides whether a flip enforces anything. `BUILTIN_METADATA_TYPE_SCHEMAS` answers both: every entry is author-written and parsed on three paths (`defineStack()`, `/api/v1/meta/types/:type`, the Studio form). Ten had no strictness at all; `seed` and `doc` are the first two converted. Five of the ten live in `system/`, which the directory triage never covered — the two lenses miss different things.

**Two live bugs, found by a check rather than by reading.** `MetadataPlugin`'s artifact loader stamps `_packageId` / `_provenance` on every registered type, so a strict schema that does not declare `MetadataProtectionFields` rejects its own loader's output — a hard 422 on the ADR-0094 overlay path. That defect had been found three times by hand (`permission`, `position`, then `seed`/`doc`). A new invariant test over the registered-type registry found it twice more on its first run: **`hook` and `datasource` had both gone strict in the #4001 data step without the envelope.** Both now declare it. The test asserts the hard case (rejects) unconditionally with no exemption list, and tracks the quieter case (silently strips, currently only `field`) separately.

**Ledger.** `strictObject` replaces the old wiring recipe as the standard, and the gate's site-counting method now counts `strictObject(` alongside `z.object(` — counting only the latter would have made every conversion look like surface disappearing, so "solved" and "deleted" would read the same. The gate caught that itself on the first conversion.

Authoring impact: on `seed` and `doc`, a key the schema never declared is now rejected instead of silently discarded — it was already being ignored, so no working behavior changes. The rejection names the surface, echoes the key and suggests the closest declared one (`rows` → `records`, `body` → `content`), with tombstones for `path` / `slug` on `doc`. The published JSON Schema is unchanged: `build-schemas.ts` converts with `io: 'output'`, which already emitted `additionalProperties: false` for these shapes. `validation` is the remaining registered type with a known envelope gap; it is a `z.lazy()` discriminated union whose variants `.extend()` a shared base, so it needs per-variant conversion rather than one call.
7 changes: 7 additions & 0 deletions content/docs/references/data/datasource.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,13 @@ const result = Datasource.parse(data);
| **schemaMode** | `Enum<'managed' \| 'external' \| 'validate-only'>` | ✅ | Schema ownership mode |
| **external** | `{ label?: string; allowedSchemas?: string[]; allowWrites: boolean; validation: object; … }` | optional | External datasource federation settings (schemaMode != "managed") |
| **origin** | `Enum<'code' \| 'runtime'>` | ✅ | Datasource provenance (server-managed, read-only) |
| **_lock** | `Enum<'none' \| 'no-overlay' \| 'no-delete' \| 'full'>` | optional | Item-level lock — controls overlay & delete (ADR-0010). |
| **_lockReason** | `string` | optional | Human-readable reason shown when a write is refused by _lock. |
| **_lockSource** | `Enum<'artifact' \| 'package' \| 'env-forced'>` | optional | Layer that set _lock (artifact \| package \| env-forced). |
| **_provenance** | `Enum<'package' \| 'org' \| 'env-forced'>` | optional | Origin of the item (package \| org \| env-forced). |
| **_packageId** | `string` | optional | Owning package machine id. |
| **_packageVersion** | `string` | optional | Owning package version. |
| **_lockDocsUrl** | `string` | optional | Optional documentation link surfaced next to _lockReason. |


---
Expand Down
7 changes: 7 additions & 0 deletions content/docs/references/data/seed.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,13 @@ const result = Seed.parse(data);
| **mode** | `Enum<'insert' \| 'update' \| 'upsert' \| 'replace' \| 'ignore'>` | ✅ | Conflict resolution strategy |
| **env** | `Enum<'prod' \| 'dev' \| 'test'>[]` | ✅ | Applicable environments |
| **records** | `Record<string, any>[]` | ✅ | Data records |
| **_lock** | `Enum<'none' \| 'no-overlay' \| 'no-delete' \| 'full'>` | optional | Item-level lock — controls overlay & delete (ADR-0010). |
| **_lockReason** | `string` | optional | Human-readable reason shown when a write is refused by _lock. |
| **_lockSource** | `Enum<'artifact' \| 'package' \| 'env-forced'>` | optional | Layer that set _lock (artifact \| package \| env-forced). |
| **_provenance** | `Enum<'package' \| 'org' \| 'env-forced'>` | optional | Origin of the item (package \| org \| env-forced). |
| **_packageId** | `string` | optional | Owning package machine id. |
| **_packageVersion** | `string` | optional | Owning package version. |
| **_lockDocsUrl** | `string` | optional | Optional documentation link surfaced next to _lockReason. |


---
Expand Down
7 changes: 7 additions & 0 deletions content/docs/references/system/doc.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,13 @@ const result = Doc.parse(data);
| **order** | `number` | optional | Sort key within a book group (ADR-0046 §6) |
| **group** | `string` | optional | Explicit book-group key (ADR-0046 §6); rules usually suffice |
| **translations** | `Record<string, { label?: string; description?: string; content: string }>` | optional | Per-locale `{label?,description?,content}` variants; the base doc is the fallback |
| **_lock** | `Enum<'none' \| 'no-overlay' \| 'no-delete' \| 'full'>` | optional | Item-level lock — controls overlay & delete (ADR-0010). |
| **_lockReason** | `string` | optional | Human-readable reason shown when a write is refused by _lock. |
| **_lockSource** | `Enum<'artifact' \| 'package' \| 'env-forced'>` | optional | Layer that set _lock (artifact \| package \| env-forced). |
| **_provenance** | `Enum<'package' \| 'org' \| 'env-forced'>` | optional | Origin of the item (package \| org \| env-forced). |
| **_packageId** | `string` | optional | Owning package machine id. |
| **_packageVersion** | `string` | optional | Owning package version. |
| **_lockDocsUrl** | `string` | optional | Optional documentation link surfaced next to _lockReason. |


---
Expand Down
83 changes: 73 additions & 10 deletions docs/audits/2026-07-unknown-key-strictness-ledger.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,11 @@ must explicitly `.strip()` back, because `.extend()` inherits `.strict()`.

## Standard wiring

`strictUnknownKeyError` in `shared/suggestions.zod.ts` (generalized from the
#3746 hand-rolled map) is the one factory every strict authoring schema wires:
`strictObject` in `shared/strict-object.ts` is the one call a strict authoring
schema needs:

```ts
z.object({ ... }, { error: strictUnknownKeyError({ surface, knownKeys, aliases, guidance, history }) }).strict()
lazySchema(() => strictObject({ surface, history, aliases?, guidance? }, { ...shape }))
```

- `aliases` — semantic near-misses edit distance cannot reach (`visibleWhen` →
Expand All @@ -40,8 +40,24 @@ z.object({ ... }, { error: strictUnknownKeyError({ surface, knownKeys, aliases,
rejection carries the upgrade — AGENTS.md Post-Task Checklist #3) and
wrong-layer pointers (`apiOperations` is response-side; `objectName` belongs
on the start node).
- Key lists live beside the schema and are **drift-guarded by tests** (an
"accepts every declared key" probe), because the schema body is lazy.
- **Both are optional.** A schema with neither still names the surface, echoes
the offending key, and suggests the closest declared one. Curation is an
upgrade, not a precondition — and treating it as a precondition is part of
why this ratchet moved as slowly as it did.

**No key list, and no drift probe.** Earlier steps hand-transcribed a
`const X_KEYS = [...] as const` beside each schema and pinned it with an
"accepts every declared key" test, because the schema body is lazy. That was 34
key arrays and 16 probe files with most of the surface still ahead — and it was
never necessary: `knownKeys` feeds only the edit-distance fallback, and the
shape object is right there at the call site. `strictObject` reads the keys from
`shape`, so the two copies become one and the probe has nothing left to catch.
`extraKeys` covers the one case the shape cannot see: a base `.extend()`ed
elsewhere.

`strictUnknownKeyError` stays exported for the schemas that cannot use the
helper — notably `z.lazy()` discriminated unions, whose variants each need
their own key set.

Every ratchet step ships only with the empirical zero-breakage pass: full
`@objectstack/spec` suite + `tsc`, downstream consumer suites, and
Expand Down Expand Up @@ -128,6 +144,34 @@ dropped at parse, and nothing failed.
`password` / `authSource` / `options` were **wired**, having been declared
and dropped on the floor. Enforcing a contract and honouring it are the same
task from two directions.
8. **Three more registered types could not represent their own ADR-0010
protection envelope** — `seed`, `doc` and `validation`, found by applying the
registered-type lens above. Exactly the gap that made `permission` return a
hard 422 on the ADR-0094 overlay path (entry 2) and that `position` carried
until step 2: `MetadataPlugin`'s artifact loader stamps `_packageId` /
`_provenance` on **every** registered type, and `getMetaItemLayered` →
`saveMetaItem` round-trips a body carrying them, so an undeclared envelope was
stripped on every parse. Declared on `seed` and `doc` as part of closing them;
`validation` still carries it (its union shape defers the conversion).

Worth noting how it kept recurring: this was the **fourth** occurrence of one
defect, found four times by four different routes, because nothing checked
the invariant directly. So it is checked now —
`kernel/metadata-type-schemas.test.ts` asserts it over the whole registry.

**It found two more on its first run.** `hook` and `datasource` had both gone
`.strict()` in the #4001 data step *without* declaring the envelope, so both
were in the worst class — rejecting their own loader's output, a live hard 422
on the ADR-0094 overlay path, sitting on `main`. Three prior hand-searches for
exactly this defect had walked past them. That is the argument for writing the
check in one line: **finding the same defect repeatedly by hand is evidence
the check is missing, not evidence the search worked.**

The check separates the two severities, because they are not the same bug:
*rejecting* the envelope is live breakage and is asserted unconditionally with
no exemption list; *stripping* it silently loses protection metadata on
round-trip and is tracked with a debt list (`field` only) — and each entry
there becomes a rejection the day its schema is closed.

This is the empirical argument for the ratchet: the inference "no metadata in
the repo carries unknown keys" was **false three times over**, and only the
Expand All @@ -151,7 +195,7 @@ block) when `position` joined the ratchet.

## File-level triage — the five authorable directories

Site counts are `z.object(` occurrences per file (2026-07-30, this branch).
Site counts are object sites — `z.object(` or `strictObject(` — per file (2026-07-30, this branch).
Classification is per the rule above; **(p)** marks a provisional call made
from the file's exports/JSDoc rather than a full read — verify before
tightening (the #4001 "sharing-rule lesson": candidates, not verdicts).
Expand Down Expand Up @@ -277,6 +321,20 @@ tightening (the #4001 "sharing-rule lesson": candidates, not verdicts).
downstream risk is the lowest on the board); it is simply unstarted. If the
step-1 question comes back "nothing is reporting", start here instead.

Done in the registered-types batch: `strictObject` (`shared/strict-object.ts`)
replaced the four-part wiring recipe, and `seed` + `doc` became the first two
conversions built on it — chosen by the registered-type lens above rather than
by directory, so both are provably parsed as well as provably authored. Both
also had to declare the ADR-0010 envelope, and the invariant test written in the
same pass found `hook` and `datasource` rejecting it outright on `main`
(findings log, entry 8). The ledger's site-counting method grew `strictObject(`
in the same change, because the gate failed on the first conversion when it did
not.

Deferred from that batch: `validation` — a `z.lazy()` discriminated union whose
variants `.extend()` a shared base, so each variant needs its own key set rather
than one `strictObject` call. It still carries the envelope gap.

Done in step 2: `security/rls.zod.ts` + `security/sharing.zod.ts` strict;
`PositionSchema` strict with the protection envelope declared (closing the
known sibling gap below).
Expand Down Expand Up @@ -355,10 +413,15 @@ the app step's `ACCOUNT_APP.defaultOpen` came from exactly this class of check.
Liveness Check workflow) holds the two claims here that are mechanically
checkable, so this map cannot go stale in silence again:

- **Site counts.** The method is stated above — `z.object(` occurrences per file
— so every number in the triage tables is verifiable. A count that no longer
matches means schemas were added or removed under a `Class` verdict nobody
re-examined. Touching a file forces you back through this ledger.
- **Site counts.** The method is stated above — `z.object(` or `strictObject(`
occurrences per file — so every number in the triage tables is verifiable. A
count that no longer matches means schemas were added or removed under a
`Class` verdict nobody re-examined. Touching a file forces you back through
this ledger. `strictObject(` had to join the count the moment the helper
existed: counting only `z.object(` would have made every conversion look like
surface *disappearing*, so "this directory got solved" and "this directory got
deleted" would produce the same number. The gate caught that itself on the
first conversion.
- **Coverage.** Every `*.zod.ts` in a triaged directory that HAS sites must have
a row. A new one is undeclared surface. The walk is **recursive**; nested files
are declared by their path relative to the section directory
Expand Down
23 changes: 22 additions & 1 deletion packages/spec/authorable-surface.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"description": "Ratchet of every AUTHORABLE key in the spec \u2014 what a metadata author may write, which for this platform IS the third-party API. Auto-updated on additions (commit the change). A key that disappears without a tombstone fails gen:schema, because these schemas are not .strict() and Zod would silently strip it. \"[RETIRED]\" marks a tombstoned key that still rejects with an upgrade prescription. See #3855, ADR-0059 \u00a75.",
"description": "Ratchet of every AUTHORABLE key in the spec what a metadata author may write, which for this platform IS the third-party API. Auto-updated on additions (commit the change). A key that disappears without a tombstone fails gen:schema, because these schemas are not .strict() and Zod would silently strip it. \"[RETIRED]\" marks a tombstoned key that still rejects with an upgrade prescription. See #3855, ADR-0059 §5.",
"keys": [
"ai/AIModelConfig:maxTokens",
"ai/AIModelConfig:model",
Expand Down Expand Up @@ -3211,6 +3211,13 @@
"data/DataTypeMapping:number",
"data/DataTypeMapping:text",
"data/DataTypeMapping:uuid",
"data/Datasource:_lock",
"data/Datasource:_lockDocsUrl",
"data/Datasource:_lockReason",
"data/Datasource:_lockSource",
"data/Datasource:_packageId",
"data/Datasource:_packageVersion",
"data/Datasource:_provenance",
"data/Datasource:active",
"data/Datasource:autoConnect",
"data/Datasource:capabilities",
Expand Down Expand Up @@ -3794,6 +3801,13 @@
"data/ScriptValidation:severity",
"data/ScriptValidation:tags",
"data/ScriptValidation:type",
"data/Seed:_lock",
"data/Seed:_lockDocsUrl",
"data/Seed:_lockReason",
"data/Seed:_lockSource",
"data/Seed:_packageId",
"data/Seed:_packageVersion",
"data/Seed:_provenance",
"data/Seed:env",
"data/Seed:externalId",
"data/Seed:mode",
Expand Down Expand Up @@ -5891,6 +5905,13 @@
"system/DistributedCacheConfig:prefetch",
"system/DistributedCacheConfig:tiers",
"system/DistributedCacheConfig:warmup",
"system/Doc:_lock",
"system/Doc:_lockDocsUrl",
"system/Doc:_lockReason",
"system/Doc:_lockSource",
"system/Doc:_packageId",
"system/Doc:_packageVersion",
"system/Doc:_provenance",
"system/Doc:content",
"system/Doc:description",
"system/Doc:group",
Expand Down
18 changes: 16 additions & 2 deletions packages/spec/scripts/lib/strictness-ledger.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,23 @@
import fs from 'node:fs';
import path from 'node:path';

/** `z.object(` occurrences — the ledger's own stated counting method. */
/**
* Object sites — `z.object(` **or** `strictObject(` — the ledger's own stated
* counting method.
*
* `strictObject(` counts because it *is* an object site; it is what a converted
* schema looks like. Counting only `z.object(` would have made every conversion
* silently shrink the ledger's measured surface, so a directory being solved and
* a directory being deleted would read identically — and a genuinely new,
* un-triaged `strictObject` schema would never register as undeclared surface.
*
* The gate caught this itself on the first conversion (`data/seed.zod.ts`, 1 → 0),
* which is the behaviour to preserve: a change in how schemas are written must
* fail this check rather than quietly rebase what it measures.
*/
export function countSites(file: string): number {
return (fs.readFileSync(file, 'utf-8').match(/z\.object\(/g) ?? []).length;
const src = fs.readFileSync(file, 'utf-8');
return (src.match(/z\.object\(|(?<![A-Za-z0-9_])strictObject\(/g) ?? []).length;
}

/**
Expand Down
10 changes: 10 additions & 0 deletions packages/spec/src/data/datasource.zod.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import { z } from 'zod';
*/
import { lazySchema } from '../shared/lazy-schema';
import { strictUnknownKeyError } from '../shared/suggestions.zod';
import { MetadataProtectionFields } from '../kernel/metadata-protection.zod';
import { validateDriverConfig } from './driver/config-registry.zod';

/*
Expand Down Expand Up @@ -611,6 +612,15 @@ export const DatasourceSchema = lazySchema(() => z.object({
*/
origin: z.enum(['code', 'runtime']).default('code')
.describe('Datasource provenance (server-managed, read-only)'),

// ADR-0010 — runtime protection envelope (internal — set by the loader).
// MISSING until the registered-type invariant test was written: `datasource`
// closed strict in the #4001 data step without declaring it, so the
// `_packageId` / `_provenance` that `MetadataPlugin` stamps on every
// registered type were REJECTED here. Same live defect as `hook`, and the
// same one `permission` hit as a 422 on the ADR-0094 overlay path before
// Tier-A declared them (#4001 findings log, entries 2/8).
...MetadataProtectionFields,
}, { error: datasourceUnknownKeyError }).strict().superRefine((ds, ctx) => {
// The `config` gate (#4410). `config` is parsed against the contract for the
// declared driver and every issue is re-pathed under the slot it came from —
Expand Down
Loading
Loading