From 78322334b3e4beb9ad2154913c994757b83be90f Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 30 Jul 2026 16:56:39 +0000 Subject: [PATCH 1/3] =?UTF-8?q?feat(migrate):=20`os=20migrate=20value-shap?= =?UTF-8?q?es`=20=E2=80=94=20the=20per-deployment=20gate=20for=20reference?= =?UTF-8?q?=20and=20structured-JSON=20value=20shapes=20(#3438)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ADR-0104 D1's second evidence gate. Media value shapes already enforce once a deployment verifies its file migration (#3681); the reference and structured-JSON classes now get a gate of their own rather than borrowing that one — the file flag attests that file values were migrated and reconciled and says nothing about whether a `lookup` id or a `location` payload is well formed. - spec: `VALUE_SHAPES_MIGRATION_ID` beside its file sibling. - objectql: `scanValueShapes` walks every stored value of the covered classes against the write path's own predicate — `valueShapeViolation` / `isScannableValueShapeField` are exported from record-validator and imported by the scan, so the two cannot drift into disagreeing about "malformed". A test pins that property directly. - objectql: `validateRecord` gains `valueShapeStrict` beside `mediaValueShapeStrict`, and the engine reads the second flag through the same memoized seam. The two flag reads are refactored into one id-parameterised helper so "every way of not knowing answers false" is written once. - cli: `os migrate value-shapes`, dry-run by default. No backfill — a malformed `location` is application data whose correct value only its author knows, so the run reports and prescribes. With nothing to convert, `--apply`'s only write is the flag row, which preserves #3617's invariant trivially. - New escape hatch `OS_ALLOW_LAX_VALUE_SHAPES`, same precedence as its media sibling: opt-out beats all-class opt-in beats the flag. The scanner deliberately does NOT record the flag: readers of a migration flag use the spec contract and only writers depend on platform-objects, so having the engine write one would invert the layering. The CLI composes the two. A truncated or partially-unreadable scan fails the gate even at zero violations — "none in the part we read" is not the claim the flag makes. Verified: objectql 1272, spec 7167, cli 621, service-storage 252 green; all ten spec artifact gates pass (api-surface regenerated for the new exports); ESLint clean on every changed file. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016zgA8CQMJeJbFEjnbib1Uv --- .changeset/adr-0104-value-shapes-scan-gate.md | 62 +++++ .../commands/migrate/files-to-references.ts | 32 +-- .../cli/src/commands/migrate/value-shapes.ts | 248 ++++++++++++++++++ .../cli/src/utils/data-migration-plugins.ts | 40 +++ packages/objectql/src/engine.ts | 128 ++++++--- packages/objectql/src/index.ts | 17 ++ .../src/validation/record-validator.ts | 114 +++++++- .../src/validation/scan-value-shapes.test.ts | 123 +++++++++ .../src/validation/scan-value-shapes.ts | 236 +++++++++++++++++ packages/spec/api-surface.json | 1 + packages/spec/src/system/migration.zod.ts | 22 ++ 11 files changed, 945 insertions(+), 78 deletions(-) create mode 100644 .changeset/adr-0104-value-shapes-scan-gate.md create mode 100644 packages/cli/src/commands/migrate/value-shapes.ts create mode 100644 packages/cli/src/utils/data-migration-plugins.ts create mode 100644 packages/objectql/src/validation/scan-value-shapes.test.ts create mode 100644 packages/objectql/src/validation/scan-value-shapes.ts diff --git a/.changeset/adr-0104-value-shapes-scan-gate.md b/.changeset/adr-0104-value-shapes-scan-gate.md new file mode 100644 index 0000000000..992ccbb735 --- /dev/null +++ b/.changeset/adr-0104-value-shapes-scan-gate.md @@ -0,0 +1,62 @@ +--- +"@objectstack/spec": minor +"@objectstack/objectql": minor +"@objectstack/cli": minor +--- + +feat(migrate): `os migrate value-shapes` — the per-deployment gate for reference and structured-JSON value shapes (#3438) + +The second of ADR-0104 D1's two evidence gates. Media value shapes already +enforce once a deployment has verified its file migration (#3681); the +reference (`lookup` / `master_detail` / `user` / `tree`) and structured-JSON +(`location` / `address` / `composite` / `repeater` / `record` / `vector`) +classes now get a gate of their own. + +```bash +os migrate value-shapes # scan: reports, writes nothing +os migrate value-shapes --apply # scan + record the deployment flag when clean +``` + +The run walks every stored value of those classes against +`valueSchemaFor(field, 'stored')` — the same predicate the write path enforces, +imported rather than re-derived — and, at zero violations, records +`sys_migration { id: 'adr-0104-value-shapes', verified_at, blocking: 0 }`. +Strict enforcement of these classes reads **that row**, never the platform +version, so upgrading changes nothing until a deployment produces its own +evidence. + +**There is no backfill, deliberately.** The file migration converts legacy +values because the platform narrowed that storage form and owes the conversion. +A malformed `location` is application data whose correct value only its author +knows, so this run reports and prescribes — naming the object, field, type, +count, offending record ids and the parse issue — and the operator fixes and +re-runs. With nothing to convert, `--apply`'s only write is the flag row, which +keeps the #3617 invariant trivially: a dry run changes nothing, and whether a +run changed this deployment's posture never depends on what it found. + +**A separate flag from the file migration**, because it attests a separate +fact. That flag says file values were migrated and their ownership reconciled; +it says nothing about whether a `lookup` id or a `location` payload is well +formed. Gating these classes on it would be borrowing evidence for a fact it +does not cover. + +- New escape hatch **`OS_ALLOW_LAX_VALUE_SHAPES=1`** returns a verified + deployment to warnings, with the same precedence as its media sibling: the + opt-out beats `OS_DATA_VALUE_SHAPE_STRICT_ENABLED`, which beats the flag. + Wrongly staying lenient costs a warning; wrongly enforcing stops a working + app from writing. +- `@objectstack/spec/system` exports `VALUE_SHAPES_MIGRATION_ID`. +- `@objectstack/objectql` exports `scanValueShapes`, `valueShapeScanPassed` + and `formatValueShapeScanReport`. The scanner is read-only and does **not** + record the flag: readers of a migration flag use the spec contract, only + writers depend on `@objectstack/platform-objects`, so the composition lives + with the CLI command rather than inverting the engine's dependencies. +- `validateRecord` gains `valueShapeStrict`, the sibling of + `mediaValueShapeStrict`. Both default to `false`: a caller that cannot say + stays lenient, so nothing starts rejecting merely because the evidence was + unavailable. + +**Nothing changes for an existing deployment until it runs the command.** A +scan that is truncated, or that cannot read an object, fails the gate even with +zero violations found — "none in the part we read" is not the claim the flag +makes. diff --git a/packages/cli/src/commands/migrate/files-to-references.ts b/packages/cli/src/commands/migrate/files-to-references.ts index 09113c745c..0a712f412b 100644 --- a/packages/cli/src/commands/migrate/files-to-references.ts +++ b/packages/cli/src/commands/migrate/files-to-references.ts @@ -17,7 +17,7 @@ import { import { bootSchemaStack } from '../../utils/schema-migrate.js'; import { OCCUPANCY_HINT, probeMigrationTarget } from '../../utils/migrate-occupancy-gate.js'; import { describeOccupancy } from '../../utils/sqlite-occupancy.js'; -import { resolveStorageCapabilityArg } from '../serve.js'; +import { buildDataMigrationPlugins } from '../../utils/data-migration-plugins.js'; async function confirm(question: string): Promise { if (!process.stdin.isTTY) return false; // non-interactive → require --yes @@ -30,36 +30,6 @@ async function confirm(question: string): Promise { } } -/** - * The settings + storage service plugins, so the booted kernel carries - * `sys_file`/`sys_migration` and the deployment's REAL storage adapter. - * Settings first: the storage plugin re-resolves its adapter from persisted - * settings when a settings service is present, which is how an S3-configured - * deployment's backfill uploads land in S3 rather than on this machine. - * Storage config goes through the SAME resolver `os serve` uses - * (`resolveStorageCapabilityArg`), so the CLI materialises bytes exactly where - * the server would. It did not, and that mattered here more than anywhere: this - * command reconciles what records claim against what storage actually holds, so - * a root that disagrees with the server's reconciles against the wrong tree. - * It built `{ driver: 'local', root }` — keys `StorageServicePluginOptions` does - * not declare — so the adapter silently used the plugin's own `./storage` - * default while the server (since framework#4096) writes under - * `.objectstack/data/uploads`. - */ -async function buildDataMigrationPlugins(): Promise { - const plugins: unknown[] = []; - try { - const { SettingsServicePlugin } = await import('@objectstack/service-settings'); - plugins.push(new SettingsServicePlugin({ registerRoutes: false })); - } catch { - // optional — without it, constructor/env-driven storage config still applies - } - const { StorageServicePlugin } = await import('@objectstack/service-storage'); - const { options } = resolveStorageCapabilityArg(process.env.OS_STORAGE_ROOT); - plugins.push(new StorageServicePlugin({ ...options, registerRoutes: false })); - return plugins; -} - /** * `os migrate files-to-references` — the ADR-0104 D3 data migration, with its * self-check gate (#3617). diff --git a/packages/cli/src/commands/migrate/value-shapes.ts b/packages/cli/src/commands/migrate/value-shapes.ts new file mode 100644 index 0000000000..18c3d9af43 --- /dev/null +++ b/packages/cli/src/commands/migrate/value-shapes.ts @@ -0,0 +1,248 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import { Command, Flags } from '@oclif/core'; +import chalk from 'chalk'; +import { createInterface } from 'node:readline'; +import { + printHeader, + printSuccess, + printWarning, + printError, + printInfo, + printStep, + createTimer, + emitJson, +} from '../../utils/format.js'; +import { bootSchemaStack } from '../../utils/schema-migrate.js'; +import { buildDataMigrationPlugins } from '../../utils/data-migration-plugins.js'; + +async function confirm(question: string): Promise { + if (!process.stdin.isTTY) return false; // non-interactive → require --yes + const rl = createInterface({ input: process.stdin, output: process.stdout }); + try { + const answer: string = await new Promise((resolve) => rl.question(question, resolve)); + return /^y(es)?$/i.test(answer.trim()); + } finally { + rl.close(); + } +} + +/** + * `os migrate value-shapes` — the ADR-0104 D1 non-media value-shape gate + * (#3438), the sibling of `os migrate files-to-references` (#3617). + * + * Scans every stored reference (`lookup` / `master_detail` / `user` / `tree`) + * and structured-JSON (`location` / `address` / `composite` / `repeater` / + * `record` / `vector`) value against the contract the write path enforces, and + * — on an `--apply` run finding zero violations — records the deployment-level + * `adr-0104-value-shapes` flag. That flag, never the platform version, is what + * turns strict enforcement of those classes on for THIS deployment. + * + * ## No backfill, deliberately + * + * Unlike its sibling this run rewrites nothing. The file migration converts + * legacy values because the platform narrowed that storage form and therefore + * owes the conversion; a malformed `location` is application data whose correct + * value only its author knows. So this reports and prescribes, and the operator + * fixes and re-runs until it is green. + * + * The happy consequence: with nothing to convert, `--apply`'s only write is the + * flag row, so #3617's invariant is trivially preserved — a dry run changes + * nothing, and whether a run changed this deployment's posture never depends on + * what the run found. + * + * ## Why it is not gated on the file migration's flag + * + * That flag attests that file values were migrated and their ownership + * reconciled. It says nothing about whether a `lookup` id or a `location` + * payload is well formed. Reusing it here would be borrowing evidence for a + * fact it does not cover — see the ADR's 2026-07-27 amendment. + */ +export default class MigrateValueShapes extends Command { + static override description = + 'Scan stored reference and structured-JSON field values against the ADR-0104 value contract. ' + + 'Read-only; --apply records the deployment-level migration flag when the scan finds zero violations.'; + + static override examples = [ + '$ os migrate value-shapes', + '$ os migrate value-shapes --apply', + '$ os migrate value-shapes --apply --yes --json', + '$ os migrate value-shapes --object contact --object account', + ]; + + static override flags = { + 'database-url': Flags.string({ + description: 'Database URL to scan (defaults to $OS_DATABASE_URL / the project DB)', + env: 'OS_DATABASE_URL', + }), + apply: Flags.boolean({ + description: + 'Record the deployment migration flag when the scan passes (the scan itself is always read-only)', + default: false, + }), + yes: Flags.boolean({ char: 'y', description: 'Skip the --apply confirmation prompt', default: false }), + object: Flags.string({ + description: 'Restrict to this object (repeatable; default: every object with a covered field)', + multiple: true, + }), + 'max-records': Flags.integer({ + description: + 'Safety bound on records scanned per object — exceeding it truncates the scan and fails the gate', + }), + json: Flags.boolean({ description: 'Output as JSON (implies non-interactive; requires --yes to apply)' }), + }; + + async run(): Promise { + const { flags } = await this.parse(MigrateValueShapes); + const timer = createTimer(); + const apply = flags.apply; + + if (!flags.json) printHeader('Migrate · value-shapes'); + + // No occupancy gate, unlike the file migration: that command rewrites rows, + // so a second writer on the same SQLite file is a real hazard. This one only + // reads. A concurrent writer can still move the counts under us, which the + // scan reports honestly rather than pretending to have frozen the database. + + if (apply && !flags.yes) { + if (flags.json || !process.stdin.isTTY) { + if (flags.json) { + await emitJson({ error: 'confirmation_required', hint: 'pass --yes' }, 0, { compact: true }); + this.exit(1); + return; + } + printWarning( + 'Apply mode records this deployment\'s migration flag, which turns on strict value-shape ' + + 'enforcement. Re-run with --yes to confirm, or run without --apply to preview.', + ); + this.exit(1); + return; + } + const ok = await confirm( + chalk.bold('\nRecord the value-shape migration flag on this database if the scan passes? [y/N] '), + ); + if (!ok) { + printInfo('Aborted — nothing recorded.'); + return; + } + } + + if (!flags.json) { + printStep(apply ? 'Booting data stack (APPLY mode)…' : 'Booting data stack (scan only)…'); + } + + let stack; + try { + stack = await bootSchemaStack({ + databaseUrl: flags['database-url'], + extraPlugins: await buildDataMigrationPlugins(), + }); + } catch (error: any) { + if (flags.json) { await emitJson({ error: error.message }, 0, { compact: true }); this.exit(1); } + printError(error.message || String(error)); + this.exit(1); + return; + } + + try { + const engine: any = stack.kernel.getService('objectql'); + if (typeof engine?.getObject !== 'function') { + throw new Error('No ObjectQL engine on this stack — cannot scan.'); + } + // An empty scan is indistinguishable from a clean one, and this command's + // verdict is what authorises strict enforcement — so refuse to run when no + // app metadata is loaded (missing artifact / wrong directory) rather than + // "verify" a database the scan never actually looked at. + const loadedObjects: string[] = + typeof engine.getConfigs === 'function' ? Object.keys(engine.getConfigs()) : []; + if (!loadedObjects.some((name) => !name.startsWith('sys_'))) { + throw new Error( + 'No app objects are loaded, so the scan would examine nothing. ' + + 'Run "os build" in your project root first (the migration reads dist/objectstack.json), then re-run.', + ); + } + + const { scanValueShapes, valueShapeScanPassed, formatValueShapeScanReport } = + await import('@objectstack/objectql'); + + // In JSON mode keep stdout parseable — route scan warnings to stderr. + const logger = flags.json + ? { info: (m: string) => console.error(m), warn: (m: string) => console.error(m) } + : { info: (m: string) => printInfo(m), warn: (m: string) => printWarning(m) }; + + const report = await scanValueShapes(engine, logger, { + objects: flags.object, + maxRecordsPerObject: flags['max-records'], + }); + const passed = valueShapeScanPassed(report); + + // The flag write is the ONLY write this command makes, and only on an + // apply run that passed. A failing apply run still records — deliberately: + // it stamps `blocking` and clears `verified_at`, so a deployment whose data + // has regressed closes its own gate rather than coasting on an old pass. + let flag: unknown = null; + if (apply) { + const { recordDataMigrationRun } = await import('@objectstack/platform-objects/system'); + const { VALUE_SHAPES_MIGRATION_ID } = await import('@objectstack/spec/system'); + flag = await recordDataMigrationRun(engine, { + migrationId: VALUE_SHAPES_MIGRATION_ID, + passed, + blocking: report.blocking, + advisory: 0, + applied: true, + // Passed as an OBJECT — `recordDataMigrationRun` serialises it. + details: { + scannedObjects: report.scannedObjects.length, + scannedRecords: report.scannedRecords, + fields: report.findings.length, + truncated: report.truncated, + unreadableObjects: report.unreadableObjects, + }, + }); + if (typeof engine.invalidateDataMigrationFlags === 'function') { + engine.invalidateDataMigrationFlags(); + } + } + + if (flags.json) { + await emitJson({ + database: stack.dbLabel, + apply, + scan: report, + gatePassed: passed, + flag, + duration: timer.elapsed(), + }); + if (!passed) this.exit(1); + return; + } + + printInfo(`Database: ${chalk.white(stack.dbLabel)}`); + console.log(''); + console.log(formatValueShapeScanReport(report).join('\n')); + console.log(''); + + if (passed && apply) { + printSuccess( + `Scan clean — recorded the deployment flag. Reference and structured-JSON value shapes ` + + `are now enforced on this deployment (${timer.elapsed()}).`, + ); + } else if (passed) { + printSuccess(`Scan clean (${timer.elapsed()}). Re-run with --apply to record the flag and enforce.`); + } else if (apply) { + printError('Scan found violations — the flag records the failure and the gate stays closed.'); + printInfo('Fix the values named above and re-run; nothing is converted for you.'); + this.exit(1); + } else { + printError('Scan found violations — fix them, then re-run with --apply.'); + this.exit(1); + } + } catch (error: any) { + if (flags.json) { await emitJson({ error: error.message }, 0, { compact: true }); this.exit(1); } + printError(error.message || String(error)); + this.exit(1); + } finally { + await stack.shutdown(); + } + } +} diff --git a/packages/cli/src/utils/data-migration-plugins.ts b/packages/cli/src/utils/data-migration-plugins.ts new file mode 100644 index 0000000000..d7ca4fbb74 --- /dev/null +++ b/packages/cli/src/utils/data-migration-plugins.ts @@ -0,0 +1,40 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import { resolveStorageCapabilityArg } from '../commands/serve.js'; + +/** + * The service plugins a gated data migration boots with, so the kernel carries + * `sys_migration` (the deployment-level flag ledger) — and, for the file + * migration, `sys_file` plus the deployment's REAL storage adapter. + * + * Settings first: the storage plugin re-resolves its adapter from persisted + * settings when a settings service is present, which is how an S3-configured + * deployment's backfill uploads land in S3 rather than on this machine. + * Storage config goes through the SAME resolver `os serve` uses + * (`resolveStorageCapabilityArg`), so the CLI materialises bytes exactly where + * the server would. It did not, and that mattered more here than anywhere: the + * file migration reconciles what records claim against what storage actually + * holds, so a root that disagrees with the server's reconciles against the + * wrong tree. + * + * ⚠️ `sys_migration` is registered by the STORAGE plugin (its first consumer), + * which is why a migration with nothing to do with files still boots it. That + * coupling is not this module's to fix — the ledger is platform infrastructure + * and should not be owned by an optional service — but until it moves, every + * gated migration boots the same set so they all find the same ledger. Tracked + * separately; `os serve` auto-wires storage, so a served deployment always has + * it registered and the runtime gates can read their flags. + */ +export async function buildDataMigrationPlugins(): Promise { + const plugins: unknown[] = []; + try { + const { SettingsServicePlugin } = await import('@objectstack/service-settings'); + plugins.push(new SettingsServicePlugin({ registerRoutes: false })); + } catch { + // optional — without it, constructor/env-driven storage config still applies + } + const { StorageServicePlugin } = await import('@objectstack/service-storage'); + const { options } = resolveStorageCapabilityArg(process.env.OS_STORAGE_ROOT); + plugins.push(new StorageServicePlugin({ ...options, registerRoutes: false })); + return plugins; +} diff --git a/packages/objectql/src/engine.ts b/packages/objectql/src/engine.ts index ab7d117af5..73a42fbc30 100644 --- a/packages/objectql/src/engine.ts +++ b/packages/objectql/src/engine.ts @@ -12,10 +12,11 @@ import { type DroppedFieldsEvent } from '@objectstack/spec/data'; import type { WriteObservabilityOptions } from '@objectstack/spec/contracts'; -import { parseAutonumberFormat, renderAutonumber, missingFieldValues, isTenancyDisabled, FILE_REFERENCE_TYPES, isFileIdToken, RAW_FILE_VALUES_CONTEXT_KEY } from '@objectstack/spec/data'; +import { parseAutonumberFormat, renderAutonumber, missingFieldValues, isTenancyDisabled, FILE_REFERENCE_TYPES, REFERENCE_VALUE_TYPES, STRUCTURED_JSON_TYPES, isFileIdToken, RAW_FILE_VALUES_CONTEXT_KEY } from '@objectstack/spec/data'; import { DATA_MIGRATION_FLAG_OBJECT, FILE_REFERENCES_MIGRATION_ID, + VALUE_SHAPES_MIGRATION_ID, isDataMigrationFlagVerified, } from '@objectstack/spec/system'; import { ExecutionContext, ExecutionContextInput, ExecutionContextSchema } from '@objectstack/spec/kernel'; @@ -2197,6 +2198,30 @@ export class ObjectQL implements IDataEngine { return this.isFileReferencesMigrationVerified(); } + /** + * Does this object declare a reference or structured-JSON field? Same + * per-schema cache and same dormancy rule as {@link objectHasMediaField}: an + * object holding none of these types can hold no violation of them, so it + * must not pay a query to learn that. + */ + private objectHasCoveredValueField(objectSchema: any): boolean { + if (!objectSchema?.fields) return false; + const cached = ObjectQL.coveredValueFieldPresence.get(objectSchema); + if (cached !== undefined) return cached; + const present = Object.values(objectSchema.fields).some( + (def: any) => def && (REFERENCE_VALUE_TYPES.has(def.type) || STRUCTURED_JSON_TYPES.has(def.type)), + ); + ObjectQL.coveredValueFieldPresence.set(objectSchema, present); + return present; + } + private static readonly coveredValueFieldPresence = new WeakMap(); + + /** The reference / structured-JSON value-shape verdict for a write. */ + private async valueShapeStrictFor(objectSchema: any): Promise { + if (!this.objectHasCoveredValueField(objectSchema)) return false; + return this.isValueShapesMigrationVerified(); + } + /** * Has this deployment completed AND verified the ADR-0104 file-as-reference * migration (#3617)? Read once per process from the `sys_migration` flag. @@ -2219,40 +2244,71 @@ export class ObjectQL implements IDataEngine { */ async isFileReferencesMigrationVerified(): Promise { if (!this.fileReferencesMigrationVerified) { - this.fileReferencesMigrationVerified = (async () => { - if (!this._registry.getObject(DATA_MIGRATION_FLAG_OBJECT)) return false; - try { - const rows = await this.find(DATA_MIGRATION_FLAG_OBJECT, { - where: { id: FILE_REFERENCES_MIGRATION_ID }, - limit: 1, - context: { isSystem: true } as ExecutionContextInput, - }); - const row: any = rows?.[0]; - if (!row || row.id !== FILE_REFERENCES_MIGRATION_ID) return false; - const verified = isDataMigrationFlagVerified({ - id: FILE_REFERENCES_MIGRATION_ID, - last_run_at: String(row.last_run_at ?? ''), - verified_at: row.verified_at == null ? null : String(row.verified_at), - // A non-numeric count must read as "not zero", not as 0 — a bad - // coercion lands on NaN, which fails the === 0 test. - blocking: typeof row.blocking === 'number' ? row.blocking : Number(row.blocking ?? Number.NaN), - }); - if (verified) { - this.logger.info( - '[value-shape] this deployment has verified the file-as-reference migration — ' + - 'media value shapes are enforced and released field files may be collected ' + - '(ADR-0104 / #3617)', - ); - } - return verified; - } catch { - return false; // unreadable evidence → stay lenient - } - })(); + this.fileReferencesMigrationVerified = this.readMigrationFlagVerified( + FILE_REFERENCES_MIGRATION_ID, + '[value-shape] this deployment has verified the file-as-reference migration — ' + + 'media value shapes are enforced and released field files may be collected ' + + '(ADR-0104 / #3617)', + ); } return this.fileReferencesMigrationVerified; } + /** + * Has this deployment completed AND verified the ADR-0104 non-media + * value-shape scan (`os migrate value-shapes`, #3438)? Same memoized seam and + * same fail-lenient posture as the file flag above — and a SEPARATE flag, + * because it attests a different fact. The file migration says file values + * were converted and reconciled; it says nothing about whether a `lookup` id + * or a `location` payload is well formed, so it may not vouch for these + * classes. + */ + async isValueShapesMigrationVerified(): Promise { + if (!this.valueShapesMigrationVerified) { + this.valueShapesMigrationVerified = this.readMigrationFlagVerified( + VALUE_SHAPES_MIGRATION_ID, + '[value-shape] this deployment has verified the value-shape scan — reference and ' + + 'structured-JSON value shapes are enforced (ADR-0104 / #3438)', + ); + } + return this.valueShapesMigrationVerified; + } + + /** + * Read one deployment migration flag and answer whether it authorises its + * consumers. Shared by both flags so the "every way of not knowing answers + * false" rule is written once: no `sys_migration` object registered, no row, + * an unreadable table, a malformed row — all `false`. Enforcement derives + * from evidence, and absent evidence is not permission. + * + * Costs nothing on a kernel without the platform objects: the registry + * lookup short-circuits before any query. + */ + private async readMigrationFlagVerified(migrationId: string, verifiedLog: string): Promise { + if (!this._registry.getObject(DATA_MIGRATION_FLAG_OBJECT)) return false; + try { + const rows = await this.find(DATA_MIGRATION_FLAG_OBJECT, { + where: { id: migrationId }, + limit: 1, + context: { isSystem: true } as ExecutionContextInput, + }); + const row: any = rows?.[0]; + if (!row || row.id !== migrationId) return false; + const verified = isDataMigrationFlagVerified({ + id: migrationId, + last_run_at: String(row.last_run_at ?? ''), + verified_at: row.verified_at == null ? null : String(row.verified_at), + // A non-numeric count must read as "not zero", not as 0 — a bad + // coercion lands on NaN, which fails the === 0 test. + blocking: typeof row.blocking === 'number' ? row.blocking : Number(row.blocking ?? Number.NaN), + }); + if (verified) this.logger.info(verifiedLog); + return verified; + } catch { + return false; // unreadable evidence → stay lenient + } + } + /** * Drop the memoized deployment migration flags so the next write re-reads * them. For a host that runs a data migration in-process and wants its @@ -2260,6 +2316,7 @@ export class ObjectQL implements IDataEngine { */ invalidateDataMigrationFlags(): void { this.fileReferencesMigrationVerified = null; + this.valueShapesMigrationVerified = null; } async destroy() { @@ -2300,6 +2357,7 @@ export class ObjectQL implements IDataEngine { * {@link invalidateDataMigrationFlags} instead of waiting for a restart. */ private fileReferencesMigrationVerified: Promise | null = null; + private valueShapesMigrationVerified: Promise | null = null; /** Lazily-built index: child object name → roll-up summary descriptors on * parent objects that aggregate it. Invalidated when packages register. */ @@ -3071,6 +3129,7 @@ export class ObjectQL implements IDataEngine { // Resolved once for the whole batch; dormant unless the object declares // a media field, and memoized after the first object that does. const mediaValueShapeStrict = await this.mediaValueShapeStrictFor(schemaForValidation); + const valueShapeStrict = await this.valueShapeStrictFor(schemaForValidation); // Locale + translation hooks for the rejection messages (#3957) — // resolved once for the batch, identical for every row. const msgCtx = this.validationMessageContext(object, opCtx.context); @@ -3078,7 +3137,7 @@ export class ObjectQL implements IDataEngine { if (rowErrors[i] !== undefined) continue; try { normalizeMultiValueFields(schemaForValidation, rows[i]); - validateRecord(schemaForValidation, rows[i], 'insert', { mediaValueShapeStrict, messages: msgCtx }); + validateRecord(schemaForValidation, rows[i], 'insert', { mediaValueShapeStrict, valueShapeStrict, messages: msgCtx }); evaluateValidationRules(schemaForValidation as any, rows[i], 'insert', { logger: this.logger, currentUser: this.buildEvalUser(opCtx.context), skipStateMachine: shouldSkipStateMachine(opCtx.context), messages: msgCtx }); } catch (e) { if (!partialMode) throw e; @@ -3360,11 +3419,12 @@ export class ObjectQL implements IDataEngine { let priorRecord: Record | null = null; const updateSchema = this._registry.getObject(object); const mediaValueShapeStrict = await this.mediaValueShapeStrictFor(updateSchema); + const valueShapeStrict = await this.valueShapeStrictFor(updateSchema); const updateMsgCtx = this.validationMessageContext(object, opCtx.context); if (hookContext.input.id) { await this.encryptSecretFields(object, hookContext.input.data as Record, opCtx.context, hookContext.input.options); normalizeMultiValueFields(updateSchema, hookContext.input.data as Record); - validateRecord(updateSchema, hookContext.input.data as Record, 'update', { mediaValueShapeStrict, messages: updateMsgCtx }); + validateRecord(updateSchema, hookContext.input.data as Record, 'update', { mediaValueShapeStrict, valueShapeStrict, messages: updateMsgCtx }); if (needsPriorRecord(updateSchema as any) || (this.hooks.get('afterUpdate')?.length ?? 0) > 0) { const priorAst: QueryAST = { object, where: { id: hookContext.input.id }, limit: 1 }; priorRecord = await driver.findOne(object, priorAst, hookContext.input.options as any); @@ -3390,7 +3450,7 @@ export class ObjectQL implements IDataEngine { } else if (options?.multi && driver.updateMany) { await this.encryptSecretFields(object, hookContext.input.data as Record, opCtx.context, hookContext.input.options); normalizeMultiValueFields(updateSchema, hookContext.input.data as Record); - validateRecord(updateSchema, hookContext.input.data as Record, 'update', { mediaValueShapeStrict, messages: updateMsgCtx }); + validateRecord(updateSchema, hookContext.input.data as Record, 'update', { mediaValueShapeStrict, valueShapeStrict, messages: updateMsgCtx }); // [#2982] Consume the middleware-composed AST seeded above, so // the injected row-scoping (RLS write filter, sharing's // editable-rows filter) actually binds the driver operation. Fail diff --git a/packages/objectql/src/index.ts b/packages/objectql/src/index.ts index f3cac898e4..f5c7169b4b 100644 --- a/packages/objectql/src/index.ts +++ b/packages/objectql/src/index.ts @@ -75,6 +75,23 @@ export type { WrapDeclarativeOptions } from './hook-wrappers.js'; // Export Validation export { ValidationError, validateRecord } from './validation/record-validator.js'; export type { FieldValidationError } from './validation/record-validator.js'; +// [ADR-0104 D1 / #3438] The value-shape scan behind `os migrate value-shapes`. +// Read-only by design: it produces the evidence and the CALLER records the +// flag. The engine deliberately does not depend on `@objectstack/platform-objects` +// (readers of a flag use the spec contract; only writers take that dependency), +// so the composition lives with the command, not here. +export { + scanValueShapes, + valueShapeScanPassed, + formatValueShapeScanReport, +} from './validation/scan-value-shapes.js'; +export type { + ValueShapeFinding, + ValueShapeScanReport, + ValueShapeScanEngine, + ValueShapeScanOptions, + ValueShapeScanLogger, +} from './validation/scan-value-shapes.js'; export { evaluateValidationRules, needsPriorRecord, legalNextStates } from './validation/rule-validator.js'; export type { EvaluateRulesOptions } from './validation/rule-validator.js'; export { diff --git a/packages/objectql/src/validation/record-validator.ts b/packages/objectql/src/validation/record-validator.ts index acf9b9d1f6..537221b9e3 100644 --- a/packages/objectql/src/validation/record-validator.ts +++ b/packages/objectql/src/validation/record-validator.ts @@ -352,6 +352,7 @@ function validateOne( skipRequired = false, mediaStrict = false, ctx?: ValidationMessageContext, + valueStrict = false, ): FieldValidationError | null { const fail = ( code: FieldErrorCode, @@ -499,16 +500,23 @@ function validateOne( // ADR-0104 R7 external URLs are the deliberate exception the migration // reports and never converts, so a deployment still holding them fails // its self-check and stays lenient here. - // - **References and structured JSON** stay warn-first: the file migration - // says nothing about whether a `lookup` or `location` value is well - // formed, so gating them on ITS flag would be borrowing evidence for a - // fact it does not cover. They flip when something can vouch for them. + // - **References and structured JSON** enforce on their OWN evidence: the + // `os migrate value-shapes` scan, which walks every stored value of these + // classes against this same schema and records + // `adr-0104-value-shapes` only at zero violations. They are NOT gated on + // the file migration's flag — that one asserts file values were migrated + // and reconciled, and says nothing about whether a `lookup` id or a + // `location` payload is well formed. Borrowing it would be an authority + // answering a question nobody asked it. + // + // Two flags rather than one, because they attest two different facts; a + // deployment can legitimately have passed either without the other. if (REFERENCE_VALUE_TYPES.has(t) || FILE_REFERENCE_TYPES.has(t) || STRUCTURED_JSON_TYPES.has(t)) { const parsed = shapeSchemaFor(def).safeParse(value); if (!parsed.success) { const detail = parsed.error.issues[0]?.message ?? 'invalid value shape'; const isMedia = FILE_REFERENCE_TYPES.has(t); - if (isMedia ? mediaStrictEffective(mediaStrict) : VALUE_SHAPE_STRICT()) { + if (isMedia ? mediaStrictEffective(mediaStrict) : valueShapeStrictEffective(valueStrict)) { return fail('invalid_type', { type: t, detail }, 'invalid_value_shape'); } // The warn-first path is a DEVELOPER log line, not an end-user message — @@ -517,11 +525,9 @@ function validateOne( const message = `${name} has an invalid ${t} value: ${detail}`; warnOnce( `${t}:${name}`, - `[value-shape] ${message} — accepted for now (ADR-0104 warn-first; ` + - (isMedia - ? 'run `os migrate files-to-references --apply` to migrate this deployment and enforce' - : 'set OS_DATA_VALUE_SHAPE_STRICT_ENABLED=1 to enforce') + - ')', + `[value-shape] ${message} — accepted for now (ADR-0104 warn-first; run \`os migrate ` + + (isMedia ? 'files-to-references' : 'value-shapes') + + ' --apply` to migrate this deployment and enforce)', ); } return null; @@ -532,7 +538,12 @@ function validateOne( return null; } -/** Strict value-shape enforcement opt-in (ADR-0104 warn-first rollout). */ +/** + * All-class strict opt-in (ADR-0104). Turns on EVERY value class at once, + * including ones whose per-deployment migration has not run here — so it is + * the "I already know my data" lever, not the blessed route. The blessed route + * is running the migration that produces the evidence. + */ function VALUE_SHAPE_STRICT(): boolean { return typeof process !== 'undefined' && process.env?.OS_DATA_VALUE_SHAPE_STRICT_ENABLED === '1'; } @@ -546,6 +557,16 @@ function LAX_MEDIA_VALUES(): boolean { return typeof process !== 'undefined' && process.env?.OS_ALLOW_LAX_MEDIA_VALUES === '1'; } +/** + * The same escape hatch for the reference / structured-JSON classes, whose + * evidence is the `adr-0104-value-shapes` scan rather than the file migration. + * Deliberately a SECOND variable: the two gates open on different evidence, so + * an operator backing out of one has said nothing about the other. + */ +function LAX_VALUE_SHAPES(): boolean { + return typeof process !== 'undefined' && process.env?.OS_ALLOW_LAX_VALUE_SHAPES === '1'; +} + /** * Does a media value-shape violation reject, for this deployment? * @@ -560,6 +581,57 @@ function mediaStrictEffective(deploymentVerified: boolean): boolean { return deploymentVerified; } +/** + * The reference / structured-JSON counterpart — identical precedence, its own + * pair of inputs. A sibling function rather than one parameterised helper so + * each gate names the evidence it opens on at its own call site; collapsing + * them would save three lines and lose the distinction the ADR spent an + * addendum drawing. + */ +function valueShapeStrictEffective(deploymentVerified: boolean): boolean { + if (LAX_VALUE_SHAPES()) return false; + if (VALUE_SHAPE_STRICT()) return true; + return deploymentVerified; +} + +/** + * Is `value` a violation of `def`'s declared stored shape — the SAME question + * `validateOne` asks, exported so the `os migrate value-shapes` scanner counts + * exactly what strict mode would reject and nothing else. + * + * The scanner must not re-derive this. Two implementations of "malformed" + * drifting by one clause is how a deployment passes a scan and then has writes + * rejected anyway — the migration would be attesting a fact the validator does + * not recognise, which is precisely the borrowed-evidence failure the ADR's + * addendum forbids one layer up. + * + * Returns the first parse issue's message, or `null` when the value conforms. + * A value that is missing (per `isMissing`) is never a violation: absence is + * the `required` check's business, not the shape contract's. + */ +export function valueShapeViolation(def: FieldDef, value: unknown): string | null { + if (isMissing(value)) return null; + const t = def.type; + if (!REFERENCE_VALUE_TYPES.has(t) && !FILE_REFERENCE_TYPES.has(t) && !STRUCTURED_JSON_TYPES.has(t)) { + return null; + } + const parsed = shapeSchemaFor(def).safeParse(value); + if (parsed.success) return null; + return parsed.error.issues[0]?.message ?? 'invalid value shape'; +} + +/** + * Is this field one the value-shape scan covers, and one a client may write? + * `system` / `readonly` / lifecycle columns are skipped for the same reason + * `validateRecord` skips them — the engine owns them, so they are never + * validated on a write and must never be counted as blocking a gate that + * governs writes. + */ +export function isScannableValueShapeField(name: string, def: FieldDef | undefined): boolean { + if (!def || SKIP_FIELDS.has(name) || def.system || def.readonly) return false; + return REFERENCE_VALUE_TYPES.has(def.type) || STRUCTURED_JSON_TYPES.has(def.type); +} + const warnedShapes = new Set(); function warnOnce(key: string, message: string): void { if (warnedShapes.has(key)) return; @@ -598,6 +670,21 @@ export interface ValidateRecordOptions { */ mediaValueShapeStrict?: boolean; + /** + * Has THIS DEPLOYMENT completed and self-check-verified the ADR-0104 + * non-media value-shape scan (`os migrate value-shapes`, #3438)? When true, a + * malformed reference (`lookup` / `master_detail` / `user` / `tree`) or + * structured-JSON (`location` / `address` / `composite` / `repeater` / + * `record` / `vector`) value rejects instead of warning. + * + * Separate from {@link mediaValueShapeStrict} because the two facts are + * attested by two different migrations and either can hold without the other. + * Same default and same reason: a caller that cannot say stays lenient, so + * nothing starts rejecting writes merely because the evidence was + * unavailable. + */ + valueShapeStrict?: boolean; + /** * Locale + translation hooks for the human half of each error (#3957). Omit * and messages render in `en` against the declared labels — the pre-#3957 @@ -624,6 +711,7 @@ export function validateRecord( const errors: FieldValidationError[] = []; const fields = objectSchema.fields; const mediaStrict = options.mediaValueShapeStrict === true; + const valueStrict = options.valueShapeStrict === true; const messages = options.messages; if (mode === 'insert') { @@ -632,7 +720,7 @@ export function validateRecord( for (const [name, def] of Object.entries(fields)) { if (SKIP_FIELDS.has(name)) continue; if (def.system || def.readonly) continue; - const err = validateOne(name, def, data[name], false, mediaStrict, messages); + const err = validateOne(name, def, data[name], false, mediaStrict, messages, valueStrict); if (err) errors.push(err); } } else { @@ -661,7 +749,7 @@ export function validateRecord( // skipRequired: PATCH-omitted fields must not 400. (No def clone — the // registry's own field object flows through so the ADR-0104 value-shape // schema cache, keyed on def identity, hits.) - const err = validateOne(name, def, value, true, mediaStrict, messages); + const err = validateOne(name, def, value, true, mediaStrict, messages, valueStrict); if (err) errors.push(err); } } diff --git a/packages/objectql/src/validation/scan-value-shapes.test.ts b/packages/objectql/src/validation/scan-value-shapes.test.ts new file mode 100644 index 0000000000..59a69c7142 --- /dev/null +++ b/packages/objectql/src/validation/scan-value-shapes.test.ts @@ -0,0 +1,123 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import { describe, it, expect } from 'vitest'; +import { scanValueShapes, valueShapeScanPassed } from './scan-value-shapes.js'; +import { validateRecord, ValidationError } from './record-validator.js'; + +const OBJECTS: Record = { + contact: { + fields: { + id: { type: 'text' }, + name: { type: 'text' }, + account: { type: 'lookup', reference: 'account' }, + geo: { type: 'location' }, + // Engine-owned: never validated on a write, so never scanned either. + owner_cache: { type: 'lookup', reference: 'sys_user', system: true }, + }, + }, + // No covered field — must not be walked at all. + note: { fields: { id: { type: 'text' }, body: { type: 'textarea' } } }, +}; + +function makeEngine(rows: Record>>) { + const reads: string[] = []; + return { + reads, + getObject: (n: string) => OBJECTS[n], + getConfigs: () => OBJECTS, + async find(object: string) { + reads.push(object); + return rows[object] ?? []; + }, + }; +} + +const silent = { info: () => {}, warn: () => {} }; + +describe('scanValueShapes (ADR-0104 D1 / #3438)', () => { + it('a clean database passes and only walks objects declaring covered fields', async () => { + const engine = makeEngine({ + contact: [{ id: 'c1', account: 'acc_1', geo: { lat: 1, lng: 2 } }], + note: [{ id: 'n1', body: 'hi' }], + }); + const report = await scanValueShapes(engine, silent); + + expect(report.blocking).toBe(0); + expect(valueShapeScanPassed(report)).toBe(true); + expect(report.scannedObjects).toEqual(['contact']); + expect(engine.reads).not.toContain('note'); + }); + + it('counts malformed values per field, with samples and the parse detail', async () => { + const engine = makeEngine({ + contact: [ + { id: 'c1', geo: { latitude: 1, longitude: 2 } }, // the pre-ADR spec shape + { id: 'c2', geo: { latitude: 3, longitude: 4 } }, + { id: 'c3', account: { id: 'acc_1', name: 'Acme' } }, // expanded, not an id + { id: 'c4', account: 'acc_2', geo: { lat: 0, lng: 0 } }, // fine + ], + }); + const report = await scanValueShapes(engine, silent); + + expect(report.scannedRecords).toBe(4); + expect(report.blocking).toBe(3); + const geo = report.findings.find((f) => f.field === 'geo')!; + expect(geo.count).toBe(2); + expect(geo.type).toBe('location'); + expect(geo.sampleRecordIds).toEqual(['c1', 'c2']); + expect(geo.detail).toBeTruthy(); + expect(report.findings.find((f) => f.field === 'account')!.count).toBe(1); + expect(valueShapeScanPassed(report)).toBe(false); + }); + + it('never counts a system/readonly field — the write path never validates one', async () => { + const engine = makeEngine({ + contact: [{ id: 'c1', owner_cache: { id: 'u1', name: 'nope' } }], + }); + const report = await scanValueShapes(engine, silent); + expect(report.blocking).toBe(0); + }); + + it('a missing or null value is not a violation — that is `required`\'s business', async () => { + const engine = makeEngine({ + contact: [{ id: 'c1', account: null, geo: undefined }, { id: 'c2' }], + }); + const report = await scanValueShapes(engine, silent); + expect(report.blocking).toBe(0); + }); + + it('an unreadable object closes the gate even with zero violations found', async () => { + const engine = { + getObject: (n: string) => OBJECTS[n], + getConfigs: () => OBJECTS, + async find(): Promise>> { + throw new Error('table missing'); + }, + }; + const report = await scanValueShapes(engine, silent); + + expect(report.blocking).toBe(0); + expect(report.unreadableObjects).toEqual(['contact']); + expect(report.truncated).toBe(true); + // "none in the part we could read" is not the claim the flag makes. + expect(valueShapeScanPassed(report)).toBe(false); + }); + + it('the scan counts exactly what strict mode rejects — one predicate, not two', async () => { + // The anti-drift property: every value the scan flags must also be a write + // rejection under the strict gate, and every value it passes must write. + const flagged = { geo: { latitude: 1, longitude: 2 } }; + const clean = { geo: { lat: 1, lng: 2 } }; + + const engine = makeEngine({ contact: [{ id: 'c1', ...flagged }] }); + const report = await scanValueShapes(engine, silent); + expect(report.blocking).toBe(1); + + expect(() => + validateRecord(OBJECTS.contact, { ...flagged }, 'update', { valueShapeStrict: true }), + ).toThrow(ValidationError); + expect(() => + validateRecord(OBJECTS.contact, { ...clean }, 'update', { valueShapeStrict: true }), + ).not.toThrow(); + }); +}); diff --git a/packages/objectql/src/validation/scan-value-shapes.ts b/packages/objectql/src/validation/scan-value-shapes.ts new file mode 100644 index 0000000000..79bd6d6cb7 --- /dev/null +++ b/packages/objectql/src/validation/scan-value-shapes.ts @@ -0,0 +1,236 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * The ADR-0104 D1 non-media value-shape scan — the evidence half of the + * per-deployment gate `os migrate value-shapes` opens (#3438). + * + * ## Why a scan and not a backfill + * + * Its sibling migration (`os migrate files-to-references`) converts legacy file + * values before reconciling them, because the platform narrowed that storage + * form and therefore owes the conversion. Nothing comparable applies here: a + * `location` stored as `{latitude, longitude}` or a `lookup` holding an + * expanded record object is APPLICATION data whose correct value only its + * author knows. So this run reports and prescribes; it never rewrites. A + * mechanical normaliser is deliberately absent until real reports show a shape + * common enough to justify one — a normaliser that guesses is how + * silently-wrong values got here in the first place. + * + * The consequence is a pleasing simplification: with nothing to convert, + * `--apply`'s only write is the flag row itself, so the #3617 invariant holds + * trivially — a dry run changes nothing, and whether a run changed this + * deployment's posture never depends on what the run happened to find. + * + * ## Why the predicate is imported + * + * `valueShapeViolation` and `isScannableValueShapeField` come from the write- + * path validator. The scan must count exactly what strict mode would reject — + * same type classes, same skip rules, same cached schemas. A second + * implementation of "malformed" drifting by one clause would let a deployment + * pass its scan and still have writes rejected, which is the flag attesting a + * fact the validator does not recognise. + */ + +import { + valueShapeViolation, + isScannableValueShapeField, +} from './record-validator.js'; + +/** One object/field pair holding at least one malformed value. */ +export interface ValueShapeFinding { + object: string; + field: string; + /** The declared field type, so the report can group by what went wrong. */ + type: string; + /** How many records of this object hold a malformed value in this field. */ + count: number; + /** A few offending record ids, so an operator can go and look. */ + sampleRecordIds: string[]; + /** The first parse issue seen — the prescription an author acts on. */ + detail: string; +} + +export interface ValueShapeScanReport { + /** Objects actually walked (those declaring at least one covered field). */ + scannedObjects: string[]; + scannedRecords: number; + /** Object/field pairs with violations. Empty ⇒ the gate may open. */ + findings: ValueShapeFinding[]; + /** Total malformed values across every finding. */ + blocking: number; + /** + * True when a per-object cap or an unreadable object cut the walk short. + * A truncated scan can never open the gate: it is evidence about part of the + * data, and the claim being made is about all of it. + */ + truncated: boolean; + /** Objects that could not be read at all — reported, and they truncate. */ + unreadableObjects: string[]; +} + +export interface ValueShapeScanEngine { + getObject(name: string): any | undefined; + getConfigs?(): Record; + find(object: string, options: Record): Promise>>; +} + +export interface ValueShapeScanOptions { + /** Restrict to these objects (default: every object with a covered field). */ + objects?: string[]; + maxRecordsPerObject?: number; +} + +export interface ValueShapeScanLogger { + info(msg: string, meta?: unknown): void; + warn(msg: string, meta?: unknown): void; +} + +const SCAN_PAGE_SIZE = 500; +const MAX_SAMPLE_IDS = 5; +const SYSTEM_CTX = { isSystem: true } as const; + +/** The covered, client-writable fields of one object. */ +function scannableFieldsOf(engine: ValueShapeScanEngine, objectName: string): Array<{ name: string; def: any }> { + const schema: any = engine.getObject(objectName); + if (!schema?.fields) return []; + return Object.entries(schema.fields) + .filter(([name, def]: [string, any]) => isScannableValueShapeField(name, def)) + .map(([name, def]: [string, any]) => ({ name, def })); +} + +/** + * Walk every stored value of the reference / structured-JSON classes and count + * the ones the write path would reject. + * + * Read-only by construction — this function has no write surface at all, which + * is why the engine type it accepts exposes none. + */ +export async function scanValueShapes( + engine: ValueShapeScanEngine, + logger: ValueShapeScanLogger, + options: ValueShapeScanOptions = {}, +): Promise { + const maxPerObject = options.maxRecordsPerObject ?? 100_000; + let scannedRecords = 0; + let truncated = false; + const unreadableObjects: string[] = []; + + const candidates = + options.objects ?? + (typeof engine.getConfigs === 'function' ? Object.keys(engine.getConfigs()) : []); + const scannedObjects = candidates.filter((name) => scannableFieldsOf(engine, name).length > 0); + + // Keyed `object.field`, so one finding aggregates every offending row. + const findings = new Map(); + + for (const object of scannedObjects) { + const fields = scannableFieldsOf(engine, object); + let offset = 0; + + for (;;) { + let page: Array>; + try { + page = await engine.find(object, { + fields: ['id', ...fields.map((f) => f.name)], + limit: SCAN_PAGE_SIZE, + offset, + context: { ...SYSTEM_CTX }, + }); + } catch (err) { + // An object we cannot read is not an object we may vouch for. Record it + // AND truncate: the gate's claim is about all the data, so a hole in + // the evidence has to close it. + logger.warn( + `[value-shape] scan: cannot read ${object} (${(err as Error)?.message ?? err}) — ` + + 'reported as unreadable; the gate stays closed', + ); + unreadableObjects.push(object); + truncated = true; + break; + } + if (!page || page.length === 0) break; + + for (const record of page) { + scannedRecords++; + for (const { name, def } of fields) { + const detail = valueShapeViolation(def, record[name]); + if (!detail) continue; + const key = `${object}.${name}`; + const existing = findings.get(key); + const recordId = String(record.id ?? '(unknown id)'); + if (existing) { + existing.count++; + if (existing.sampleRecordIds.length < MAX_SAMPLE_IDS) existing.sampleRecordIds.push(recordId); + } else { + findings.set(key, { + object, + field: name, + type: String(def?.type ?? 'unknown'), + count: 1, + sampleRecordIds: [recordId], + detail, + }); + } + } + } + + offset += page.length; + if (page.length < SCAN_PAGE_SIZE) break; + if (offset >= maxPerObject) { + truncated = true; + break; + } + } + } + + const list = [...findings.values()].sort((a, b) => b.count - a.count); + return { + scannedObjects, + scannedRecords, + findings: list, + blocking: list.reduce((n, f) => n + f.count, 0), + truncated, + unreadableObjects, + }; +} + +/** + * Does this scan authorise the gate? Zero violations AND a complete walk — + * `blocking === 0` on a truncated scan means "none in the part we read", which + * is not the claim the flag makes. + */ +export function valueShapeScanPassed(report: ValueShapeScanReport): boolean { + return report.blocking === 0 && !report.truncated; +} + +/** Human-readable report body, shared by the CLI's dry-run and apply output. */ +export function formatValueShapeScanReport(report: ValueShapeScanReport): string[] { + const lines: string[] = []; + lines.push( + `Scanned ${report.scannedRecords} record(s) across ${report.scannedObjects.length} object(s) ` + + 'for reference / structured-JSON value shapes.', + ); + if (report.findings.length === 0) { + lines.push('✓ No malformed values found.'); + } else { + lines.push(`✗ ${report.blocking} malformed value(s) in ${report.findings.length} field(s):`); + for (const f of report.findings) { + lines.push( + ` • ${f.object}.${f.field} (${f.type}) — ${f.count} record(s): ${f.detail}` + + `\n e.g. ${f.sampleRecordIds.join(', ')}`, + ); + } + lines.push( + 'These are application data, not platform data: fix the values (or the code that writes', + 'them) and re-run. Nothing here is converted for you — the correct value is one only the', + "app's author knows.", + ); + } + if (report.unreadableObjects.length > 0) { + lines.push(`⚠ Unreadable object(s): ${report.unreadableObjects.join(', ')}`); + } + if (report.truncated) { + lines.push('⚠ Scan incomplete — the gate cannot open on partial evidence.'); + } + return lines; +} diff --git a/packages/spec/api-surface.json b/packages/spec/api-surface.json index 6e3aeef13f..f625f9e8ae 100644 --- a/packages/spec/api-surface.json +++ b/packages/spec/api-surface.json @@ -1316,6 +1316,7 @@ "UserActivityStatus (type)", "VALIDATION_MESSAGE_FALLBACK_LOCALE (const)", "VALIDATION_MESSAGE_KEY_PREFIX (const)", + "VALUE_SHAPES_MIGRATION_ID (const)", "ValidationMessageTranslator (type)", "VectorClock (type)", "VectorClockSchema (const)", diff --git a/packages/spec/src/system/migration.zod.ts b/packages/spec/src/system/migration.zod.ts index 5f1845af94..70194c7c51 100644 --- a/packages/spec/src/system/migration.zod.ts +++ b/packages/spec/src/system/migration.zod.ts @@ -137,6 +137,28 @@ export const DATA_MIGRATION_FLAG_OBJECT = 'sys_migration'; */ export const FILE_REFERENCES_MIGRATION_ID = 'adr-0104-file-references'; +/** + * Well-known migration id: ADR-0104 D1 non-media value shapes — every stored + * reference (`lookup` / `master_detail` / `user` / `tree`) and structured-JSON + * (`location` / `address` / `composite` / `repeater` / `record` / `vector`) + * value scanned against `valueSchemaFor(field, 'stored')` with zero + * violations. Gates strict enforcement of those classes on this deployment + * (#3438). + * + * Deliberately SEPARATE from {@link FILE_REFERENCES_MIGRATION_ID}, which + * asserts a different fact. That flag says this deployment's file-field values + * were migrated and their ownership reconciled; it says nothing about whether a + * `lookup` id or a `location` payload is well formed. Gating these classes on + * it would be borrowing evidence for a fact it does not cover — the same error + * one layer down as an authority answering a question nobody asked (see the + * ADR's 2026-07-27 amendment). + * + * Unlike the file migration this one has no backfill: a malformed `location` is + * an application-data fix only its author can make, so the run reports and + * prescribes, and `--apply`'s only write is the flag row itself. + */ +export const VALUE_SHAPES_MIGRATION_ID = 'adr-0104-value-shapes'; + export const DataMigrationFlagSchema = lazySchema(() => z.object({ id: z.string().describe('Migration id (e.g. adr-0104-file-references) — one row per data migration'), last_run_at: z.string().datetime().describe('When this migration last completed a gated (apply-mode) run on this deployment'), From 48173360cd6fd345b61f6f8d9023cd0823a666bf Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 30 Jul 2026 17:05:13 +0000 Subject: [PATCH 2/3] docs(cli): document `os migrate value-shapes` alongside its sibling MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The docs-drift check on the command's own PR flagged content/docs/deployment/cli.mdx, and that one was a real gap rather than package-level noise: the file documents `os migrate files-to-references` in full, so shipping a second gated data migration without an entry would have left it discoverable only by reading the source. Covers the deliberate asymmetry (this run converts nothing — a malformed `location` is application data whose correct value only its author knows), the truncation rule (a partial scan cannot open the gate), and why the flag is separate from the file migration's rather than reusing it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016zgA8CQMJeJbFEjnbib1Uv --- content/docs/deployment/cli.mdx | 44 +++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/content/docs/deployment/cli.mdx b/content/docs/deployment/cli.mdx index 2f51c1db70..10d30c6ab9 100644 --- a/content/docs/deployment/cli.mdx +++ b/content/docs/deployment/cli.mdx @@ -628,6 +628,7 @@ where the data lives. | Command | Description | |---------|-------------| | `os migrate files-to-references` | Convert legacy file-field values to `sys_file` references, verify the ownership ledger, and record the deployment's migration flag | +| `os migrate value-shapes` | Scan stored reference and structured-JSON field values against the platform's value contract, and record the deployment's migration flag when clean | ```bash os migrate files-to-references # Dry run: full report, writes nothing @@ -659,6 +660,49 @@ Exit status is `0` only when the self-check passes, so CI can gate on it. | **Media value shapes** | A malformed `file` / `image` / `avatar` / `video` / `audio` value is **rejected** (`400 invalid_type`) instead of warned about. Set `OS_ALLOW_LAX_MEDIA_VALUES=1` to re-open leniency while diagnosing. | | **Released-file collection** | A field file whose one owning record lets go (the field is cleared or the record deleted) is tombstoned into the declared 30-day grace window; re-referencing the id within the window revives it, and after it the platform sweep reclaims the row and its bytes. Unverified deployments keep every released file forever. | +#### `os migrate value-shapes` + +The same gate for the **non-media** value classes — references (`lookup`, +`master_detail`, `user`, `tree`) and structured JSON (`location`, `address`, +`composite`, `repeater`, `record`, `vector`). + +```bash +os migrate value-shapes # Scan: full report, writes nothing +os migrate value-shapes --apply # Scan, then record the flag if clean (prompts) +os migrate value-shapes --apply --yes --json # CI / scripts +os migrate value-shapes --object contact # Restrict to one object (repeatable) +``` + +**This one converts nothing.** Its sibling rewrites legacy file values because +the platform narrowed that storage form and therefore owes the conversion; a +`location` stored as `{latitude, longitude}` instead of `{lat, lng}` is +*application* data whose correct value only its author knows. So the run reports +— object, field, type, how many records, sample record ids, and the parse issue +— and you fix the values (or the code writing them) and re-run until it is +green. Because there is nothing to convert, `--apply`'s only write is the flag +row itself. + +A scan that is **truncated** (by `--max-records`) or that cannot read an object +fails the gate even with zero violations found: "none in the part we read" is +not the claim the flag makes. + +| Once verified | Effect | +| :--- | :--- | +| **Reference + structured-JSON value shapes** | A malformed value of those classes is **rejected** (`400 invalid_type`) instead of warned about. Set `OS_ALLOW_LAX_VALUE_SHAPES=1` to re-open leniency while diagnosing. | + +This flag is deliberately **separate** from the file migration's. That one +attests that file values were migrated and their ownership reconciled — it says +nothing about whether a `lookup` id or a `location` payload is well formed, so +it may not vouch for these classes. A deployment can legitimately have passed +either without the other. + + +`OS_DATA_VALUE_SHAPE_STRICT_ENABLED=1` turns on **every** value class at once, +regardless of which migrations this deployment has run. It is the "I already +know my data" lever, not the route to strictness — the route is running the +migration that produces the evidence. + + Other value classes are unaffected: a `lookup` or `location` value keeps its own warn-first rollout, because this migration is evidence about *file* values and says nothing about theirs. From 13606e084bff232a3870e8dc11e23f3e5b712036 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 30 Jul 2026 18:39:29 +0000 Subject: [PATCH 3/3] docs: reconcile the value-shapes gate with the creation-attestation notes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #4215 landed #3438's third item (a store created from empty attests both flags) while this branch was open, so the merged docs claimed "both flags" without the second one ever being named. - cli.mdx: the file migration's closing paragraph and writing-rules callout had been split off from their own section by the new one inserted above them; move them back so each command's rules sit under that command, and point the "other value classes keep warning" sentence at the gate that now exists for them. - cli.mdx: the created-from-empty section offers an escape hatch per class, so it names both — the media one and the one this branch adds. - v17.mdx: document `os migrate value-shapes` and why its flag is separate, and give the upgrade checklist its own bullet. The old wording told readers OS_DATA_VALUE_SHAPE_STRICT_ENABLED opts in "classes with no migration behind them", which stopped being true here. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016zgA8CQMJeJbFEjnbib1Uv --- content/docs/deployment/cli.mdx | 41 +++++++++++++++++++-------------- content/docs/releases/v17.mdx | 23 +++++++++++++++++- 2 files changed, 46 insertions(+), 18 deletions(-) diff --git a/content/docs/deployment/cli.mdx b/content/docs/deployment/cli.mdx index dfcb8e9bf1..e9780c8037 100644 --- a/content/docs/deployment/cli.mdx +++ b/content/docs/deployment/cli.mdx @@ -660,6 +660,23 @@ Exit status is `0` only when the self-check passes, so CI can gate on it. | **Media value shapes** | A malformed `file` / `image` / `avatar` / `video` / `audio` value is **rejected** (`400 invalid_type`) instead of warned about. Set `OS_ALLOW_LAX_MEDIA_VALUES=1` to re-open leniency while diagnosing. | | **Released-file collection** | A field file whose one owning record lets go (the field is cleared or the record deleted) is tombstoned into the declared 30-day grace window; re-referencing the id within the window revives it, and after it the platform sweep reclaims the row and its bytes. Unverified deployments keep every released file forever. | +Other value classes are unaffected: a `lookup` or `location` value keeps its own +warn-first rollout until `os migrate value-shapes` (below) supplies *their* +evidence, because this migration is evidence about *file* values and says +nothing about theirs. + + +A dry run writes **nothing** — not the conversions, and not the flag either, +even when the self-check would pass. `--apply` is the only writing mode. A +later run that *fails* its self-check clears the flag's verified state, so a +database that has drifted closes its own gate. + +A running server reads the flag once; after migrating, **restart it** for +enforcement (and release-time tombstoning) to take effect. The sweep's final +delete check re-reads the flag fresh, so a later failing run stops collection +without a restart. + + #### `os migrate value-shapes` The same gate for the **non-media** value classes — references (`lookup`, @@ -703,21 +720,9 @@ know my data" lever, not the route to strictness — the route is running the migration that produces the evidence. -Other value classes are unaffected: a `lookup` or `location` value keeps its own -warn-first rollout, because this migration is evidence about *file* values and -says nothing about theirs. - - -A dry run writes **nothing** — not the conversions, and not the flag either, -even when the self-check would pass. `--apply` is the only writing mode. A -later run that *fails* its self-check clears the flag's verified state, so a -database that has drifted closes its own gate. - -A running server reads the flag once; after migrating, **restart it** for -enforcement (and release-time tombstoning) to take effect. The sweep's final -delete check re-reads the flag fresh, so a later failing run stops collection -without a restart. - +Same writing rules as its sibling: a dry run writes **nothing**, `--apply` is +the only writing mode, a later failing run clears the verified state, and a +running server reads the flag once — **restart it** after a successful apply. #### A database created by this version needs no migration @@ -735,8 +740,10 @@ attests nothing and produces its evidence by running the command, because Importing legacy values into such a deployment is rejected at the write path rather than silently accepted. That is the intended outcome; if you must admit -them temporarily, `OS_ALLOW_LAX_MEDIA_VALUES=1` re-opens leniency, and -re-running the migration re-establishes the flag from the data itself. +them temporarily, `OS_ALLOW_LAX_MEDIA_VALUES=1` (media) and +`OS_ALLOW_LAX_VALUE_SHAPES=1` (references and structured JSON) re-open leniency +per class, and re-running the corresponding migration re-establishes its flag +from the data itself. ### Scaffolding diff --git a/content/docs/releases/v17.mdx b/content/docs/releases/v17.mdx index 7a52c0b158..7824fa29e0 100644 --- a/content/docs/releases/v17.mdx +++ b/content/docs/releases/v17.mdx @@ -984,6 +984,23 @@ records** — never the platform version — is what authorises both strict medi value shapes and irreversible file collection. Upgrading changes neither; running the migration does. +**The non-media classes have their own gate** (#3438) — references (`lookup`, +`master_detail`, `user`, `tree`) and structured JSON (`location`, `address`, +`composite`, `repeater`, `record`, `vector`): + +```bash +os migrate value-shapes # scan: reports, writes nothing +os migrate value-shapes --apply # scan, then record the flag if clean +``` + +A separate flag because it attests a separate fact: the file migration says file +values were converted and their ownership reconciled, which tells you nothing +about whether a `lookup` id or a `location` payload is well formed. This one +converts nothing — a malformed `location` is application data only its author +can correct — so it reports the object, field, type, count, sample record ids +and parse issue, and you re-run until it is clean. `OS_ALLOW_LAX_VALUE_SHAPES=1` +re-opens leniency while diagnosing. + **A database created by 17 attests both flags at creation** (#3438), so a new deployment enforces from its first boot instead of waiting for someone to run a migration that, for an empty store, does nothing. The platform attests only a @@ -1306,7 +1323,11 @@ objectui commits on top of the pin 16.1.0 shipped. media value shapes *and* released-file collection for this deployment, so read the report before you `--apply`. Do **not** reach for `OS_DATA_VALUE_SHAPE_STRICT_ENABLED` to get there: it opts every value class - in at once, including ones with no migration behind them. + in at once, regardless of which migrations this deployment has actually run. +- **Reference and structured-JSON values:** run `os migrate value-shapes` and + fix what it reports before `--apply`. It converts nothing — the values it + names are application data — and a scan that was truncated or could not read + an object fails the gate even at zero violations. - **Datasources:** verify every declared datasource connects in every environment — a bound datasource that cannot connect now fails the boot instead of failing every later query.