feat(spec): field runtime value-shape contract — ADR-0104 phase 1 (D1)

.changeset/adr-0104-d1-value-shape-contract.md

@@ -0,0 +1,41 @@
+---
+"@objectstack/spec": minor
+"@objectstack/objectql": minor
+"@objectstack/rest": patch
+"@objectstack/driver-sql": patch
+---
+
+feat(spec): field runtime value-shape contract — ADR-0104 phase 1 (D1)
+
+`@objectstack/spec/data` now owns the runtime VALUE shape of every field type
+(`field-value.zod.ts`): semantic type classes (`STRING_VALUE_TYPES`,
+`NUMERIC_VALUE_TYPES`, `REFERENCE_VALUE_TYPES`, `FILE_REFERENCE_TYPES`,
+`STRUCTURED_JSON_TYPES`, `MULTI_CAPABLE_TYPES`, …), the shared
+`isMultiValueField`, and `valueSchemaFor(field, 'stored' | 'expanded')`. The
+four consumers that each hand-copied this knowledge (objectql record-validator,
+rest import-coerce, driver-sql column classification, qa conformance) now
+derive from the spec, and the field-zoo round-trip MATRIX is asserted against
+the contract so the two cannot drift.
+
+**Write-path change (objectql, warn-first):** previously-unvalidated types —
+single `lookup`/`master_detail`/`user`/`tree`, `file`/`image`/`avatar`/
+`video`/`audio`, `location`, `address`, `composite`, `repeater`, `record`,
+`vector` — are now checked against the contract. A violation **logs a warning
+and passes** in this release (legacy rows must not strand their records);
+set `OS_DATA_VALUE_SHAPE_STRICT_ENABLED=1` to enforce as a
+`400 VALIDATION_FAILED`. The flip to strict-by-default rides a later minor
+(ADR-0104 R1/R2).
+
+**Deprecations (removal rides the next spec major), FROM → TO:**
+
+- `CurrencyValueSchema` (`{value, currency}`) → none. A `currency` field's
+  value is a **bare number** everywhere in the runtime (validator, SQL `float`
+  column, import coercion, field-zoo oracle); the currency code lives in field
+  config. Use `valueSchemaFor({type: 'currency'})`.
+- `LocationCoordinatesSchema` (`{latitude, longitude}`) → `LocationValueSchema`
+  (`{lat, lng}`) — the shape the platform actually stores.
+- `AddressSchema` is **adopted** (unchanged) as the enforced `address` value
+  contract via `AddressValueSchema`.
+
+No stored data changes shape; the contract codifies deployed reality
+("reality wins", ADR-0104 D1).

content/docs/references/data/field-value.mdx

@@ -0,0 +1,141 @@
+---
+title: Field Value
+description: Field Value protocol schemas
+---
+
+{/* ⚠️  AUTO-GENERATED — DO NOT EDIT. Run build-docs.ts to regenerate. Hand-written docs live in the module folders under content/docs/. */}
+
+Field runtime VALUE-shape contract (ADR-0104 D1).
+
+`FieldSchema` owns what a field *definition* looks like; this module owns
+
+what a field's runtime *value* looks like — the shape the write path
+
+accepts, drivers persist, and an unexpanded API read returns. Before this
+
+module the knowledge lived as private, hand-copied type sets in objectql's
+
+record-validator, rest's import-coerce, driver-sql, and verify; adding one
+
+multi-capable or JSON-shaped type meant updating four lists or silently
+
+corrupting data. Those consumers now derive from the classes below.
+
+Two canonical forms exist per field (ADR-0104 D1):
+
+- `stored`   — the storage/wire form (e.g. lookup ⇒ record-id string,
+
+`date` ⇒ `YYYY-MM-DD`, select ⇒ option code).
+
+- `expanded` — the enriched `$expand` read form (lookup ⇒ the related
+
+record object). For types without an expansion,
+
+expanded ≡ stored.
+
+"Reality wins": where the deployed stored shape is coherent, the contract
+
+adopts it — deployed data is a wire contract we don't get to rewrite by
+
+editing Zod. This is why `currency` is a bare number (not the never-consumed
+
+`CurrencyValueSchema` object) and `location` is `\{lat, lng\}` (what field-zoo
+
+stores), not the never-consumed `\{latitude, longitude\}` shape.
+
+Purity: schemas/constants/derivation only — no runtime logic, no caching
+
+(Prime Directive #2). Consumers cache `valueSchemaFor` results per field
+
+definition; building a Zod schema per write is the one performance trap
+
+this contract has (ADR-0104 performance budget).
+
+<Callout type="info">
+**Source:** `packages/spec/src/data/field-value.zod.ts`
+</Callout>
+
+## TypeScript Usage
+
+```typescript
+import { AddressValue, CalendarDateValue, ClockTimeValue, FileLikeValue, InstantValue, LocationValue, ReferenceIdValue } from '@objectstack/spec/data';
+import type { AddressValue, CalendarDateValue, ClockTimeValue, FileLikeValue, InstantValue, LocationValue, ReferenceIdValue } from '@objectstack/spec/data';
+
+// Validate data
+const result = AddressValue.parse(data);
+```
+
+---
+
+## AddressValue
+
+### Properties
+
+| Property | Type | Required | Description |
+| :--- | :--- | :--- | :--- |
+| **street** | `string` | optional | Street address |
+| **city** | `string` | optional | City name |
+| **state** | `string` | optional | State/Province |
+| **postalCode** | `string` | optional | Postal/ZIP code |
+| **country** | `string` | optional | Country name or code |
+| **countryCode** | `string` | optional | ISO country code (e.g., US, GB) |
+| **formatted** | `string` | optional | Formatted address string |
+
+
+---
+
+
+---
+
+
+---
+
+## FileLikeValue
+
+### Union Options
+
+This schema accepts one of the following structures:
+
+#### Option 1
+
+Type: `string`
+
+---
+
+#### Option 2
+
+### Properties
+
+| Property | Type | Required | Description |
+| :--- | :--- | :--- | :--- |
+| **url** | `string` | optional |  |
+| **name** | `string` | optional |  |
+| **size** | `number` | optional |  |
+| **alt** | `string` | optional |  |
+| **duration** | `number` | optional |  |
+
+---
+
+
+---
+
+
+---
+
+## LocationValue
+
+### Properties
+
+| Property | Type | Required | Description |
+| :--- | :--- | :--- | :--- |
+| **lat** | `number` | ✅ | Latitude |
+| **lng** | `number` | ✅ | Longitude |
+| **altitude** | `number` | optional | Altitude in meters |
+| **accuracy** | `number` | optional | Accuracy in meters |
+
+
+---
+
+
+---
+

content/docs/references/data/index.mdx

@@ -18,6 +18,7 @@ This section contains all protocol schemas for the data layer of ObjectStack.
   <Card href="/docs/references/data/external-lookup" title="External Lookup" description="Source: packages/spec/src/data/external-lookup.zod.ts" />
   <Card href="/docs/references/data/feed" title="Feed" description="Source: packages/spec/src/data/feed.zod.ts" />
   <Card href="/docs/references/data/field" title="Field" description="Source: packages/spec/src/data/field.zod.ts" />
+  <Card href="/docs/references/data/field-value" title="Field Value" description="Source: packages/spec/src/data/field-value.zod.ts" />
   <Card href="/docs/references/data/filter" title="Filter" description="Source: packages/spec/src/data/filter.zod.ts" />
   <Card href="/docs/references/data/hook" title="Hook" description="Source: packages/spec/src/data/hook.zod.ts" />
   <Card href="/docs/references/data/hook-body" title="Hook Body" description="Source: packages/spec/src/data/hook-body.zod.ts" />

content/docs/references/data/meta.json

@@ -25,6 +25,8 @@
     "document",
     "feed",
     "seed",
-    "seed-loader"
+    "seed-loader",
+    "---More---",
+    "field-value"
   ]
 }

packages/objectql/src/validation/record-validator.test.ts

@@ -345,3 +345,63 @@ describe('validateRecord — url field accepts relative + inline URLs', () => {
     expect(() => validateRecord(schema, { image: 'notaurl' }, 'update')).toThrow(/valid URL/i);
   });
 });
+
+/**
+ * ADR-0104 D1 — value-shape contract for previously-opaque types.
+ *
+ * Warn-first rollout: a shape violation on reference/file/structured-JSON
+ * types logs (once per field) and passes; `OS_DATA_VALUE_SHAPE_STRICT_ENABLED=1`
+ * turns it into a normal invalid_type rejection.
+ */
+describe('validateRecord — ADR-0104 value shapes (warn-first / strict)', () => {
+  const schema = {
+    fields: {
+      account: { type: 'lookup', reference: 'accounts' },
+      geo: { type: 'location' },
+      doc: { type: 'file' },
+      dims: { type: 'vector' },
+    },
+  };
+
+  const withStrict = (fn: () => void) => {
+    process.env.OS_DATA_VALUE_SHAPE_STRICT_ENABLED = '1';
+    try { fn(); } finally { delete process.env.OS_DATA_VALUE_SHAPE_STRICT_ENABLED; }
+  };
+
+  it('accepts contract-conformant values in both modes', () => {
+    const data = {
+      account: 'acc_0001',
+      geo: { lat: 37.77, lng: -122.42 },
+      doc: { url: 'https://cdn/f.pdf', name: 'f.pdf', size: 1024 },
+      dims: [0.1, 0.2],
+    };
+    expect(() => validateRecord(schema, { ...data }, 'update')).not.toThrow();
+    withStrict(() => expect(() => validateRecord(schema, { ...data }, 'update')).not.toThrow());
+  });
+
+  it('warn-first: malformed shapes pass by default (legacy rows must not strand records)', () => {
+    expect(() => validateRecord(schema, { geo: { latitude: 1, longitude: 2 } }, 'update')).not.toThrow();
+    expect(() => validateRecord(schema, { account: { id: 'acc_1' } }, 'update')).not.toThrow();
+  });
+
+  it('strict: malformed shapes reject with invalid_type', () => {
+    withStrict(() => {
+      try {
+        validateRecord(schema, { geo: { latitude: 37.77, longitude: -122.42 } }, 'update');
+        expect.unreachable('expected ValidationError');
+      } catch (e) {
+        expect(e).toBeInstanceOf(ValidationError);
+        const err = e as ValidationError;
+        expect(err.fields[0]?.field).toBe('geo');
+        expect(err.fields[0]?.code).toBe('invalid_type');
+      }
+      // expanded-form object at a stored-form position (unexpanded write)
+      expect(() => validateRecord(schema, { account: { id: 'acc_1', name: 'Acme' } }, 'update')).toThrow(ValidationError);
+      // scalar at a vector
+      expect(() => validateRecord(schema, { dims: 'not-a-vector' }, 'update')).toThrow(ValidationError);
+      // file: id/url string AND inline object both remain legal pre-D3
+      expect(() => validateRecord(schema, { doc: 'file_01HXYZ' }, 'update')).not.toThrow();
+      expect(() => validateRecord(schema, { doc: 42 }, 'update')).toThrow(ValidationError);
+    });
+  });
+});

packages/objectql/src/validation/record-validator.ts

@@ -31,6 +31,14 @@
  * the UI can highlight the specific input.
  */
 
+import {
+  isMultiValueField as specIsMultiValueField,
+  valueSchemaFor,
+  REFERENCE_VALUE_TYPES,
+  FILE_REFERENCE_TYPES,
+  STRUCTURED_JSON_TYPES,
+} from '@objectstack/spec/data';
+
 // Lifecycle columns the engine always owns and the client never supplies. These
 // are skipped by NAME because they are not author-declared business fields.
 // NOTE: `organization_id` / `tenant_id` are intentionally NOT here (#1592) — the
@@ -145,17 +153,11 @@ function optionValues(options: FieldDef['options']): string[] {
 /**
  * A field whose persisted value is an ARRAY of scalars: either an
  * inherently-multi type, or a single-value type flagged `multiple: true`.
- * Per the spec (field.zod.ts), `multiple` applies to select/lookup/file/image;
- * `radio` shares the select branch and `user` is stored identically to
- * `lookup` (FK column, `multiple` ⇒ JSON array) — the runtime expands
- * `Field.user` with `type: 'user'`, so it must be recognized here too.
+ * THE definition is the spec's (ADR-0104 D1) — previously a hand-copy here
+ * and in rest/import-coerce that had to be kept in lock-step manually.
  */
-const MULTI_CAPABLE_TYPES = new Set(['select', 'radio', 'lookup', 'user', 'file', 'image']);
-
 function isMultiValueField(def: FieldDef): boolean {
-  const t = def.type;
-  if (t === 'multiselect' || t === 'checkboxes' || t === 'tags') return true;
-  return MULTI_CAPABLE_TYPES.has(t as string) && def.multiple === true;
+  return specIsMultiValueField(def as { type: string; multiple?: boolean });
 }
 
 /**
@@ -223,12 +225,12 @@ export function coerceBooleanFields<T extends Record<string, unknown>>(
   return (copy ?? row) as T;
 }
 
-function validateOne(name: string, def: FieldDef, value: unknown): FieldValidationError | null {
+function validateOne(name: string, def: FieldDef, value: unknown, skipRequired = false): FieldValidationError | null {
   // ── required ────────────────────────────────────────────────────
   // `autonumber` is runtime-owned: the value is generated by the engine /
   // driver (the SQL driver assigns it from a persistent sequence AFTER this
   // validation runs), so a missing value is never a client error — see #1603.
-  if (def.required && isMissing(value) && def.type !== 'autonumber') {
+  if (!skipRequired && def.required && isMissing(value) && def.type !== 'autonumber') {
     return { field: name, code: 'required', message: `${name} is required` };
   }
   if (isMissing(value)) return null; // nothing else to check
@@ -338,12 +340,62 @@ function validateOne(name: string, def: FieldDef, value: unknown): FieldValidati
     return null;
   }
 
-  // Other types (lookup, file, formula, json, location, etc.) — no
-  // strict shape check at this layer; reference integrity is handled
-  // elsewhere (lookup) and the rest are opaque payloads.
+  // ── previously-opaque types: value-shape contract (ADR-0104 D1) ─────
+  // Single-value references (id string), file-likes (id/url string or the
+  // pre-D3 inline metadata object), and structured JSON payloads
+  // (location/address/composite/repeater/record/vector) are checked against
+  // the spec's `valueSchemaFor`. Warn-first rollout (ADR-0104 R1/R2): a
+  // violation logs instead of rejecting, so legacy rows written under the
+  // lax regime don't strand their records; set
+  // `OS_DATA_VALUE_SHAPE_STRICT_ENABLED=1` to enforce as a 400. The flip to
+  // error-by-default rides a later minor once telemetry is quiet.
+  if (REFERENCE_VALUE_TYPES.has(t) || FILE_REFERENCE_TYPES.has(t) || STRUCTURED_JSON_TYPES.has(t)) {
+    const parsed = shapeSchemaFor(def).safeParse(value);
+    if (!parsed.success) {
+      const detail = parsed.error.issues[0]?.message ?? 'invalid value shape';
+      const message = `${name} has an invalid ${t} value: ${detail}`;
+      if (VALUE_SHAPE_STRICT()) {
+        return { field: name, code: 'invalid_type', message };
+      }
+      warnOnce(`${t}:${name}`, `[value-shape] ${message} — accepted for now (ADR-0104 warn-first; set OS_DATA_VALUE_SHAPE_STRICT_ENABLED=1 to enforce)`);
+    }
+    return null;
+  }
+
+  // Remaining types (formula/summary/autonumber outputs, json/code payloads)
+  // are explicitly open per the spec contract — see field-value.zod.ts.
   return null;
 }
 
+/** Strict value-shape enforcement opt-in (ADR-0104 warn-first rollout). */
+function VALUE_SHAPE_STRICT(): boolean {
+  return typeof process !== 'undefined' && process.env?.OS_DATA_VALUE_SHAPE_STRICT_ENABLED === '1';
+}
+
+const warnedShapes = new Set<string>();
+function warnOnce(key: string, message: string): void {
+  if (warnedShapes.has(key)) return;
+  warnedShapes.add(key);
+  console.warn(message);
+}
+
+/**
+ * Per-field-definition cache of the spec's derived value schema. Building a
+ * Zod schema is an order of magnitude costlier than parsing with it, so the
+ * derivation runs once per field def (ADR-0104 performance budget). Keyed on
+ * def identity — `validateRecord`'s update path passes the registry's own
+ * field objects (no clones), so the map hits.
+ */
+const shapeSchemaCache = new WeakMap<FieldDef, ReturnType<typeof valueSchemaFor>>();
+function shapeSchemaFor(def: FieldDef): ReturnType<typeof valueSchemaFor> {
+  let schema = shapeSchemaCache.get(def);
+  if (!schema) {
+    schema = valueSchemaFor(def as { type: string; multiple?: boolean; options?: FieldDef['options'] }, 'stored');
+    shapeSchemaCache.set(def, schema);
+  }
+  return schema;
+}
+
 /**
  * Validate a payload against a list of declared fields. `objectSchema`
  * comes from `ObjectQL.getRegistry().getObject(name)` and exposes a
@@ -377,8 +429,10 @@ export function validateRecord(
       const def = fields[name];
       if (!def) continue;
       if (def.system || def.readonly) continue;
-      // Clone def with required=false so PATCH-omitted-fields don't 400.
-      const err = validateOne(name, { ...def, required: false }, value);
+      // skipRequired: PATCH-omitted fields must not 400. (No def clone — the
+      // registry's own field object flows through so the ADR-0104 value-shape
+      // schema cache, keyed on def identity, hits.)
+      const err = validateOne(name, def, value, true);
       if (err) errors.push(err);
     }
   }