fix(driver-memory,driver-mongodb): one storage form per temporal field type (#4047)

.changeset/datetime-storage-form-memory-mongodb.md

@@ -0,0 +1,40 @@
+---
+"@objectstack/driver-memory": patch
+"@objectstack/driver-mongodb": patch
+---
+
+fix(driver-memory,driver-mongodb): `Field.datetime` has one storage form per driver (#4047)
+
+The non-SQL counterpart of ADR-0053 D-B (#3912). Both drivers let the writer
+decide a datetime value's runtime type, and both compare across types by type
+bracket rather than by value — so a string comparand never matched a `Date`
+value, in either direction, for **every** operator including `$gte`.
+
+A datetime column genuinely held both forms: the drivers' own
+`created_at`/`updated_at` defaults bind a `Date` (mongo) or an ISO string
+(memory), while REST/JSON writes, relative-date tokens and `initialData`
+fixtures supply the other. A dashboard date window therefore answered with
+whichever half happened to match the comparand's type — on MongoDB, where
+`created_at` is a BSON `Date` and dashboard bounds are strings, that meant
+**no rows at all**, which is worse than the final-day loss #3777 fixed.
+
+Each driver now has one canonical form, applied on write and to every filter
+comparand:
+
+| Driver | `datetime` | `date` |
+|---|---|---|
+| `driver-mongodb` | BSON `Date` — the dialect's native instant, its `timestamptz` | `YYYY-MM-DD` text |
+| `driver-memory` | canonical UTC ISO text (sorts chronologically under the string comparison mingo performs; survives JSON persistence) | `YYYY-MM-DD` text |
+
+Both learn their temporal fields from `syncSchema`, so an object that was never
+declared is left exactly as written — the drivers do not guess types from
+values. `driver-memory` additionally converges rows already in the table when
+the schema arrives, which catches `initialData` fixtures and anything a
+persistence adapter restored (the in-memory analogue of
+`backfillCanonicalDatetimes`, and idempotent like it).
+
+`Field.date` deliberately stays timezone-naive text on both — converting it to
+an instant would invent a midnight and re-couple it to a zone. The
+calendar-day bound semantics from #3777/#4042 are unchanged and now compose
+with the converged storage: the whole-day rewrite runs on the calendar string
+first, and only the resulting bound is converted to the storage form.

docs/adr/0053-date-and-datetime-semantics.md

@@ -717,3 +717,73 @@ row-result coverage for the `$lte`/`$between` upper-bound cells now lives in
 storage, dialect physical forms, boundary rollovers) and the strategy/preview
 suites; the full matrix program (relative-token × live-driver × timezone)
 remains open under D-A3.
+
+---
+
+## Addendum (2026-07-30) — the non-SQL drivers get one storage form too (#4047)
+
+> **Status:** landed. Extends the 2026-07-29 addendum (D-B, `Field.datetime`
+> has ONE storage form per SQL dialect) to `driver-memory` and
+> `driver-mongodb`, which had the same defect for the same reason and one worse
+> symptom.
+
+### D-E1 — Type-bracket comparison makes a mixed column WORSE than it is on SQL
+
+MongoDB compares across BSON types by type bracket, and mingo copies that rule
+for JS types: a string comparand never matches a `Date` value, in either
+direction, **for every operator** — `$gte` included, not just the upper bound
+#3777 was about. SQLite's affinity rules at least left one half reachable; here
+a window silently answers with whichever half matches the comparand's type.
+
+And both columns really were mixed. Each driver's own `created_at`/`updated_at`
+default writes one form (`new Date()` on mongo, `new Date().toISOString()` in
+memory) while REST/JSON writes, relative-date tokens and `initialData` fixtures
+supply the other. On MongoDB the dashboard's default window — string bounds
+against a BSON-`Date` `created_at` — therefore returned **nothing at all**.
+
+### D-E2 — One canon per driver, chosen by what the store natively has
+
+| Driver | `datetime` | Why |
+|---|---|---|
+| `driver-mongodb` | BSON `Date` | the dialect's native instant: indexable, range-queryable, timezone-unambiguous. The `timestamptz` of this driver. |
+| `driver-memory` | canonical UTC ISO text | no native instant type exists; ISO-8601 UTC sorts chronologically under the plain string comparison mingo performs, and it is the wire form, so it survives JSON persistence unchanged. |
+
+`Field.date` stays timezone-naive `YYYY-MM-DD` text on both, for the Phase 1
+reason: an instant would invent a midnight and re-couple the value to a zone.
+
+Both drivers learn their temporal fields from `syncSchema` — the only place a
+driver is handed an object definition — into the direct equivalent of
+`SqlDriver.datetimeFields`/`dateFields`. An **undeclared** object is left
+exactly as written: these drivers are schemaless stores, and guessing a type
+from a value would coerce data the platform never claimed to own.
+
+### D-E3 — Existing rows, and how this composes with the bound semantics
+
+`driver-memory` converges the rows already in a table when the schema arrives,
+which is what catches `initialData` fixtures and persistence-adapter restores
+(both land before any schema is declared). It is the in-memory analogue of
+`backfillCanonicalDatetimes` and idempotent like it. MongoDB gets no automatic
+backfill — rewriting a real collection is a migration, not a boot step — so a
+pre-existing collection keeps whatever it holds until migrated; new and updated
+documents converge from the first write.
+
+Ordering is load-bearing where D-D meets D-E: the calendar-day upper-bound
+rewrite (#3777/#4042) is a *calendar* operation and runs on the bare-day
+STRING first; only the resulting bound is converted to the storage form.
+Converting first would hand `nextUtcCalendarDay` a `Date`, which it correctly
+refuses to widen — and the whole-day window would silently narrow back to an
+instant.
+
+### D-E4 — Consequences for D-A3
+
+The conformance matrix's **driver** axis now has non-SQL cells worth asserting,
+and its **storage-form** axis gains `mixed-writer-form` for them. Row-result
+cover lives in `mongodb-datetime-storage.test.ts` (against a real MongoDB via
+`mongodb-memory-server`) and `memory-datetime-storage.test.ts`. Still open
+under D-A3: the relative-token × live-driver × timezone combinations.
+
+One evaluator remains unaligned on both the D-D and D-E rules:
+`@objectstack/formula`'s `matchesFilterCondition` (the RLS write-side `check`
+path), which cannot depend on `@objectstack/core` for the shared primitive.
+Deciding its home — a private copy, as `formula` already keeps for `today()`,
+or lowering the utility into `spec`/`types` — is the remaining piece.

packages/plugins/driver-memory/src/memory-datetime-storage.test.ts

@@ -0,0 +1,183 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * `Field.datetime` has ONE storage form in the in-memory driver — canonical
+ * UTC ISO text (#4047), the memory twin of ADR-0053 D-B (#3912).
+ *
+ * mingo compares across JS types the way MongoDB compares across BSON types:
+ * a string comparand never matches a `Date` value, in either direction, for
+ * EVERY operator — `$gte` included. And a datetime column really did hold
+ * both: the driver's own `created_at`/`updated_at` defaults write
+ * `new Date().toISOString()` (string) while `initialData` fixtures and direct
+ * SDK callers hand it `Date` objects. So a date window answered with whichever
+ * half matched the comparand's type and silently dropped the other.
+ *
+ * Canonical ISO TEXT is the right canon here — unlike mongo, this store has no
+ * native instant type, and ISO-8601 UTC text sorts chronologically under plain
+ * string comparison, which is exactly what mingo does. It is also the wire
+ * form, so a value round-trips through JSON persistence unchanged.
+ */
+
+import { describe, it, expect, beforeEach } from 'vitest';
+import { InMemoryDriver } from './memory-driver.js';
+
+const ids = (rows: any[]) => rows.map((r: any) => r.id).sort();
+
+/** The object declaration is what teaches the driver which fields are temporal. */
+const TASK_SCHEMA = {
+  name: 'task',
+  fields: {
+    title: { type: 'string' },
+    created_at: { type: 'datetime' },
+    due_at: { type: 'datetime' },
+    created_on: { type: 'date' },
+  },
+};
+
+describe('InMemoryDriver Field.datetime storage (#4047)', () => {
+  let driver: InMemoryDriver;
+
+  beforeEach(async () => {
+    driver = new InMemoryDriver({});
+    await driver.connect();
+    await driver.syncSchema('task', TASK_SCHEMA);
+  });
+
+  /** Both writer forms of the same instant, in one column. */
+  async function seedMixed(): Promise<void> {
+    const rows: Array<[string, unknown]> = [
+      ['s_morning', '2026-07-28T09:15:00.000Z'],
+      ['s_evening', '2026-07-28T21:40:00.000Z'],
+      ['d_midnight', new Date('2026-07-28T00:00:00Z')],
+      ['d_yesterday', new Date('2026-07-27T14:00:00Z')],
+      ['d_next_day', new Date('2026-07-29T00:00:00Z')],
+      ['s_old', '2026-04-19T10:00:00.000Z'],
+    ];
+    for (const [id, created_at] of rows) {
+      await driver.create('task', { id, title: id, created_at });
+    }
+  }
+
+  it('stores every writer form as canonical UTC ISO text', async () => {
+    await seedMixed();
+    const raw = await driver.find('task', {} as any);
+    for (const row of raw) {
+      expect(typeof (row as any).created_at, `${(row as any).id} stored form`).toBe('string');
+      expect((row as any).created_at).toMatch(/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/);
+    }
+    const midnight = raw.find((r: any) => r.id === 'd_midnight') as any;
+    expect(midnight.created_at).toBe('2026-07-28T00:00:00.000Z');
+  });
+
+  it('a date window reaches rows written in BOTH forms', async () => {
+    await seedMixed();
+    const found = await driver.find('task', {
+      where: { created_at: { $gte: '2026-04-29', $lte: '2026-07-28' } },
+    } as any);
+    expect(ids(found)).toEqual(['d_midnight', 'd_yesterday', 's_evening', 's_morning']);
+  });
+
+  it('a Date-object comparand reaches the same rows', async () => {
+    await seedMixed();
+    const found = await driver.find('task', {
+      where: {
+        created_at: {
+          $gte: new Date('2026-04-29T00:00:00Z'),
+          $lt: new Date('2026-07-29T00:00:00Z'),
+        },
+      },
+    } as any);
+    expect(ids(found)).toEqual(['d_midnight', 'd_yesterday', 's_evening', 's_morning']);
+  });
+
+  it('keeps #3777/#4042 bound semantics on top of the converged storage', async () => {
+    await seedMixed();
+    const gte = await driver.find('task', { where: { created_at: { $gte: '2026-07-28' } } } as any);
+    expect(ids(gte)).toEqual(['d_midnight', 'd_next_day', 's_evening', 's_morning']);
+
+    const lt = await driver.find('task', { where: { created_at: { $lt: '2026-07-28' } } } as any);
+    expect(ids(lt)).toEqual(['d_yesterday', 's_old']);
+
+    const instant = await driver.find('task', {
+      where: { created_at: { $lte: '2026-07-28T12:00:00.000Z' } },
+    } as any);
+    expect(ids(instant)).toEqual(['d_midnight', 'd_yesterday', 's_morning', 's_old']);
+  });
+
+  it('$between and the array spelling take the same coercion', async () => {
+    await seedMixed();
+    const between = await driver.find('task', {
+      where: { created_at: { $between: ['2026-04-29', '2026-07-28'] } },
+    } as any);
+    expect(ids(between)).toEqual(['d_midnight', 'd_yesterday', 's_evening', 's_morning']);
+
+    const array = await driver.find('task', {
+      where: [['created_at', '>=', '2026-04-29'], 'and', ['created_at', '<=', '2026-07-28']],
+    } as any);
+    expect(ids(array)).toEqual(['d_midnight', 'd_yesterday', 's_evening', 's_morning']);
+  });
+
+  it('normalises rows supplied as initialData, not just ones written through create()', async () => {
+    // A seeded fixture is a WRITE the driver never saw — but it is the shape
+    // most tests and demos use, so leaving it un-normalised would keep the
+    // mixed column alive on exactly the path people look at first.
+    const seeded = new InMemoryDriver({
+      initialData: {
+        task: [
+          { id: 'i_date', created_at: new Date('2026-07-28T09:15:00Z') },
+          { id: 'i_iso', created_at: '2026-07-28T21:40:00.000Z' },
+        ],
+      },
+    });
+    await seeded.connect();
+    await seeded.syncSchema('task', TASK_SCHEMA);
+
+    const found = await seeded.find('task', {
+      where: { created_at: { $gte: '2026-07-28', $lte: '2026-07-28' } },
+    } as any);
+    expect(ids(found)).toEqual(['i_date', 'i_iso']);
+  });
+
+  it('update() puts the new value in the same form', async () => {
+    // `created_at` is deliberately immutable in this driver (it re-preserves
+    // the original on every update), so the mutable datetime field is the
+    // honest subject for an update-path assertion.
+    await driver.create('task', { id: 'u1', due_at: '2026-04-19T10:00:00.000Z' });
+    await driver.update('task', 'u1', { due_at: new Date('2026-07-28T09:15:00Z') });
+    const row: any = (await driver.find('task', { where: { id: 'u1' } } as any))[0];
+    expect(row.due_at).toBe('2026-07-28T09:15:00.000Z');
+
+    // …and the converged value is reachable by a date window, which is the
+    // point of converging it.
+    const found = await driver.find('task', {
+      where: { due_at: { $gte: '2026-07-28', $lte: '2026-07-28' } },
+    } as any);
+    expect(ids(found)).toEqual(['u1']);
+  });
+
+  it('a Field.date column stays calendar-day text — not widened into an instant', async () => {
+    for (const [id, created_on] of [
+      ['on_early', '2026-04-19'],
+      ['on_mid', '2026-07-28'],
+      ['on_late', '2026-07-29'],
+      // A Date written to a `date` field collapses to its UTC calendar day.
+      ['on_obj', new Date('2026-07-28T21:40:00Z')],
+    ] as const) {
+      await driver.create('task', { id, created_on });
+    }
+    const all = await driver.find('task', {} as any);
+    for (const row of all) expect(typeof (row as any).created_on).toBe('string');
+    expect((all.find((r: any) => r.id === 'on_obj') as any).created_on).toBe('2026-07-28');
+
+    const found = await driver.find('task', {
+      where: { created_on: { $gte: '2026-04-29', $lte: '2026-07-28' } },
+    } as any);
+    expect(ids(found)).toEqual(['on_mid', 'on_obj']);
+  });
+
+  it('an undeclared object is left alone (no schema → no coercion)', async () => {
+    await driver.create('freeform', { id: 'f1', when: new Date('2026-07-28T09:15:00Z') });
+    const row: any = (await driver.find('freeform', {} as any))[0];
+    expect(row.when).toBeInstanceOf(Date);
+  });
+});