fix(spec,service-analytics): the matrix's token axis + its sixth consumer, and the dropped $between it found (ADR-0053 D-A3.1)

.changeset/analytics-between-predicate-dropped.md

@@ -0,0 +1,29 @@
+---
+"@objectstack/service-analytics": patch
+---
+
+fix(service-analytics): a `$between` analytics filter no longer vanishes from the query (ADR-0053 D-A3.1)
+
+A dashboard widget or dataset whose filter used `$between` was querying **every
+row**. `normalizeAnalyticsFilters` maps Mongo-style operators onto the internal
+pipeline form, `$between` was missing from that map, and an unmapped operator is
+skipped — so the predicate was silently dropped from the compiled WHERE clause.
+Both strategies read that normalizer, so both the raw-SQL and the ObjectQL
+aggregate paths were affected. The symptom is #3650's: a chart that draws the
+whole dataset instead of the requested window, with nothing in the SQL to
+suggest a filter was ever asked for.
+
+`$between [min, max]` now lowers to its two bounds (`gte` + `lte`) instead of
+gaining an operator of its own, so a range's max inherits the calendar-day
+whole-day rule (#3777) from each strategy's existing upper-bound handling —
+`NativeSQLStrategy` compiles a bare-day upper bound half-open itself, and the
+ObjectQL path gets the same rule from the driver — rather than needing a second
+implementation to keep in step. A malformed `$between` (not a two-element
+array) now throws instead of being dropped, matching the stance driver-memory
+took for the same shape in #3948: an unbounded read is exactly the failure this
+prevents, and it is indistinguishable from a legitimately wide query.
+
+Found by giving the temporal conformance matrix its missing sixth consumer
+(`native-sql-temporal-conformance.test.ts`), which executes the shared cases
+against a real SQLite engine and asserts row ids — a dropped predicate is
+invisible to the SQL-string assertions the strategy's other suites use.

.changeset/analytics-filter-operator-coverage.md

@@ -0,0 +1,42 @@
+---
+"@objectstack/service-analytics": patch
+---
+
+fix(service-analytics): every authorable filter operator now reaches the query (#4128)
+
+Closes the cause behind the `$between` defect rather than just that instance.
+`normalizeAnalyticsFilters` skipped any operator missing from its map, and a
+skipped predicate does not narrow a query — it **widens** it: the compiled SQL
+stays valid and returns rows the author excluded. Four operators from the
+spec's authorable vocabulary sat in that state, plus one that was mapped
+incorrectly.
+
+- **`$startsWith` / `$endsWith`** were dropped entirely. Both strategies now
+  compile them — anchored `LIKE 'x%'` / `LIKE '%x'` on the raw-SQL path, and
+  the canonical `$startsWith` / `$endsWith` operators (which every driver
+  implements directly) on the ObjectQL path, so an anchored match does not
+  depend on regex dialect.
+- **`$null`** was dropped. It is the shape the console emits for an "is empty"
+  / "is not empty" filter, so such a widget was showing every row. Now compiles
+  to `IS NULL` / `IS NOT NULL` per its boolean.
+- **`$exists`** was mapped value-*independently* to `set`, so `{$exists: false}`
+  compiled to `IS NOT NULL` — the exact inverse of what it asks for. It and
+  `$null` are now resolved explicitly, because a key→name map cannot express an
+  operator whose meaning flips with its value.
+- **`$notContains`** reached the ObjectQL strategy, which had no arm for it and
+  fell through to a `default` returning a bare value — compiling "does not
+  contain x" as "**equals** x".
+- **Unknown operators now throw** on both surfaces instead of being silently
+  dropped (normalizer) or reinterpreted as an equality (ObjectQL strategy). An
+  operator outside the vocabulary is a caller error, and a loud one beats a
+  silently widened read — the call driver-memory made for the same shape in
+  #3948.
+
+Still declared as a gap, but no longer a silent one: `$or` / `$not` are skipped,
+since expressing them needs a recursive WHERE builder rather than the flat
+array the strategies consume.
+
+Cover is `filter-operator-coverage.test.ts`, which runs the whole vocabulary
+against a real SQLite engine and asserts **row ids** — six of its cases fail
+without this change. A dropped predicate is invisible to the SQL-string
+assertions the strategies' other suites use, which is how these survived.

.changeset/temporal-conformance-token-axis.md

@@ -0,0 +1,26 @@
+---
+"@objectstack/spec": minor
+---
+
+feat(spec): the temporal conformance matrix gains its relative-token axis (ADR-0053 D-A3, #4081)
+
+The matrix (#4098) landed with the token axis documented as still open — the
+cases took resolved comparands, while authors actually write `{today}` /
+`{90_days_ago}` / `{current_month_end}`, and nothing proved the resolved
+token reaches the same rows on every backend. Closed here:
+
+- `TemporalCase` gains `tokenFilter?` (the same filter spelled in tokens) and
+  `dateRange?` (the analytics `timeDimensions.dateRange` spelling of the same
+  window — the surface #3650 broke), plus the pinned instant `TEMPORAL_NOW`
+  consumers hand to `resolveFilterTokens`. A resolver drift and an evaluator
+  drift are now distinguishable at a glance: the literal spelling fails for
+  one, the token spelling for the other.
+- `TemporalRow` gains `writerForm` (`wire` ISO string vs `native` `Date`), so
+  driver consumers seed genuinely mixed writer populations through their own
+  `create()` — the exact column shape that produced #4047 (D-E4).
+- New rows/cells: a pre-epoch row (negative epoch ms, the #3773 family), and
+  the `date` equality/`$in` cases — #1874's original `date == today` shape.
+- Two more sweeps consume the table: a `driver-sqlite-wasm` conformance test
+  pinning the inherited SqlDriver seam, and a legacy-storage sweep in
+  `driver-sql` running every case over an un-backfilled mixed epoch/naive
+  column through the read-repair path.

docs/adr/0053-date-and-datetime-semantics.md

@@ -857,6 +857,25 @@ Two things, which is the argument for having built it:
    exists to provide. Fixed in the same change, sharing the `$lte` bound helper
    so the two cannot drift again.
 
+   **The same defect had a twin one layer over, found the same way when the
+   sixth consumer was added (#4081 follow-up).** `NativeSQLStrategy` — the
+   surface #3650 was actually about — was listed in the matrix's own backend
+   table but had no consumer, because every existing suite for it asserts the
+   emitted SQL string, and a *dropped* predicate is invisible to a string
+   assertion: the SQL stays valid, just wider. Executing the matrix against a
+   real engine showed `$between` returning the **entire table**. The cause was
+   one layer below the strategy, in the shared `filter-normalizer`: `$between`
+   was absent from `MONGO_TO_CUBE_OP`, and an unmapped operator is `continue`d,
+   so the predicate vanished from the WHERE clause. Both raw-SQL and ObjectQL
+   strategies read that normalizer, so both were affected. `$between` now
+   **lowers to its two bounds** (`gte` + `lte`) rather than gaining an operator
+   of its own: each strategy's upper-bound arm already carries the whole-day
+   calendar rule (`NativeSQLStrategy` compiles half-open directly, the ObjectQL
+   path inherits it from the driver), so a range's max gets that rule by
+   construction instead of via a second implementation. A malformed `$between`
+   now throws, the anti-silent-widening stance driver-memory took for the same
+   shape in #3948.
+
 2. **A measured, irreducible limit.** `$gt` with a bare-day comparand on a
    `datetime` column cannot agree across backends. A typed backend anchors the
    bound to midnight and excludes a value stored at exactly 00:00; a type-blind
@@ -876,7 +895,19 @@ Two things, which is the argument for having built it:
 
 - The matrix is the ratchet the previous four fixes lacked: a backend that
   drifts now fails a named case whose note says which incident it is repeating.
-- Coverage still open, and deliberately so: the **relative-token** axis
+- ~~Coverage still open, and deliberately so: the **relative-token** axis
   (`{today}`, `{30_days_ago}` resolved through `filter-tokens.ts` and run
   end-to-end per driver) is not yet in the table. The cases here take resolved
-  comparands, which is the layer where the four incidents actually happened.
+  comparands, which is the layer where the four incidents actually happened.~~
+  Closed in the #4081 follow-up: cases now carry a `tokenFilter` (and, for the
+  analytics window path, a `dateRange`) spelling — the same filter written in
+  `{today}` / `{90_days_ago}` / period tokens — which every consumer that can
+  reach `@objectstack/core` resolves against the pinned `TEMPORAL_NOW` and
+  must land on the same row ids as the literal spelling. `formula` alone skips
+  the token sweep (its dependencies deliberately stop at `spec`; tokens are
+  resolved before a filter reaches it). The same follow-up added the
+  `writerForm` seeding hint (D-E4's mixed-writer populations through each
+  driver's own `create()`), a pre-epoch row, the `date` equality/`$in` cells
+  (#1874's original shape), a `driver-sqlite-wasm` consumer pinning the
+  inheritance seam, and a legacy-storage sweep in the `driver-sql` consumer so
+  the un-backfilled read-repair path answers the same table.

packages/plugins/driver-memory/src/memory-temporal-conformance.test.ts

@@ -8,12 +8,22 @@
  * evaluator are all held to one standard — see `temporal-conformance.ts` for
  * the four divergences that standard exists to prevent, one of which (#4047)
  * was this driver's.
+ *
+ * Rows are seeded in their tagged writer forms (ISO string vs JS `Date` — the
+ * D-E4 mixed-writer axis), reproducing exactly the mixed column #4047 hit;
+ * the sweep proves the converged storage answers every shared case. Cases
+ * carrying a token spelling also run through `resolveFilterTokens` at the
+ * pinned `TEMPORAL_NOW` — the D-A3 "token → row results" axis (#4081).
  */
 
 import { describe, it, expect, beforeAll } from 'vitest';
-import { TEMPORAL_CASES, TEMPORAL_ROWS } from '@objectstack/spec/data';
+import { TEMPORAL_CASES, TEMPORAL_NOW, TEMPORAL_ROWS } from '@objectstack/spec/data';
+import { resolveFilterTokens } from '@objectstack/core';
 import { InMemoryDriver } from './memory-driver.js';
 
+const resolveTokens = <T,>(filter: T): T =>
+  resolveFilterTokens(filter, { now: new Date(TEMPORAL_NOW) });
+
 describe('driver-memory — temporal conformance', () => {
   let driver: InMemoryDriver;
 
@@ -27,7 +37,13 @@ describe('driver-memory — temporal conformance', () => {
       fields: { at: { type: 'datetime' }, on: { type: 'date' }, why: { type: 'string' } },
     });
     for (const r of TEMPORAL_ROWS) {
-      await driver.create('conformance', { id: r.id, at: r.at, on: r.on, why: r.why });
+      await driver.create('conformance', {
+        id: r.id,
+        // The mixed-writer axis (D-E4): both shapes must converge on write.
+        at: r.writerForm === 'native' ? new Date(r.at) : r.at,
+        on: r.on,
+        why: r.why,
+      });
     }
   });
 
@@ -37,5 +53,13 @@ describe('driver-memory — temporal conformance', () => {
       const got = (rows as any[]).map((r) => r.id).sort();
       expect(got, c.note).toEqual([...c.expected].sort());
     });
+
+    if (c.tokenFilter) {
+      it(`${c.name} — via relative tokens`, async () => {
+        const rows = await driver.find('conformance', { where: resolveTokens(c.tokenFilter) } as any);
+        const got = (rows as any[]).map((r) => r.id).sort();
+        expect(got, c.note).toEqual([...c.expected].sort());
+      });
+    }
   }
 });

packages/plugins/driver-mongodb/src/mongodb-temporal-conformance.test.ts

@@ -11,13 +11,22 @@
  * was this driver's and was the worst of them: type-bracket comparison meant a
  * string bound matched no BSON `Date` row at all, so the default dashboard
  * window returned nothing.
+ *
+ * Rows are seeded in their tagged writer forms (ISO string vs JS `Date` — the
+ * D-E4 mixed-writer axis), the exact writer population #4047 hit. Cases
+ * carrying a token spelling also run through `resolveFilterTokens` at the
+ * pinned `TEMPORAL_NOW` — the D-A3 "token → row results" axis (#4081).
  */
 
 import { describe, it, expect, beforeAll, afterAll } from 'vitest';
 import { MongoMemoryServer } from 'mongodb-memory-server';
-import { TEMPORAL_CASES, TEMPORAL_ROWS } from '@objectstack/spec/data';
+import { TEMPORAL_CASES, TEMPORAL_NOW, TEMPORAL_ROWS } from '@objectstack/spec/data';
+import { resolveFilterTokens } from '@objectstack/core';
 import { MongoDBDriver } from './mongodb-driver.js';
 
+const resolveTokens = <T,>(filter: T): T =>
+  resolveFilterTokens(filter, { now: new Date(TEMPORAL_NOW) });
+
 let sharedMongod: MongoMemoryServer | undefined;
 try {
   sharedMongod = await MongoMemoryServer.create({ instance: { launchTimeout: 60_000 } });
@@ -42,7 +51,13 @@ describe.skipIf(!sharedMongod)('driver-mongodb — temporal conformance', () =>
       fields: { at: { type: 'datetime' }, on: { type: 'date' }, why: { type: 'string' } },
     });
     for (const r of TEMPORAL_ROWS) {
-      await driver.create('conformance', { id: r.id, at: r.at, on: r.on, why: r.why });
+      await driver.create('conformance', {
+        id: r.id,
+        // The mixed-writer axis (D-E4): the exact population #4047 hit.
+        at: r.writerForm === 'native' ? new Date(r.at) : r.at,
+        on: r.on,
+        why: r.why,
+      });
     }
   }, 90_000);
 
@@ -57,5 +72,13 @@ describe.skipIf(!sharedMongod)('driver-mongodb — temporal conformance', () =>
       const got = (rows as any[]).map((r) => r.id).sort();
       expect(got, c.note).toEqual([...c.expected].sort());
     });
+
+    if (c.tokenFilter) {
+      it(`${c.name} — via relative tokens`, async () => {
+        const rows = await driver.find('conformance', { where: resolveTokens(c.tokenFilter) } as any);
+        const got = (rows as any[]).map((r) => r.id).sort();
+        expect(got, c.note).toEqual([...c.expected].sort());
+      });
+    }
   }
 });

packages/plugins/driver-sql/src/sql-driver-temporal-conformance.test.ts

@@ -14,15 +14,40 @@
  * `Field.date` and coerces comparands accordingly. That the same case yields the
  * same row set here and in the type-blind backends is the whole assertion.
  *
+ * Three sweeps, because this driver owns two extra axes (#4081):
+ *   1. canonical storage — rows seeded through `create()` in their tagged
+ *      writer forms (ISO string vs JS `Date`, the D-E4 mixed-writer axis),
+ *      which `formatInput` must converge;
+ *   2. the same cases spelled in relative tokens, resolved through
+ *      `@objectstack/core`'s `resolveFilterTokens` at the pinned `TEMPORAL_NOW`
+ *      — the D-A3 "token → row results" axis;
+ *   3. legacy storage — the same rows seeded RAW as pre-#3912 forms (INTEGER
+ *      epoch ms / zone-naive TEXT) with the canonical marker cleared, so the
+ *      read-repair path answers the same table.
+ *
  * Runs against a real SQLite, and — through the `Temporal Conformance
  * (live PG + MySQL)` CI job, which runs this package's whole suite under
  * `TZ=America/New_York` against servers on `Asia/Shanghai` — against real
  * Postgres and MySQL too, with no workflow change needed.
  */
 
 import { describe, it, expect, beforeAll, afterAll } from 'vitest';
-import { TEMPORAL_CASES, TEMPORAL_ROWS } from '@objectstack/spec/data';
+import { TEMPORAL_CASES, TEMPORAL_NOW, TEMPORAL_ROWS } from '@objectstack/spec/data';
+import { resolveFilterTokens } from '@objectstack/core';
 import { SqlDriver } from '../src/index.js';
+import { LegacyStorageDriver } from './legacy-datetime-storage.testkit.js';
+
+const resolveTokens = <T,>(filter: T): T =>
+  resolveFilterTokens(filter, { now: new Date(TEMPORAL_NOW) });
+
+const CONFORMANCE_OBJECT = {
+  name: 'conformance',
+  fields: {
+    at: { type: 'datetime' },
+    on: { type: 'date' },
+    why: { type: 'string' },
+  },
+};
 
 describe('sql-driver — temporal conformance', () => {
   let driver: SqlDriver;
@@ -35,20 +60,18 @@ describe('sql-driver — temporal conformance', () => {
     });
     // The declaration is what makes this the typed half — `at` is an instant,
     // `on` a calendar day, and the driver's coercion follows from that.
-    await driver.initObjects([
-      {
-        name: 'conformance',
-        fields: {
-          at: { type: 'datetime' },
-          on: { type: 'date' },
-          why: { type: 'string' },
-        },
-      },
-    ]);
+    await driver.initObjects([CONFORMANCE_OBJECT]);
     for (const r of TEMPORAL_ROWS) {
       await driver.create(
         'conformance',
-        { id: r.id, at: r.at, on: r.on, why: r.why },
+        {
+          id: r.id,
+          // The writer-form axis (D-E4): both writer populations must
+          // converge to the one canonical storage form on write.
+          at: r.writerForm === 'native' ? new Date(r.at) : r.at,
+          on: r.on,
+          why: r.why,
+        },
         { bypassTenantAudit: true } as any,
       );
     }
@@ -58,6 +81,57 @@ describe('sql-driver — temporal conformance', () => {
     await driver.disconnect?.();
   });
 
+  for (const c of TEMPORAL_CASES) {
+    it(c.name, async () => {
+      const rows = await driver.find('conformance', { where: c.filter } as any);
+      const got = (rows as any[]).map((r) => r.id).sort();
+      expect(got, c.note).toEqual([...c.expected].sort());
+    });
+
+    if (c.tokenFilter) {
+      it(`${c.name} — via relative tokens`, async () => {
+        const rows = await driver.find('conformance', { where: resolveTokens(c.tokenFilter) } as any);
+        const got = (rows as any[]).map((r) => r.id).sort();
+        expect(got, c.note).toEqual([...c.expected].sort());
+      });
+    }
+  }
+});
+
+describe('sql-driver — temporal conformance on un-backfilled legacy storage', () => {
+  let driver: LegacyStorageDriver;
+
+  beforeAll(async () => {
+    driver = new LegacyStorageDriver({
+      client: 'better-sqlite3',
+      connection: { filename: ':memory:' },
+      useNullAsDefault: true,
+    });
+    await driver.initObjects([CONFORMANCE_OBJECT]);
+    // The two pre-#3912 storage forms, split by the same writer-form tag:
+    // `native` writes landed as INTEGER epoch ms (a bound JS Date), `wire`
+    // writes as zone-naive TEXT (CURRENT_TIMESTAMP / REST payloads). One
+    // column, both forms — the read-repair path must answer the same table
+    // the canonical sweep does. (`on` is unaffected: bare-day text has been
+    // the date canon since Phase 1.)
+    await driver.seedLegacyRows(
+      'conformance',
+      'at',
+      TEMPORAL_ROWS.map((r) => ({
+        id: r.id,
+        at: r.writerForm === 'native' ? Date.parse(r.at) : r.at.replace('T', ' ').replace('Z', ''),
+        on: r.on,
+        why: r.why,
+      })),
+    );
+  });
+
+  afterAll(async () => {
+    await driver.disconnect?.();
+  });
+
+  // Literal spellings only: the token axis is orthogonal to storage form and
+  // already swept above — a divergence here is a repair-path bug by construction.
   for (const c of TEMPORAL_CASES) {
     it(c.name, async () => {
       const rows = await driver.find('conformance', { where: c.filter } as any);