fix(service-analytics): the matrix's sixth consumer, and the dropped $between it found (ADR-0053 D-A3.1)

NativeSQLStrategy is the surface #3650 was actually about, and it was listed
in the conformance matrix's own backend table with no consumer — because
every existing suite for it asserts the emitted SQL string, and a DROPPED
predicate is invisible to a string assertion: the SQL stays valid, just
wider. Executing the shared cases against a real engine (sql.js, the same
pure-WASM engine driver-sql falls back to) and asserting row ids is what
D-A3 asked for, and it immediately showed `$between` returning the entire
table.

The cause sat one layer below the strategy, in the shared filter-normalizer:
`$between` was absent from MONGO_TO_CUBE_OP, and an unmapped operator is
`continue`d, so the predicate vanished from the compiled WHERE clause. Both
strategies read that normalizer, so the ObjectQL aggregate path was affected
too. User-visible symptom: a widget with a range filter charts the whole
dataset, with nothing in the SQL to suggest a filter was ever requested.

`$between [min, max]` now LOWERS to its two bounds (gte + lte) rather than
gaining an operator of its own, so a range's max inherits the calendar-day
whole-day rule (#3777) from each strategy's existing upper-bound handling —
NativeSQLStrategy compiles a bare-day upper bound half-open itself, the
ObjectQL path gets the same rule from the driver — instead of needing a
second implementation to keep in step. That is how #4098 closed the same
defect on the preview evaluator. A malformed `$between` throws rather than
being dropped, the stance driver-memory took for the same shape in #3948.

The consumer runs the literal, relative-token and timeDimensions.dateRange
spellings of every shared case. Rows are seeded canonical, where the
driver's temporal hooks are identities, so the context omits them and
exercises the same "absent = identity" path Postgres takes (D-A2); the
un-backfilled mixed column stays covered where the driver truth lives.

The cause itself is NOT fully closed: `$startsWith`, `$endsWith`, `$null`
and `$regex` are still silently dropped by the same `continue`. Filed as
#4128 rather than expanded into this change, with the #3948 precedent for
turning the fallback into a throw.

service-analytics 391 green (was 360; +31 from the new consumer).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TqqZmPS5a4gJGBoCTwipFr

Author: claude

.changeset/analytics-between-predicate-dropped.md

@@ -0,0 +1,29 @@
+---
+"@objectstack/service-analytics": patch
+---
+
+fix(service-analytics): a `$between` analytics filter no longer vanishes from the query (ADR-0053 D-A3.1)
+
+A dashboard widget or dataset whose filter used `$between` was querying **every
+row**. `normalizeAnalyticsFilters` maps Mongo-style operators onto the internal
+pipeline form, `$between` was missing from that map, and an unmapped operator is
+skipped — so the predicate was silently dropped from the compiled WHERE clause.
+Both strategies read that normalizer, so both the raw-SQL and the ObjectQL
+aggregate paths were affected. The symptom is #3650's: a chart that draws the
+whole dataset instead of the requested window, with nothing in the SQL to
+suggest a filter was ever asked for.
+
+`$between [min, max]` now lowers to its two bounds (`gte` + `lte`) instead of
+gaining an operator of its own, so a range's max inherits the calendar-day
+whole-day rule (#3777) from each strategy's existing upper-bound handling —
+`NativeSQLStrategy` compiles a bare-day upper bound half-open itself, and the
+ObjectQL path gets the same rule from the driver — rather than needing a second
+implementation to keep in step. A malformed `$between` (not a two-element
+array) now throws instead of being dropped, matching the stance driver-memory
+took for the same shape in #3948: an unbounded read is exactly the failure this
+prevents, and it is indistinguishable from a legitimately wide query.
+
+Found by giving the temporal conformance matrix its missing sixth consumer
+(`native-sql-temporal-conformance.test.ts`), which executes the shared cases
+against a real SQLite engine and asserts row ids — a dropped predicate is
+invisible to the SQL-string assertions the strategy's other suites use.

docs/adr/0053-date-and-datetime-semantics.md

@@ -857,6 +857,25 @@ Two things, which is the argument for having built it:
    exists to provide. Fixed in the same change, sharing the `$lte` bound helper
    so the two cannot drift again.
 
+   **The same defect had a twin one layer over, found the same way when the
+   sixth consumer was added (#4081 follow-up).** `NativeSQLStrategy` — the
+   surface #3650 was actually about — was listed in the matrix's own backend
+   table but had no consumer, because every existing suite for it asserts the
+   emitted SQL string, and a *dropped* predicate is invisible to a string
+   assertion: the SQL stays valid, just wider. Executing the matrix against a
+   real engine showed `$between` returning the **entire table**. The cause was
+   one layer below the strategy, in the shared `filter-normalizer`: `$between`
+   was absent from `MONGO_TO_CUBE_OP`, and an unmapped operator is `continue`d,
+   so the predicate vanished from the WHERE clause. Both raw-SQL and ObjectQL
+   strategies read that normalizer, so both were affected. `$between` now
+   **lowers to its two bounds** (`gte` + `lte`) rather than gaining an operator
+   of its own: each strategy's upper-bound arm already carries the whole-day
+   calendar rule (`NativeSQLStrategy` compiles half-open directly, the ObjectQL
+   path inherits it from the driver), so a range's max gets that rule by
+   construction instead of via a second implementation. A malformed `$between`
+   now throws, the anti-silent-widening stance driver-memory took for the same
+   shape in #3948.
+
 2. **A measured, irreducible limit.** `$gt` with a bare-day comparand on a
    `datetime` column cannot agree across backends. A typed backend anchors the
    bound to midnight and excludes a value stored at exactly 00:00; a type-blind

packages/services/service-analytics/src/__tests__/native-sql-temporal-conformance.test.ts

@@ -0,0 +1,174 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * Temporal conformance for the analytics raw-SQL strategy (ADR-0053 D-A3),
+ * executed against a real SQLite engine (`sql.js`, pure WASM).
+ *
+ * The cases come from `@objectstack/spec/data` so this backend, the three
+ * drivers, the draft preview and `formula`'s write-side `check` evaluator are
+ * all held to one standard — see `temporal-conformance.ts` for the four
+ * divergences that standard exists to prevent.
+ *
+ * ## Why this surface needed its own consumer
+ *
+ * `NativeSQLStrategy` is the surface #3650 broke — the dashboard window was
+ * dropped and the chart drew all history — and it is the one listed in the
+ * matrix's own backend table that had no consumer. It is also the only backend
+ * that hand-compiles SQL text rather than delegating to a driver's compiler, so
+ * "the other backends are green" says nothing about it.
+ *
+ * Every other suite for this strategy (`native-sql-datetime-filter.test.ts`,
+ * `native-sql-datetime-filter-column.test.ts`) asserts the emitted SQL string.
+ * That is a different question, with a lower ceiling: a string assertion checks
+ * the compiler emits what its author wrote down, and a dropped predicate is
+ * invisible to it — the SQL is still valid, just wider. This file asserts ROW
+ * IDS, which is what D-A3 demanded and what catches a predicate that silently
+ * went missing. (It found one: see the `$between` note in
+ * `filter-normalizer.ts`.)
+ *
+ * ## Storage form
+ *
+ * Rows are seeded in the canonical post-#3912 form — UTC ISO text for the
+ * `datetime` column, bare `YYYY-MM-DD` for the `date` one — which is the state
+ * a backfilled deployment is in, and the state in which the driver's
+ * `temporalFilterValue` / `temporalFilterColumnSql` hooks are identities. The
+ * context therefore omits them, exercising the same "absent = identity" path a
+ * Postgres deployment takes (ADR-0053 D-A2). The un-backfilled mixed column is
+ * a different axis, covered where the driver truth lives:
+ * `native-sql-datetime-filter-column.test.ts` for the emitted normalisation and
+ * `driver-sql`'s own legacy sweep for row results.
+ *
+ * ## Why `sql.js` and not `better-sqlite3`
+ *
+ * Same reason as `read-scope-sql-conformance.test.ts`: the native binding is
+ * loadable only by the exact Node ABI it was built for and aborts the vitest
+ * worker on CI's Node, taking the whole file's cases silently with it. `sql.js`
+ * is the pure-WASM engine `driver-sql` itself falls back to.
+ */
+
+import { describe, it, expect, beforeAll, afterAll } from 'vitest';
+import { TEMPORAL_CASES, TEMPORAL_NOW, TEMPORAL_ROWS } from '@objectstack/spec/data';
+import type { Cube } from '@objectstack/spec/data';
+import type { AnalyticsQuery, StrategyContext } from '@objectstack/spec/contracts';
+import { resolveFilterTokens } from '@objectstack/core';
+
+import { NativeSQLStrategy } from '../strategies/native-sql-strategy.js';
+
+/**
+ * Dimension ids match the fixture's property names (`at` / `on`) so the shared
+ * cases apply unchanged, while the COLUMNS are deliberately different
+ * (`happened_at` / `happened_on`) — that is what proves the strategy resolved
+ * the real column rather than echoing the member name.
+ */
+const CUBE: Cube = {
+  name: 'conformance',
+  title: 'Conformance',
+  sql: 'conformance',
+  measures: { total: { name: 'total', label: 'Total', type: 'count', sql: '*' } },
+  dimensions: {
+    id: { name: 'id', label: 'Id', type: 'string', sql: 'id' },
+    at: { name: 'at', label: 'At', type: 'time', sql: 'happened_at' },
+    on: { name: 'on', label: 'On', type: 'time', sql: 'happened_on' },
+  },
+  public: false,
+} as unknown as Cube;
+
+const resolveTokens = <T,>(filter: T): T =>
+  resolveFilterTokens(filter, { now: new Date(TEMPORAL_NOW) });
+
+/** Point sql.js at the `.wasm` shipped inside its own package (Node-safe). */
+async function locateWasm(): Promise<((file: string) => string) | undefined> {
+  try {
+    const { createRequire } = await import('node:module');
+    const require = createRequire(import.meta.url);
+    const pkgJsonPath = require.resolve('sql.js/package.json');
+    const { dirname, join } = await import('node:path');
+    const dir = dirname(pkgJsonPath);
+    return (file: string) => join(dir, 'dist', file);
+  } catch {
+    return undefined;
+  }
+}
+
+describe('NativeSQLStrategy — temporal conformance', () => {
+  let db: any;
+  let ctx: StrategyContext;
+
+  beforeAll(async () => {
+    const mod: any = await import('sql.js');
+    const initSqlJs = mod.default ?? mod;
+    const locateFile = await locateWasm();
+    const SQL = await initSqlJs(locateFile ? { locateFile } : undefined);
+
+    db = new SQL.Database();
+    db.run(`
+      CREATE TABLE "conformance" (
+        "id" TEXT PRIMARY KEY,
+        "happened_at" TEXT,
+        "happened_on" TEXT,
+        "why" TEXT
+      );
+    `);
+    const insert = db.prepare(
+      `INSERT INTO "conformance" ("id","happened_at","happened_on","why") VALUES (?,?,?,?)`,
+    );
+    for (const r of TEMPORAL_ROWS) insert.run([r.id, r.at, r.on, r.why]);
+    insert.free();
+
+    ctx = {
+      getCube: (name: string) => (name === 'conformance' ? CUBE : undefined),
+      queryCapabilities: () => ({ nativeSql: true, objectqlAggregate: false, inMemory: false }),
+      // The strategy binds `$1`-style placeholders in ascending order, each
+      // pushed immediately before it is referenced, so a positional rewrite to
+      // SQLite's `?` preserves the pairing.
+      executeRawSql: async (_object: string, sql: string, params: unknown[]) => {
+        const stmt = db.prepare(sql.replace(/\$\d+/g, '?'));
+        stmt.bind(params as any[]);
+        const out: Record<string, unknown>[] = [];
+        while (stmt.step()) out.push(stmt.getAsObject());
+        stmt.free();
+        return out;
+      },
+    } as StrategyContext;
+  });
+
+  afterAll(() => {
+    db?.close();
+  });
+
+  /** Group by `id` so the result rows ARE the matched row ids. */
+  const idsFor = async (query: Omit<AnalyticsQuery, 'cube' | 'measures' | 'dimensions'>) => {
+    const result = await new NativeSQLStrategy().execute(
+      { cube: 'conformance', measures: ['total'], dimensions: ['id'], ...query } as AnalyticsQuery,
+      ctx,
+    );
+    return result.rows.map((r) => String(r.id)).sort();
+  };
+
+  for (const c of TEMPORAL_CASES) {
+    it(c.name, async () => {
+      expect(await idsFor({ where: c.filter }), c.note).toEqual([...c.expected].sort());
+    });
+
+    // The D-A3 token axis (#4081): the same case spelled in relative tokens,
+    // resolved at the pinned instant, must reach the same rows.
+    if (c.tokenFilter) {
+      it(`${c.name} — via relative tokens`, async () => {
+        expect(await idsFor({ where: resolveTokens(c.tokenFilter) }), c.note).toEqual(
+          [...c.expected].sort(),
+        );
+      });
+    }
+
+    // The dashboard-window path — the shape #3650 dropped entirely. No
+    // granularity, or `canHandle` correctly declines to the ObjectQL strategy.
+    if (c.dateRange) {
+      it(`${c.name} — via timeDimensions.dateRange`, async () => {
+        expect(
+          await idsFor({ timeDimensions: [{ dimension: c.field, dateRange: resolveTokens(c.dateRange) }] }),
+          c.note,
+        ).toEqual([...c.expected].sort());
+      });
+    }
+  }
+});

packages/services/service-analytics/src/strategies/filter-normalizer.ts

@@ -84,6 +84,37 @@ function flattenCondition(cond: Record<string, unknown>, out: NormalizedAnalytic
       const opKeys = Object.keys(wrapper).filter(k => k.startsWith('$'));
       if (opKeys.length > 0) {
         for (const opKey of opKeys) {
+          // `$between [min, max]` LOWERS to its two bounds rather than getting a
+          // `between` operator of its own. Both strategies already carry the
+          // calendar-day whole-day rule on their upper bound — NativeSQLStrategy
+          // compiles a bare-day `lte` half-open (#3777), ObjectQLStrategy hands
+          // `$lte` to the driver, which does the same — so a range's max
+          // inherits that rule by construction instead of needing a second
+          // implementation to keep in step. (The preview evaluator's `$between`
+          // gap was closed the same way, sharing its `$lte` helper.)
+          //
+          // Before this, `$between` was simply absent from the operator map and
+          // fell to the `continue` below: the predicate VANISHED from the WHERE
+          // clause, so a dashboard widget carrying a range filter charted the
+          // entire dataset — #3650's symptom, on the surface #3650 was about.
+          // The temporal conformance matrix caught it as row results
+          // (`native-sql-temporal-conformance.test.ts`).
+          if (opKey === '$between') {
+            const v = wrapper[opKey];
+            if (!Array.isArray(v) || v.length !== 2) {
+              // Never drop it: an unbounded read is the failure mode this whole
+              // branch exists to prevent, and it is indistinguishable from a
+              // legitimately wide query. Same stance driver-memory took for the
+              // same shape (#3948).
+              throw new Error(
+                `[analytics] "$between" on "${key}" needs a two-element [min, max] array, got ` +
+                `${JSON.stringify(v)}. Dropping the predicate would silently widen the query to every row.`,
+              );
+            }
+            out.push({ member: key, operator: 'gte', values: [stringifyForCube(v[0])] });
+            out.push({ member: key, operator: 'lte', values: [stringifyForCube(v[1])] });
+            continue;
+          }
           const cubeOp = MONGO_TO_CUBE_OP[opKey];
           if (!cubeOp) continue;
           const v = wrapper[opKey];