diff --git a/.changeset/analytics-between-predicate-dropped.md b/.changeset/analytics-between-predicate-dropped.md new file mode 100644 index 0000000000..ac54a581ef --- /dev/null +++ b/.changeset/analytics-between-predicate-dropped.md @@ -0,0 +1,29 @@ +--- +"@objectstack/service-analytics": patch +--- + +fix(service-analytics): a `$between` analytics filter no longer vanishes from the query (ADR-0053 D-A3.1) + +A dashboard widget or dataset whose filter used `$between` was querying **every +row**. `normalizeAnalyticsFilters` maps Mongo-style operators onto the internal +pipeline form, `$between` was missing from that map, and an unmapped operator is +skipped — so the predicate was silently dropped from the compiled WHERE clause. +Both strategies read that normalizer, so both the raw-SQL and the ObjectQL +aggregate paths were affected. The symptom is #3650's: a chart that draws the +whole dataset instead of the requested window, with nothing in the SQL to +suggest a filter was ever asked for. + +`$between [min, max]` now lowers to its two bounds (`gte` + `lte`) instead of +gaining an operator of its own, so a range's max inherits the calendar-day +whole-day rule (#3777) from each strategy's existing upper-bound handling — +`NativeSQLStrategy` compiles a bare-day upper bound half-open itself, and the +ObjectQL path gets the same rule from the driver — rather than needing a second +implementation to keep in step. A malformed `$between` (not a two-element +array) now throws instead of being dropped, matching the stance driver-memory +took for the same shape in #3948: an unbounded read is exactly the failure this +prevents, and it is indistinguishable from a legitimately wide query. + +Found by giving the temporal conformance matrix its missing sixth consumer +(`native-sql-temporal-conformance.test.ts`), which executes the shared cases +against a real SQLite engine and asserts row ids — a dropped predicate is +invisible to the SQL-string assertions the strategy's other suites use. diff --git a/.changeset/analytics-filter-operator-coverage.md b/.changeset/analytics-filter-operator-coverage.md new file mode 100644 index 0000000000..bfcc1c0957 --- /dev/null +++ b/.changeset/analytics-filter-operator-coverage.md @@ -0,0 +1,42 @@ +--- +"@objectstack/service-analytics": patch +--- + +fix(service-analytics): every authorable filter operator now reaches the query (#4128) + +Closes the cause behind the `$between` defect rather than just that instance. +`normalizeAnalyticsFilters` skipped any operator missing from its map, and a +skipped predicate does not narrow a query — it **widens** it: the compiled SQL +stays valid and returns rows the author excluded. Four operators from the +spec's authorable vocabulary sat in that state, plus one that was mapped +incorrectly. + +- **`$startsWith` / `$endsWith`** were dropped entirely. Both strategies now + compile them — anchored `LIKE 'x%'` / `LIKE '%x'` on the raw-SQL path, and + the canonical `$startsWith` / `$endsWith` operators (which every driver + implements directly) on the ObjectQL path, so an anchored match does not + depend on regex dialect. +- **`$null`** was dropped. It is the shape the console emits for an "is empty" + / "is not empty" filter, so such a widget was showing every row. Now compiles + to `IS NULL` / `IS NOT NULL` per its boolean. +- **`$exists`** was mapped value-*independently* to `set`, so `{$exists: false}` + compiled to `IS NOT NULL` — the exact inverse of what it asks for. It and + `$null` are now resolved explicitly, because a key→name map cannot express an + operator whose meaning flips with its value. +- **`$notContains`** reached the ObjectQL strategy, which had no arm for it and + fell through to a `default` returning a bare value — compiling "does not + contain x" as "**equals** x". +- **Unknown operators now throw** on both surfaces instead of being silently + dropped (normalizer) or reinterpreted as an equality (ObjectQL strategy). An + operator outside the vocabulary is a caller error, and a loud one beats a + silently widened read — the call driver-memory made for the same shape in + #3948. + +Still declared as a gap, but no longer a silent one: `$or` / `$not` are skipped, +since expressing them needs a recursive WHERE builder rather than the flat +array the strategies consume. + +Cover is `filter-operator-coverage.test.ts`, which runs the whole vocabulary +against a real SQLite engine and asserts **row ids** — six of its cases fail +without this change. A dropped predicate is invisible to the SQL-string +assertions the strategies' other suites use, which is how these survived. diff --git a/.changeset/temporal-conformance-token-axis.md b/.changeset/temporal-conformance-token-axis.md new file mode 100644 index 0000000000..3f75d40a2b --- /dev/null +++ b/.changeset/temporal-conformance-token-axis.md @@ -0,0 +1,26 @@ +--- +"@objectstack/spec": minor +--- + +feat(spec): the temporal conformance matrix gains its relative-token axis (ADR-0053 D-A3, #4081) + +The matrix (#4098) landed with the token axis documented as still open — the +cases took resolved comparands, while authors actually write `{today}` / +`{90_days_ago}` / `{current_month_end}`, and nothing proved the resolved +token reaches the same rows on every backend. Closed here: + +- `TemporalCase` gains `tokenFilter?` (the same filter spelled in tokens) and + `dateRange?` (the analytics `timeDimensions.dateRange` spelling of the same + window — the surface #3650 broke), plus the pinned instant `TEMPORAL_NOW` + consumers hand to `resolveFilterTokens`. A resolver drift and an evaluator + drift are now distinguishable at a glance: the literal spelling fails for + one, the token spelling for the other. +- `TemporalRow` gains `writerForm` (`wire` ISO string vs `native` `Date`), so + driver consumers seed genuinely mixed writer populations through their own + `create()` — the exact column shape that produced #4047 (D-E4). +- New rows/cells: a pre-epoch row (negative epoch ms, the #3773 family), and + the `date` equality/`$in` cases — #1874's original `date == today` shape. +- Two more sweeps consume the table: a `driver-sqlite-wasm` conformance test + pinning the inherited SqlDriver seam, and a legacy-storage sweep in + `driver-sql` running every case over an un-backfilled mixed epoch/naive + column through the read-repair path. diff --git a/docs/adr/0053-date-and-datetime-semantics.md b/docs/adr/0053-date-and-datetime-semantics.md index 510b5420b5..3873a6dbb9 100644 --- a/docs/adr/0053-date-and-datetime-semantics.md +++ b/docs/adr/0053-date-and-datetime-semantics.md @@ -857,6 +857,25 @@ Two things, which is the argument for having built it: exists to provide. Fixed in the same change, sharing the `$lte` bound helper so the two cannot drift again. + **The same defect had a twin one layer over, found the same way when the + sixth consumer was added (#4081 follow-up).** `NativeSQLStrategy` — the + surface #3650 was actually about — was listed in the matrix's own backend + table but had no consumer, because every existing suite for it asserts the + emitted SQL string, and a *dropped* predicate is invisible to a string + assertion: the SQL stays valid, just wider. Executing the matrix against a + real engine showed `$between` returning the **entire table**. The cause was + one layer below the strategy, in the shared `filter-normalizer`: `$between` + was absent from `MONGO_TO_CUBE_OP`, and an unmapped operator is `continue`d, + so the predicate vanished from the WHERE clause. Both raw-SQL and ObjectQL + strategies read that normalizer, so both were affected. `$between` now + **lowers to its two bounds** (`gte` + `lte`) rather than gaining an operator + of its own: each strategy's upper-bound arm already carries the whole-day + calendar rule (`NativeSQLStrategy` compiles half-open directly, the ObjectQL + path inherits it from the driver), so a range's max gets that rule by + construction instead of via a second implementation. A malformed `$between` + now throws, the anti-silent-widening stance driver-memory took for the same + shape in #3948. + 2. **A measured, irreducible limit.** `$gt` with a bare-day comparand on a `datetime` column cannot agree across backends. A typed backend anchors the bound to midnight and excludes a value stored at exactly 00:00; a type-blind @@ -876,7 +895,19 @@ Two things, which is the argument for having built it: - The matrix is the ratchet the previous four fixes lacked: a backend that drifts now fails a named case whose note says which incident it is repeating. -- Coverage still open, and deliberately so: the **relative-token** axis +- ~~Coverage still open, and deliberately so: the **relative-token** axis (`{today}`, `{30_days_ago}` resolved through `filter-tokens.ts` and run end-to-end per driver) is not yet in the table. The cases here take resolved - comparands, which is the layer where the four incidents actually happened. + comparands, which is the layer where the four incidents actually happened.~~ + Closed in the #4081 follow-up: cases now carry a `tokenFilter` (and, for the + analytics window path, a `dateRange`) spelling — the same filter written in + `{today}` / `{90_days_ago}` / period tokens — which every consumer that can + reach `@objectstack/core` resolves against the pinned `TEMPORAL_NOW` and + must land on the same row ids as the literal spelling. `formula` alone skips + the token sweep (its dependencies deliberately stop at `spec`; tokens are + resolved before a filter reaches it). The same follow-up added the + `writerForm` seeding hint (D-E4's mixed-writer populations through each + driver's own `create()`), a pre-epoch row, the `date` equality/`$in` cells + (#1874's original shape), a `driver-sqlite-wasm` consumer pinning the + inheritance seam, and a legacy-storage sweep in the `driver-sql` consumer so + the un-backfilled read-repair path answers the same table. diff --git a/packages/plugins/driver-memory/src/memory-temporal-conformance.test.ts b/packages/plugins/driver-memory/src/memory-temporal-conformance.test.ts index b2255b461a..02fee7d5d1 100644 --- a/packages/plugins/driver-memory/src/memory-temporal-conformance.test.ts +++ b/packages/plugins/driver-memory/src/memory-temporal-conformance.test.ts @@ -8,12 +8,22 @@ * evaluator are all held to one standard — see `temporal-conformance.ts` for * the four divergences that standard exists to prevent, one of which (#4047) * was this driver's. + * + * Rows are seeded in their tagged writer forms (ISO string vs JS `Date` — the + * D-E4 mixed-writer axis), reproducing exactly the mixed column #4047 hit; + * the sweep proves the converged storage answers every shared case. Cases + * carrying a token spelling also run through `resolveFilterTokens` at the + * pinned `TEMPORAL_NOW` — the D-A3 "token → row results" axis (#4081). */ import { describe, it, expect, beforeAll } from 'vitest'; -import { TEMPORAL_CASES, TEMPORAL_ROWS } from '@objectstack/spec/data'; +import { TEMPORAL_CASES, TEMPORAL_NOW, TEMPORAL_ROWS } from '@objectstack/spec/data'; +import { resolveFilterTokens } from '@objectstack/core'; import { InMemoryDriver } from './memory-driver.js'; +const resolveTokens = (filter: T): T => + resolveFilterTokens(filter, { now: new Date(TEMPORAL_NOW) }); + describe('driver-memory — temporal conformance', () => { let driver: InMemoryDriver; @@ -27,7 +37,13 @@ describe('driver-memory — temporal conformance', () => { fields: { at: { type: 'datetime' }, on: { type: 'date' }, why: { type: 'string' } }, }); for (const r of TEMPORAL_ROWS) { - await driver.create('conformance', { id: r.id, at: r.at, on: r.on, why: r.why }); + await driver.create('conformance', { + id: r.id, + // The mixed-writer axis (D-E4): both shapes must converge on write. + at: r.writerForm === 'native' ? new Date(r.at) : r.at, + on: r.on, + why: r.why, + }); } }); @@ -37,5 +53,13 @@ describe('driver-memory — temporal conformance', () => { const got = (rows as any[]).map((r) => r.id).sort(); expect(got, c.note).toEqual([...c.expected].sort()); }); + + if (c.tokenFilter) { + it(`${c.name} — via relative tokens`, async () => { + const rows = await driver.find('conformance', { where: resolveTokens(c.tokenFilter) } as any); + const got = (rows as any[]).map((r) => r.id).sort(); + expect(got, c.note).toEqual([...c.expected].sort()); + }); + } } }); diff --git a/packages/plugins/driver-mongodb/src/mongodb-temporal-conformance.test.ts b/packages/plugins/driver-mongodb/src/mongodb-temporal-conformance.test.ts index 3b40b7eeb2..cf6736757f 100644 --- a/packages/plugins/driver-mongodb/src/mongodb-temporal-conformance.test.ts +++ b/packages/plugins/driver-mongodb/src/mongodb-temporal-conformance.test.ts @@ -11,13 +11,22 @@ * was this driver's and was the worst of them: type-bracket comparison meant a * string bound matched no BSON `Date` row at all, so the default dashboard * window returned nothing. + * + * Rows are seeded in their tagged writer forms (ISO string vs JS `Date` — the + * D-E4 mixed-writer axis), the exact writer population #4047 hit. Cases + * carrying a token spelling also run through `resolveFilterTokens` at the + * pinned `TEMPORAL_NOW` — the D-A3 "token → row results" axis (#4081). */ import { describe, it, expect, beforeAll, afterAll } from 'vitest'; import { MongoMemoryServer } from 'mongodb-memory-server'; -import { TEMPORAL_CASES, TEMPORAL_ROWS } from '@objectstack/spec/data'; +import { TEMPORAL_CASES, TEMPORAL_NOW, TEMPORAL_ROWS } from '@objectstack/spec/data'; +import { resolveFilterTokens } from '@objectstack/core'; import { MongoDBDriver } from './mongodb-driver.js'; +const resolveTokens = (filter: T): T => + resolveFilterTokens(filter, { now: new Date(TEMPORAL_NOW) }); + let sharedMongod: MongoMemoryServer | undefined; try { sharedMongod = await MongoMemoryServer.create({ instance: { launchTimeout: 60_000 } }); @@ -42,7 +51,13 @@ describe.skipIf(!sharedMongod)('driver-mongodb — temporal conformance', () => fields: { at: { type: 'datetime' }, on: { type: 'date' }, why: { type: 'string' } }, }); for (const r of TEMPORAL_ROWS) { - await driver.create('conformance', { id: r.id, at: r.at, on: r.on, why: r.why }); + await driver.create('conformance', { + id: r.id, + // The mixed-writer axis (D-E4): the exact population #4047 hit. + at: r.writerForm === 'native' ? new Date(r.at) : r.at, + on: r.on, + why: r.why, + }); } }, 90_000); @@ -57,5 +72,13 @@ describe.skipIf(!sharedMongod)('driver-mongodb — temporal conformance', () => const got = (rows as any[]).map((r) => r.id).sort(); expect(got, c.note).toEqual([...c.expected].sort()); }); + + if (c.tokenFilter) { + it(`${c.name} — via relative tokens`, async () => { + const rows = await driver.find('conformance', { where: resolveTokens(c.tokenFilter) } as any); + const got = (rows as any[]).map((r) => r.id).sort(); + expect(got, c.note).toEqual([...c.expected].sort()); + }); + } } }); diff --git a/packages/plugins/driver-sql/src/sql-driver-temporal-conformance.test.ts b/packages/plugins/driver-sql/src/sql-driver-temporal-conformance.test.ts index 958275a8dd..321a60d2f5 100644 --- a/packages/plugins/driver-sql/src/sql-driver-temporal-conformance.test.ts +++ b/packages/plugins/driver-sql/src/sql-driver-temporal-conformance.test.ts @@ -14,6 +14,17 @@ * `Field.date` and coerces comparands accordingly. That the same case yields the * same row set here and in the type-blind backends is the whole assertion. * + * Three sweeps, because this driver owns two extra axes (#4081): + * 1. canonical storage — rows seeded through `create()` in their tagged + * writer forms (ISO string vs JS `Date`, the D-E4 mixed-writer axis), + * which `formatInput` must converge; + * 2. the same cases spelled in relative tokens, resolved through + * `@objectstack/core`'s `resolveFilterTokens` at the pinned `TEMPORAL_NOW` + * — the D-A3 "token → row results" axis; + * 3. legacy storage — the same rows seeded RAW as pre-#3912 forms (INTEGER + * epoch ms / zone-naive TEXT) with the canonical marker cleared, so the + * read-repair path answers the same table. + * * Runs against a real SQLite, and — through the `Temporal Conformance * (live PG + MySQL)` CI job, which runs this package's whole suite under * `TZ=America/New_York` against servers on `Asia/Shanghai` — against real @@ -21,8 +32,22 @@ */ import { describe, it, expect, beforeAll, afterAll } from 'vitest'; -import { TEMPORAL_CASES, TEMPORAL_ROWS } from '@objectstack/spec/data'; +import { TEMPORAL_CASES, TEMPORAL_NOW, TEMPORAL_ROWS } from '@objectstack/spec/data'; +import { resolveFilterTokens } from '@objectstack/core'; import { SqlDriver } from '../src/index.js'; +import { LegacyStorageDriver } from './legacy-datetime-storage.testkit.js'; + +const resolveTokens = (filter: T): T => + resolveFilterTokens(filter, { now: new Date(TEMPORAL_NOW) }); + +const CONFORMANCE_OBJECT = { + name: 'conformance', + fields: { + at: { type: 'datetime' }, + on: { type: 'date' }, + why: { type: 'string' }, + }, +}; describe('sql-driver — temporal conformance', () => { let driver: SqlDriver; @@ -35,20 +60,18 @@ describe('sql-driver — temporal conformance', () => { }); // The declaration is what makes this the typed half — `at` is an instant, // `on` a calendar day, and the driver's coercion follows from that. - await driver.initObjects([ - { - name: 'conformance', - fields: { - at: { type: 'datetime' }, - on: { type: 'date' }, - why: { type: 'string' }, - }, - }, - ]); + await driver.initObjects([CONFORMANCE_OBJECT]); for (const r of TEMPORAL_ROWS) { await driver.create( 'conformance', - { id: r.id, at: r.at, on: r.on, why: r.why }, + { + id: r.id, + // The writer-form axis (D-E4): both writer populations must + // converge to the one canonical storage form on write. + at: r.writerForm === 'native' ? new Date(r.at) : r.at, + on: r.on, + why: r.why, + }, { bypassTenantAudit: true } as any, ); } @@ -58,6 +81,57 @@ describe('sql-driver — temporal conformance', () => { await driver.disconnect?.(); }); + for (const c of TEMPORAL_CASES) { + it(c.name, async () => { + const rows = await driver.find('conformance', { where: c.filter } as any); + const got = (rows as any[]).map((r) => r.id).sort(); + expect(got, c.note).toEqual([...c.expected].sort()); + }); + + if (c.tokenFilter) { + it(`${c.name} — via relative tokens`, async () => { + const rows = await driver.find('conformance', { where: resolveTokens(c.tokenFilter) } as any); + const got = (rows as any[]).map((r) => r.id).sort(); + expect(got, c.note).toEqual([...c.expected].sort()); + }); + } + } +}); + +describe('sql-driver — temporal conformance on un-backfilled legacy storage', () => { + let driver: LegacyStorageDriver; + + beforeAll(async () => { + driver = new LegacyStorageDriver({ + client: 'better-sqlite3', + connection: { filename: ':memory:' }, + useNullAsDefault: true, + }); + await driver.initObjects([CONFORMANCE_OBJECT]); + // The two pre-#3912 storage forms, split by the same writer-form tag: + // `native` writes landed as INTEGER epoch ms (a bound JS Date), `wire` + // writes as zone-naive TEXT (CURRENT_TIMESTAMP / REST payloads). One + // column, both forms — the read-repair path must answer the same table + // the canonical sweep does. (`on` is unaffected: bare-day text has been + // the date canon since Phase 1.) + await driver.seedLegacyRows( + 'conformance', + 'at', + TEMPORAL_ROWS.map((r) => ({ + id: r.id, + at: r.writerForm === 'native' ? Date.parse(r.at) : r.at.replace('T', ' ').replace('Z', ''), + on: r.on, + why: r.why, + })), + ); + }); + + afterAll(async () => { + await driver.disconnect?.(); + }); + + // Literal spellings only: the token axis is orthogonal to storage form and + // already swept above — a divergence here is a repair-path bug by construction. for (const c of TEMPORAL_CASES) { it(c.name, async () => { const rows = await driver.find('conformance', { where: c.filter } as any); diff --git a/packages/plugins/driver-sqlite-wasm/src/sqlite-wasm-temporal-conformance.test.ts b/packages/plugins/driver-sqlite-wasm/src/sqlite-wasm-temporal-conformance.test.ts new file mode 100644 index 0000000000..45705370f1 --- /dev/null +++ b/packages/plugins/driver-sqlite-wasm/src/sqlite-wasm-temporal-conformance.test.ts @@ -0,0 +1,68 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * Temporal conformance for the wasm driver (ADR-0053 D-A3, #4081). + * + * `SqliteWasmDriver extends SqlDriver`, so the whole temporal seam — canonical + * datetime storage (#3912), the calendar-day bound rewrites (#3777/#4042), the + * comparand coercion — is inherited, not re-implemented. This runs the shared + * `@objectstack/spec/data` table against the wasm engine so the suite fails if + * this driver ever stops sharing that seam — the same guard the #3773/#3777 + * pin tests established one case at a time, now spanning the full matrix, + * token spellings included. + */ + +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import { TEMPORAL_CASES, TEMPORAL_NOW, TEMPORAL_ROWS } from '@objectstack/spec/data'; +import { resolveFilterTokens } from '@objectstack/core'; +import { SqliteWasmDriver } from './index.js'; + +const resolveTokens = (filter: T): T => + resolveFilterTokens(filter, { now: new Date(TEMPORAL_NOW) }); + +describe('driver-sqlite-wasm — temporal conformance', () => { + let driver: SqliteWasmDriver; + + beforeAll(async () => { + driver = new SqliteWasmDriver({ filename: ':memory:' }); + await driver.initObjects([ + { + name: 'conformance', + fields: { at: { type: 'datetime' }, on: { type: 'date' }, why: { type: 'string' } }, + }, + ]); + for (const r of TEMPORAL_ROWS) { + await driver.create( + 'conformance', + { + id: r.id, + // The mixed-writer axis (D-E4): both shapes must converge on write. + at: r.writerForm === 'native' ? new Date(r.at) : r.at, + on: r.on, + why: r.why, + }, + { bypassTenantAudit: true } as any, + ); + } + }); + + afterAll(async () => { + await (driver as any).knex.destroy(); + }); + + for (const c of TEMPORAL_CASES) { + it(c.name, async () => { + const rows = await driver.find('conformance', { where: c.filter } as any); + const got = (rows as any[]).map((r) => r.id).sort(); + expect(got, c.note).toEqual([...c.expected].sort()); + }); + + if (c.tokenFilter) { + it(`${c.name} — via relative tokens`, async () => { + const rows = await driver.find('conformance', { where: resolveTokens(c.tokenFilter) } as any); + const got = (rows as any[]).map((r) => r.id).sort(); + expect(got, c.note).toEqual([...c.expected].sort()); + }); + } + } +}); diff --git a/packages/services/service-analytics/src/__tests__/filter-operator-coverage.test.ts b/packages/services/service-analytics/src/__tests__/filter-operator-coverage.test.ts new file mode 100644 index 0000000000..94098c7ee9 --- /dev/null +++ b/packages/services/service-analytics/src/__tests__/filter-operator-coverage.test.ts @@ -0,0 +1,193 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * Every authorable filter operator reaches the query — the closure of #4128. + * + * `normalizeAnalyticsFilters` maps the spec's `FilterCondition` vocabulary onto + * the internal pipeline form. Whatever it fails to map used to be `continue`d, + * and that is not "unsupported": the predicate DISAPPEARS, the compiled SQL + * stays valid, and the query returns rows the author excluded. It reads as a + * chart drawn over the whole dataset (#3650's symptom) and is invisible to a + * test that asserts the emitted SQL string — which is what every other suite + * for these strategies does, and why four operators sat broken behind one that + * had already been found (`$between`, ADR-0053 D-A3.1). + * + * So this file asserts ROW IDS, against a real SQLite (`sql.js`, the pure-WASM + * engine `driver-sql` itself falls back to — see the note in + * `read-scope-sql-conformance.test.ts` for why not `better-sqlite3`). A dropped + * predicate cannot hide from it: the row set is simply wrong. + * + * The table below is the whole authorable vocabulary of `filter.zod.ts`, so a + * new operator added to the spec without a home in the pipeline fails here + * rather than silently widening a customer's dashboard. + */ + +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import type { Cube } from '@objectstack/spec/data'; +import type { AnalyticsQuery, FilterCondition } from '@objectstack/spec/data'; +import type { StrategyContext } from '@objectstack/spec/contracts'; + +import { NativeSQLStrategy } from '../strategies/native-sql-strategy.js'; +import { normalizeAnalyticsFilters } from '../strategies/filter-normalizer.js'; + +interface Row { + id: string; + name: string | null; + score: number; +} + +/** `n_null` carries a NULL name — the row the null predicates turn on. */ +const ROWS: Row[] = [ + { id: 'a_alpha', name: 'alpha-one', score: 10 }, + { id: 'b_alphex', name: 'alphex-two', score: 20 }, + { id: 'c_beta', name: 'beta-one', score: 30 }, + { id: 'd_null', name: null, score: 40 }, +]; + +const CUBE: Cube = { + name: 'ops', + title: 'Ops', + sql: 'ops', + measures: { total: { name: 'total', label: 'Total', type: 'count', sql: '*' } }, + dimensions: { + id: { name: 'id', label: 'Id', type: 'string', sql: 'id' }, + name: { name: 'name', label: 'Name', type: 'string', sql: 'name' }, + score: { name: 'score', label: 'Score', type: 'number', sql: 'score' }, + }, + public: false, +} as unknown as Cube; + +/** + * One case per operator in `filter.zod.ts`'s authorable vocabulary. + * `expected` is the row ids the filter MUST return — the full row set means + * the predicate went missing. + */ +const CASES: Array<{ op: string; filter: FilterCondition; expected: string[]; note?: string }> = [ + { op: '$eq', filter: { name: { $eq: 'alpha-one' } }, expected: ['a_alpha'] }, + { op: '$ne', filter: { name: { $ne: 'alpha-one' } }, expected: ['b_alphex', 'c_beta'] }, + { op: '$gt', filter: { score: { $gt: 20 } }, expected: ['c_beta', 'd_null'] }, + { op: '$gte', filter: { score: { $gte: 20 } }, expected: ['b_alphex', 'c_beta', 'd_null'] }, + { op: '$lt', filter: { score: { $lt: 20 } }, expected: ['a_alpha'] }, + { op: '$lte', filter: { score: { $lte: 20 } }, expected: ['a_alpha', 'b_alphex'] }, + { op: '$in', filter: { name: { $in: ['alpha-one', 'beta-one'] } }, expected: ['a_alpha', 'c_beta'] }, + { op: '$nin', filter: { name: { $nin: ['alpha-one'] } }, expected: ['b_alphex', 'c_beta'] }, + { op: '$between', filter: { score: { $between: [20, 30] } }, expected: ['b_alphex', 'c_beta'], note: '#4128: was dropped → every row.' }, + { op: '$contains', filter: { name: { $contains: 'one' } }, expected: ['a_alpha', 'c_beta'] }, + { + op: '$notContains', + filter: { name: { $notContains: 'one' } }, + expected: ['b_alphex'], + note: 'On the ObjectQL path this had no arm and fell to the default, compiling "does not contain" as an EQUALITY.', + }, + { + op: '$startsWith', + filter: { name: { $startsWith: 'alpha' } }, + expected: ['a_alpha'], + note: '#4128: was dropped → every row. b_alphex proves the anchor is a prefix, not a substring.', + }, + { + op: '$endsWith', + filter: { name: { $endsWith: '-one' } }, + expected: ['a_alpha', 'c_beta'], + note: '#4128: was dropped → every row.', + }, + { + op: '$null: true', + filter: { name: { $null: true } }, + expected: ['d_null'], + note: '#4128: was dropped → every row. This is the shape the console emits for an "is empty" filter.', + }, + { op: '$null: false', filter: { name: { $null: false } }, expected: ['a_alpha', 'b_alphex', 'c_beta'] }, + { op: '$exists: true', filter: { name: { $exists: true } }, expected: ['a_alpha', 'b_alphex', 'c_beta'] }, + { + op: '$exists: false', + filter: { name: { $exists: false } }, + expected: ['d_null'], + note: 'Was mapped value-INDEPENDENTLY to `set`, so it compiled to IS NOT NULL — the exact inverse of what it asks.', + }, + { op: 'implicit equality', filter: { name: 'beta-one' }, expected: ['c_beta'] }, + { op: 'bare null', filter: { name: null }, expected: ['d_null'] }, + { + op: '$and', + filter: { $and: [{ score: { $gte: 20 } }, { name: { $contains: 'one' } }] }, + expected: ['c_beta'], + }, +]; + +/** Point sql.js at the `.wasm` shipped inside its own package (Node-safe). */ +async function locateWasm(): Promise<((file: string) => string) | undefined> { + try { + const { createRequire } = await import('node:module'); + const require = createRequire(import.meta.url); + const pkgJsonPath = require.resolve('sql.js/package.json'); + const { dirname, join } = await import('node:path'); + const dir = dirname(pkgJsonPath); + return (file: string) => join(dir, 'dist', file); + } catch { + return undefined; + } +} + +describe('analytics filters — every authorable operator reaches the query (#4128)', () => { + let db: any; + let ctx: StrategyContext; + + beforeAll(async () => { + const mod: any = await import('sql.js'); + const initSqlJs = mod.default ?? mod; + const locateFile = await locateWasm(); + const SQL = await initSqlJs(locateFile ? { locateFile } : undefined); + + db = new SQL.Database(); + db.run(`CREATE TABLE "ops" ("id" TEXT PRIMARY KEY, "name" TEXT, "score" INTEGER);`); + const insert = db.prepare(`INSERT INTO "ops" ("id","name","score") VALUES (?,?,?)`); + for (const r of ROWS) insert.run([r.id, r.name, r.score]); + insert.free(); + + ctx = { + getCube: (name: string) => (name === 'ops' ? CUBE : undefined), + queryCapabilities: () => ({ nativeSql: true, objectqlAggregate: false, inMemory: false }), + executeRawSql: async (_object: string, sql: string, params: unknown[]) => { + const stmt = db.prepare(sql.replace(/\$\d+/g, '?')); + stmt.bind(params as any[]); + const out: Record[] = []; + while (stmt.step()) out.push(stmt.getAsObject()); + stmt.free(); + return out; + }, + } as StrategyContext; + }); + + afterAll(() => { + db?.close(); + }); + + for (const c of CASES) { + it(`${c.op} narrows to the rows it names`, async () => { + const result = await new NativeSQLStrategy().execute( + { cube: 'ops', measures: ['total'], dimensions: ['id'], where: c.filter } as AnalyticsQuery, + ctx, + ); + const got = result.rows.map((r) => String(r.id)).sort(); + expect(got, c.note).toEqual(c.expected); + // Belt and braces: a predicate that silently vanished returns the whole + // fixture, which is the one wrong answer that looks like a working query. + expect(got.length, `${c.op} matched every row — the predicate was dropped`).toBeLessThan(ROWS.length); + }); + } + + it('an operator outside the vocabulary throws instead of widening the query', () => { + // The failure mode this whole file exists to prevent: silently returning + // rows the filter excludes. A typo'd or non-spec operator is a caller + // error, and a loud one — the same call driver-memory made in #3948. + expect(() => + normalizeAnalyticsFilters({ where: { name: { $sortOf: 'alpha' } } }), + ).toThrow(/Unsupported filter operator "\$sortOf"/); + }); + + it('a malformed $between throws rather than binding a half-open guess', () => { + expect(() => normalizeAnalyticsFilters({ where: { score: { $between: [10] } } })).toThrow( + /two-element/, + ); + }); +}); diff --git a/packages/services/service-analytics/src/__tests__/native-sql-temporal-conformance.test.ts b/packages/services/service-analytics/src/__tests__/native-sql-temporal-conformance.test.ts new file mode 100644 index 0000000000..e741855c51 --- /dev/null +++ b/packages/services/service-analytics/src/__tests__/native-sql-temporal-conformance.test.ts @@ -0,0 +1,174 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * Temporal conformance for the analytics raw-SQL strategy (ADR-0053 D-A3), + * executed against a real SQLite engine (`sql.js`, pure WASM). + * + * The cases come from `@objectstack/spec/data` so this backend, the three + * drivers, the draft preview and `formula`'s write-side `check` evaluator are + * all held to one standard — see `temporal-conformance.ts` for the four + * divergences that standard exists to prevent. + * + * ## Why this surface needed its own consumer + * + * `NativeSQLStrategy` is the surface #3650 broke — the dashboard window was + * dropped and the chart drew all history — and it is the one listed in the + * matrix's own backend table that had no consumer. It is also the only backend + * that hand-compiles SQL text rather than delegating to a driver's compiler, so + * "the other backends are green" says nothing about it. + * + * Every other suite for this strategy (`native-sql-datetime-filter.test.ts`, + * `native-sql-datetime-filter-column.test.ts`) asserts the emitted SQL string. + * That is a different question, with a lower ceiling: a string assertion checks + * the compiler emits what its author wrote down, and a dropped predicate is + * invisible to it — the SQL is still valid, just wider. This file asserts ROW + * IDS, which is what D-A3 demanded and what catches a predicate that silently + * went missing. (It found one: see the `$between` note in + * `filter-normalizer.ts`.) + * + * ## Storage form + * + * Rows are seeded in the canonical post-#3912 form — UTC ISO text for the + * `datetime` column, bare `YYYY-MM-DD` for the `date` one — which is the state + * a backfilled deployment is in, and the state in which the driver's + * `temporalFilterValue` / `temporalFilterColumnSql` hooks are identities. The + * context therefore omits them, exercising the same "absent = identity" path a + * Postgres deployment takes (ADR-0053 D-A2). The un-backfilled mixed column is + * a different axis, covered where the driver truth lives: + * `native-sql-datetime-filter-column.test.ts` for the emitted normalisation and + * `driver-sql`'s own legacy sweep for row results. + * + * ## Why `sql.js` and not `better-sqlite3` + * + * Same reason as `read-scope-sql-conformance.test.ts`: the native binding is + * loadable only by the exact Node ABI it was built for and aborts the vitest + * worker on CI's Node, taking the whole file's cases silently with it. `sql.js` + * is the pure-WASM engine `driver-sql` itself falls back to. + */ + +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import { TEMPORAL_CASES, TEMPORAL_NOW, TEMPORAL_ROWS } from '@objectstack/spec/data'; +import type { Cube } from '@objectstack/spec/data'; +import type { AnalyticsQuery, StrategyContext } from '@objectstack/spec/contracts'; +import { resolveFilterTokens } from '@objectstack/core'; + +import { NativeSQLStrategy } from '../strategies/native-sql-strategy.js'; + +/** + * Dimension ids match the fixture's property names (`at` / `on`) so the shared + * cases apply unchanged, while the COLUMNS are deliberately different + * (`happened_at` / `happened_on`) — that is what proves the strategy resolved + * the real column rather than echoing the member name. + */ +const CUBE: Cube = { + name: 'conformance', + title: 'Conformance', + sql: 'conformance', + measures: { total: { name: 'total', label: 'Total', type: 'count', sql: '*' } }, + dimensions: { + id: { name: 'id', label: 'Id', type: 'string', sql: 'id' }, + at: { name: 'at', label: 'At', type: 'time', sql: 'happened_at' }, + on: { name: 'on', label: 'On', type: 'time', sql: 'happened_on' }, + }, + public: false, +} as unknown as Cube; + +const resolveTokens = (filter: T): T => + resolveFilterTokens(filter, { now: new Date(TEMPORAL_NOW) }); + +/** Point sql.js at the `.wasm` shipped inside its own package (Node-safe). */ +async function locateWasm(): Promise<((file: string) => string) | undefined> { + try { + const { createRequire } = await import('node:module'); + const require = createRequire(import.meta.url); + const pkgJsonPath = require.resolve('sql.js/package.json'); + const { dirname, join } = await import('node:path'); + const dir = dirname(pkgJsonPath); + return (file: string) => join(dir, 'dist', file); + } catch { + return undefined; + } +} + +describe('NativeSQLStrategy — temporal conformance', () => { + let db: any; + let ctx: StrategyContext; + + beforeAll(async () => { + const mod: any = await import('sql.js'); + const initSqlJs = mod.default ?? mod; + const locateFile = await locateWasm(); + const SQL = await initSqlJs(locateFile ? { locateFile } : undefined); + + db = new SQL.Database(); + db.run(` + CREATE TABLE "conformance" ( + "id" TEXT PRIMARY KEY, + "happened_at" TEXT, + "happened_on" TEXT, + "why" TEXT + ); + `); + const insert = db.prepare( + `INSERT INTO "conformance" ("id","happened_at","happened_on","why") VALUES (?,?,?,?)`, + ); + for (const r of TEMPORAL_ROWS) insert.run([r.id, r.at, r.on, r.why]); + insert.free(); + + ctx = { + getCube: (name: string) => (name === 'conformance' ? CUBE : undefined), + queryCapabilities: () => ({ nativeSql: true, objectqlAggregate: false, inMemory: false }), + // The strategy binds `$1`-style placeholders in ascending order, each + // pushed immediately before it is referenced, so a positional rewrite to + // SQLite's `?` preserves the pairing. + executeRawSql: async (_object: string, sql: string, params: unknown[]) => { + const stmt = db.prepare(sql.replace(/\$\d+/g, '?')); + stmt.bind(params as any[]); + const out: Record[] = []; + while (stmt.step()) out.push(stmt.getAsObject()); + stmt.free(); + return out; + }, + } as StrategyContext; + }); + + afterAll(() => { + db?.close(); + }); + + /** Group by `id` so the result rows ARE the matched row ids. */ + const idsFor = async (query: Omit) => { + const result = await new NativeSQLStrategy().execute( + { cube: 'conformance', measures: ['total'], dimensions: ['id'], ...query } as AnalyticsQuery, + ctx, + ); + return result.rows.map((r) => String(r.id)).sort(); + }; + + for (const c of TEMPORAL_CASES) { + it(c.name, async () => { + expect(await idsFor({ where: c.filter }), c.note).toEqual([...c.expected].sort()); + }); + + // The D-A3 token axis (#4081): the same case spelled in relative tokens, + // resolved at the pinned instant, must reach the same rows. + if (c.tokenFilter) { + it(`${c.name} — via relative tokens`, async () => { + expect(await idsFor({ where: resolveTokens(c.tokenFilter) }), c.note).toEqual( + [...c.expected].sort(), + ); + }); + } + + // The dashboard-window path — the shape #3650 dropped entirely. No + // granularity, or `canHandle` correctly declines to the ObjectQL strategy. + if (c.dateRange) { + it(`${c.name} — via timeDimensions.dateRange`, async () => { + expect( + await idsFor({ timeDimensions: [{ dimension: c.field, dateRange: resolveTokens(c.dateRange) }] }), + c.note, + ).toEqual([...c.expected].sort()); + }); + } + } +}); diff --git a/packages/services/service-analytics/src/__tests__/preview-temporal-conformance.test.ts b/packages/services/service-analytics/src/__tests__/preview-temporal-conformance.test.ts index 5964fc22b5..e7f8994fe1 100644 --- a/packages/services/service-analytics/src/__tests__/preview-temporal-conformance.test.ts +++ b/packages/services/service-analytics/src/__tests__/preview-temporal-conformance.test.ts @@ -18,8 +18,13 @@ */ import { describe, it, expect } from 'vitest'; -import { TEMPORAL_CASES, TEMPORAL_ROWS } from '@objectstack/spec/data'; -import { matchesWhere } from '../preview-evaluator.js'; +import { TEMPORAL_CASES, TEMPORAL_NOW, TEMPORAL_ROWS } from '@objectstack/spec/data'; +import type { Cube } from '@objectstack/spec/data'; +import { resolveFilterTokens } from '@objectstack/core'; +import { evaluateAnalyticsQueryOverRows, matchesWhere } from '../preview-evaluator.js'; + +const resolveTokens = (filter: T): T => + resolveFilterTokens(filter, { now: new Date(TEMPORAL_NOW) }); describe('preview-evaluator — temporal conformance', () => { for (const c of TEMPORAL_CASES) { @@ -27,5 +32,45 @@ describe('preview-evaluator — temporal conformance', () => { const got = TEMPORAL_ROWS.filter((r) => matchesWhere(r as any, c.filter as any)).map((r) => r.id); expect(got, c.note).toEqual(c.expected); }); + + // The D-A3 token axis (#4081): the same case spelled in relative tokens, + // resolved at the pinned instant, must reach the same rows. + if (c.tokenFilter) { + it(`${c.name} — via relative tokens`, () => { + const where = resolveTokens(c.tokenFilter); + const got = TEMPORAL_ROWS.filter((r) => matchesWhere(r as any, where as any)).map((r) => r.id); + expect(got, c.note).toEqual(c.expected); + }); + } + } +}); + +describe('preview-evaluator — timeDimensions.dateRange temporal conformance', () => { + // The dashboard-window path — the surface #3650 broke (the range was + // dropped entirely and every row charted). Windows tagged with a + // `dateRange` spelling run through the full evaluator, grouped by `id` so + // the output rows ARE the matched row ids. + const CUBE = { + name: 'conformance_ds', + sql: 'conformance', + dimensions: { id: { name: 'id', type: 'string', sql: 'id' } }, + measures: { count: { name: 'count', type: 'count', sql: '*' } }, + } as unknown as Cube; + + for (const c of TEMPORAL_CASES) { + if (!c.dateRange) continue; + it(`${c.name} — via timeDimensions.dateRange`, () => { + const result = evaluateAnalyticsQueryOverRows( + { + measures: ['count'], + dimensions: ['id'], + timeDimensions: [{ dimension: c.field, dateRange: resolveTokens(c.dateRange) }], + }, + CUBE, + TEMPORAL_ROWS.map((r) => ({ ...r })), + ); + const got = result.rows.map((r) => String(r.id)).sort(); + expect(got, c.note).toEqual([...c.expected].sort()); + }); } }); diff --git a/packages/services/service-analytics/src/strategies/filter-normalizer.ts b/packages/services/service-analytics/src/strategies/filter-normalizer.ts index e1b2dda16d..12872dd32e 100644 --- a/packages/services/service-analytics/src/strategies/filter-normalizer.ts +++ b/packages/services/service-analytics/src/strategies/filter-normalizer.ts @@ -16,6 +16,32 @@ * Strategies stay simple — they only need to know one shape — and the * spec is honoured: dashboard metadata is authored once in the * canonical MongoDB form and the server normalizes at the boundary. + * + * # Coverage — a dropped predicate WIDENS the query, so nothing is dropped + * + * Failing to map an operator is not "not supporting" it: the predicate simply + * disappears, the compiled SQL stays valid, and the query returns rows the + * author excluded. It reads as a chart drawn over the whole dataset (#3650's + * symptom) and is invisible to any test that asserts the emitted SQL string. + * `$between`, `$startsWith`, `$endsWith` and `$null` each sat broken that way + * (#4128), so what this maps is now a complete capability claim over + * `filter.zod.ts`'s authorable vocabulary: + * + * - mapped 1:1 — `$eq` `$ne` `$gt` `$gte` `$lt` `$lte` `$in` `$nin` + * `$contains` `$notContains` `$startsWith` `$endsWith`; + * - value-DEPENDENT, so resolved explicitly rather than through the map — + * `$null` and `$exists`, whose meaning flips with their boolean; + * - lowered — `$and` (flattened in place), and `$between`, which becomes its + * two bounds so each strategy's existing upper-bound handling applies the + * calendar-day whole-day rule (see the note at the lowering); + * - anything else THROWS. An operator outside the vocabulary is a caller + * error, and a loud one beats a silently widened read — the call + * driver-memory made for the same shape in #3948. + * + * The one remaining gap is declared, not silent: the `$or` / `$not` + * combinators are still skipped, because expressing them needs a recursive + * WHERE builder rather than this flat array. Row-result cover for everything + * above lives in `filter-operator-coverage.test.ts`. */ export interface NormalizedAnalyticsFilter { @@ -24,6 +50,14 @@ export interface NormalizedAnalyticsFilter { values: string[]; } +/** + * The value-INDEPENDENT operators: the pipeline name depends only on the key. + * + * `$null` and `$exists` are deliberately absent — their meaning flips with + * their boolean value, which a key→name map cannot express. Putting `$exists` + * here anyway is what made `{$exists: false}` compile to `IS NOT NULL`, the + * exact inverse of what it asks for; both are handled explicitly below. + */ const MONGO_TO_CUBE_OP: Record = { $eq: 'equals', $ne: 'notEquals', @@ -35,7 +69,8 @@ const MONGO_TO_CUBE_OP: Record = { $nin: 'notIn', $contains: 'contains', $notContains: 'notContains', - $exists: 'set', + $startsWith: 'startsWith', + $endsWith: 'endsWith', }; /** @@ -84,8 +119,67 @@ function flattenCondition(cond: Record, out: NormalizedAnalytic const opKeys = Object.keys(wrapper).filter(k => k.startsWith('$')); if (opKeys.length > 0) { for (const opKey of opKeys) { + // `$between [min, max]` LOWERS to its two bounds rather than getting a + // `between` operator of its own. Both strategies already carry the + // calendar-day whole-day rule on their upper bound — NativeSQLStrategy + // compiles a bare-day `lte` half-open (#3777), ObjectQLStrategy hands + // `$lte` to the driver, which does the same — so a range's max + // inherits that rule by construction instead of needing a second + // implementation to keep in step. (The preview evaluator's `$between` + // gap was closed the same way, sharing its `$lte` helper.) + // + // Before this, `$between` was simply absent from the operator map and + // fell to the `continue` below: the predicate VANISHED from the WHERE + // clause, so a dashboard widget carrying a range filter charted the + // entire dataset — #3650's symptom, on the surface #3650 was about. + // The temporal conformance matrix caught it as row results + // (`native-sql-temporal-conformance.test.ts`). + if (opKey === '$between') { + const v = wrapper[opKey]; + if (!Array.isArray(v) || v.length !== 2) { + // Never drop it: an unbounded read is the failure mode this whole + // branch exists to prevent, and it is indistinguishable from a + // legitimately wide query. Same stance driver-memory took for the + // same shape (#3948). + throw new Error( + `[analytics] "$between" on "${key}" needs a two-element [min, max] array, got ` + + `${JSON.stringify(v)}. Dropping the predicate would silently widen the query to every row.`, + ); + } + out.push({ member: key, operator: 'gte', values: [stringifyForCube(v[0])] }); + out.push({ member: key, operator: 'lte', values: [stringifyForCube(v[1])] }); + continue; + } + + // The two null predicates read their BOOLEAN, not just their key — + // which is why neither can live in MONGO_TO_CUBE_OP. `$null: true` + // asks for IS NULL (`notSet`), `$null: false` for IS NOT NULL + // (`set`); `$exists` is the mirror image. `$null` is the shape the + // console emits for an "is empty" / "is not empty" filter + // (`is_null`/`is_not_null` normalise to it in `filter.zod.ts`), so + // dropping it silently meant such a widget showed every row. + if (opKey === '$null' || opKey === '$exists') { + const isNull = opKey === '$null' ? wrapper[opKey] === true : wrapper[opKey] === false; + out.push({ member: key, operator: isNull ? 'notSet' : 'set', values: [] }); + continue; + } + const cubeOp = MONGO_TO_CUBE_OP[opKey]; - if (!cubeOp) continue; + if (!cubeOp) { + // NEVER drop: a missing predicate does not narrow the query, it + // WIDENS it — the compiled SQL stays valid and simply returns rows + // the author excluded, which is indistinguishable from a + // legitimately broad query and invisible to any test that asserts + // the emitted SQL. That failure mode is #3650's, and skipping + // unmapped operators is how `$between` reproduced it (#4128). + // driver-memory made the same call for the same reason in #3948. + throw new Error( + `[analytics] Unsupported filter operator "${opKey}" on "${key}". ` + + `Supported: ${Object.keys(MONGO_TO_CUBE_OP).join(', ')}, $between, $null, $exists ` + + `(and $and; $or/$not are not yet compiled by the analytics strategies). ` + + `Dropping it would silently widen the query to rows the filter excludes.`, + ); + } const v = wrapper[opKey]; const values = Array.isArray(v) ? v.map(stringifyForCube) diff --git a/packages/services/service-analytics/src/strategies/native-sql-strategy.ts b/packages/services/service-analytics/src/strategies/native-sql-strategy.ts index 8b86d78b10..3b4cb0b05b 100644 --- a/packages/services/service-analytics/src/strategies/native-sql-strategy.ts +++ b/packages/services/service-analytics/src/strategies/native-sql-strategy.ts @@ -484,6 +484,14 @@ export class NativeSQLStrategy implements AnalyticsStrategy { const opMap: Record = { equals: '=', notEquals: '!=', gt: '>', gte: '>=', lt: '<', lte: '<=', contains: 'LIKE', notContains: 'NOT LIKE', + startsWith: 'LIKE', endsWith: 'LIKE', + }; + /** The LIKE pattern each string operator wraps its comparand in. */ + const likePattern: Record string> = { + contains: (v) => `%${v}%`, + notContains: (v) => `%${v}%`, + startsWith: (v) => `${v}%`, + endsWith: (v) => `%${v}`, }; // Null predicates and the LIKE family read the column as stored — the former @@ -504,8 +512,11 @@ export class NativeSQLStrategy implements AnalyticsStrategy { const sqlOp = opMap[operator]; if (!sqlOp || !values || values.length === 0) return null; - if (operator === 'contains' || operator === 'notContains') { - params.push(`%${values[0]}%`); + // The LIKE family reads the column as stored — a substring/prefix/suffix + // match is on the raw text — so it keeps the un-normalised reference. + const pattern = likePattern[operator]; + if (pattern) { + params.push(pattern(values[0])); return `${rawCol} ${sqlOp} $${params.length}`; } diff --git a/packages/services/service-analytics/src/strategies/objectql-strategy.ts b/packages/services/service-analytics/src/strategies/objectql-strategy.ts index 9b2c8955eb..5c21747488 100644 --- a/packages/services/service-analytics/src/strategies/objectql-strategy.ts +++ b/packages/services/service-analytics/src/strategies/objectql-strategy.ts @@ -820,9 +820,25 @@ export class ObjectQLStrategy implements AnalyticsStrategy { case 'lt': return { $lt: v0 }; case 'lte': return { $lte: v0 }; case 'contains': return { $regex: values[0] }; + // `notContains` had no arm and fell to the `default` below, which returns + // a BARE VALUE — i.e. `{field: 'x'}`, an equality. "does not contain x" + // was compiled as "equals x". These three pass through as the canonical + // spec operators every driver implements directly, so an anchored match + // stays anchored rather than depending on regex dialect (#4128). + case 'notContains': return { $notContains: values[0] }; + case 'startsWith': return { $startsWith: values[0] }; + case 'endsWith': return { $endsWith: values[0] }; case 'in': return { $in: all }; case 'notIn': return { $nin: all }; - default: return v0; + default: + // Was `return v0` — a silent reinterpretation of the operator as an + // equality, the write-side twin of the normalizer's dropped predicate + // (#4128). Every operator `normalizeAnalyticsFilters` can emit is + // handled above, so reaching here means the two drifted apart. + throw new Error( + `[analytics] ObjectQL strategy cannot express filter operator "${operator}". ` + + `Treating it as an equality would silently query something the author did not ask for.`, + ); } } diff --git a/packages/spec/api-surface.json b/packages/spec/api-surface.json index 9707b1c325..7e58113a53 100644 --- a/packages/spec/api-surface.json +++ b/packages/spec/api-surface.json @@ -530,6 +530,7 @@ "StateMachineValidationSchema (const)", "StringOperatorSchema (const)", "TEMPORAL_CASES (const)", + "TEMPORAL_NOW (const)", "TEMPORAL_ROWS (const)", "TITLE_ELIGIBLE (const)", "TITLE_ELIGIBLE_TYPES (const)", @@ -537,6 +538,7 @@ "TemporalCase (interface)", "TemporalFieldKind (type)", "TemporalRow (interface)", + "TemporalWriterForm (type)", "TenancyConfig (type)", "TenancyConfigSchema (const)", "TimeUpdateInterval (const)", diff --git a/packages/spec/src/data/temporal-conformance.ts b/packages/spec/src/data/temporal-conformance.ts index d00c76670b..4b47e81eec 100644 --- a/packages/spec/src/data/temporal-conformance.ts +++ b/packages/spec/src/data/temporal-conformance.ts @@ -65,10 +65,54 @@ * shared: both readings include the midnight row, so they agree by luck of the * boundary being inclusive — which is worth pinning precisely because it is * luck. Tracked separately rather than papered over. + * + * # The relative-token axis (D-A3's "token → row results", #4081) + * + * The four incidents happened below token resolution, but the tokens are how + * authors actually WRITE these filters — `{today}`, `{90_days_ago}`, + * `{current_month_end}` — and nothing proved the resolved comparand reaches + * the same rows on every backend. So a case may carry + * {@link TemporalCase.tokenFilter} (and, for the analytics window path, + * {@link TemporalCase.dateRange}): the SAME filter spelled in tokens. + * Consumers that can reach `@objectstack/core` resolve it via + * `resolveFilterTokens` pinned to {@link TEMPORAL_NOW} and must land on the + * same {@link TemporalCase.expected} as the literal spelling — so a resolver + * drift and an evaluator drift are distinguishable at a glance. `formula` + * deliberately depends on nothing but `spec` and skips the token sweep: by the + * time a filter reaches it, tokens are already resolved. + * + * # The writer-form seeding hint (D-E4 `mixed-writer-form`) + * + * Storage form stays out of scope here (see "What belongs here"), but HOW a + * row is written is exactly how #4047 happened: the drivers' mixed columns + * were produced by two writer populations (REST/JSON ISO strings vs SDK + * `Date`s). {@link TemporalRow.writerForm} tags each row so a driver consumer + * can seed a genuinely mixed writer population through its own `create()` — + * the assertion stays "which rows match", the convergence itself remains each + * driver's own suite's claim. */ import type { FilterCondition } from './filter.zod'; +/** + * The pinned reference instant for resolving {@link TemporalCase.tokenFilter} + * and {@link TemporalCase.dateRange}: consumers pass + * `{ now: new Date(TEMPORAL_NOW) }` (UTC, no timezone) to + * `resolveFilterTokens`. Mid-day, so no token resolution sits on a day + * boundary; `{today}` resolves to the fixture's boundary day `2026-07-28`, + * `{90_days_ago}` to `2026-04-29`, `{yesterday}` to `2026-07-27`. + */ +export const TEMPORAL_NOW = '2026-07-28T12:00:00.000Z'; + +/** + * Which write-path shape a driver consumer should seed the row through, where + * the distinction exists (D-E4): `wire` = the ISO-8601 string a REST/JSON + * write delivers; `native` = a JS `Date`, what SDK callers and the drivers' + * own timestamp defaults produce. Both writer populations appear inside AND + * outside every window, so a backend that converges only one form fails. + */ +export type TemporalWriterForm = 'wire' | 'native'; + /** * A row in the temporal fixture. * @@ -83,6 +127,8 @@ export interface TemporalRow { at: string; /** `Field.date` — the calendar day of `at`, timezone-naive. */ on: string; + /** Writer-population seeding hint for drivers — see {@link TemporalWriterForm}. */ + writerForm: TemporalWriterForm; /** Why the row is in the fixture — surfaced when a case fails. */ why: string; } @@ -94,14 +140,15 @@ export interface TemporalRow { * strictly after it, and `d_next` on the next midnight — the exclusive edge. */ export const TEMPORAL_ROWS: readonly TemporalRow[] = [ - { id: 'a_old', at: '2026-04-19T10:00:00.000Z', on: '2026-04-19', why: 'well before any window here' }, - { id: 'b_prev', at: '2026-07-27T14:00:00.000Z', on: '2026-07-27', why: 'the day before the boundary day' }, - { id: 'c_open', at: '2026-07-28T00:00:00.000Z', on: '2026-07-28', why: 'boundary day at exactly 00:00 — the only instant a midnight-anchored bound keeps' }, - { id: 'd_mid', at: '2026-07-28T09:15:00.000Z', on: '2026-07-28', why: 'boundary day, morning — dropped by the #3777 bug' }, - { id: 'e_late', at: '2026-07-28T21:40:00.000Z', on: '2026-07-28', why: 'boundary day, evening — dropped by the #3777 bug' }, - { id: 'f_next', at: '2026-07-29T00:00:00.000Z', on: '2026-07-29', why: 'next midnight — the exclusive edge a half-open bound must NOT keep' }, - { id: 'g_eom', at: '2026-07-31T23:59:59.999Z', on: '2026-07-31', why: 'last representable instant of a month — month rollover' }, - { id: 'h_leap', at: '2024-02-29T12:00:00.000Z', on: '2024-02-29', why: 'leap day — February rollover' }, + { id: 'a_epoch', at: '1969-12-31T23:00:00.000Z', on: '1969-12-31', writerForm: 'wire', why: 'pre-epoch instant (negative epoch ms) — any surface assuming a non-negative epoch, or reading one as a Julian day (#3773), breaks here first' }, + { id: 'a_old', at: '2026-04-19T10:00:00.000Z', on: '2026-04-19', writerForm: 'wire', why: 'well before any window here' }, + { id: 'b_prev', at: '2026-07-27T14:00:00.000Z', on: '2026-07-27', writerForm: 'native', why: 'the day before the boundary day' }, + { id: 'c_open', at: '2026-07-28T00:00:00.000Z', on: '2026-07-28', writerForm: 'native', why: 'boundary day at exactly 00:00 — the only instant a midnight-anchored bound keeps' }, + { id: 'd_mid', at: '2026-07-28T09:15:00.000Z', on: '2026-07-28', writerForm: 'wire', why: 'boundary day, morning — dropped by the #3777 bug' }, + { id: 'e_late', at: '2026-07-28T21:40:00.000Z', on: '2026-07-28', writerForm: 'wire', why: 'boundary day, evening — dropped by the #3777 bug' }, + { id: 'f_next', at: '2026-07-29T00:00:00.000Z', on: '2026-07-29', writerForm: 'native', why: 'next midnight — the exclusive edge a half-open bound must NOT keep' }, + { id: 'g_eom', at: '2026-07-31T23:59:59.999Z', on: '2026-07-31', writerForm: 'wire', why: 'last representable instant of a month — month rollover' }, + { id: 'h_leap', at: '2024-02-29T12:00:00.000Z', on: '2024-02-29', writerForm: 'native', why: 'leap day — February rollover' }, ] as const; /** Which declared field type a case filters on. */ @@ -119,6 +166,17 @@ export interface TemporalCase { field: 'at' | 'on'; kind: TemporalFieldKind; filter: FilterCondition; + /** + * The same filter spelled with relative-date tokens — see "The + * relative-token axis" in the module doc. Resolve against + * {@link TEMPORAL_NOW}; the resolved filter must reach {@link expected}. + */ + tokenFilter?: FilterCondition; + /** + * The analytics `timeDimensions.dateRange` spelling of the same window + * (tokens allowed) for the dashboard-window path — the surface #3650 broke. + */ + dateRange?: [string, string]; /** Ids of matching rows, ascending. */ expected: string[]; /** Why the case is here — surfaced in failure output. */ @@ -137,6 +195,8 @@ export const TEMPORAL_CASES: readonly TemporalCase[] = [ field: 'at', kind: 'datetime', filter: { at: { $gte: '2026-04-29', $lte: '2026-07-28' } }, + tokenFilter: { at: { $gte: '{90_days_ago}', $lte: '{today}' } }, + dateRange: ['{90_days_ago}', '{today}'], expected: ['b_prev', 'c_open', 'd_mid', 'e_late'], note: '#3777: the default dashboard window. Pre-fix returned only b_prev + c_open — everything after 00:00 on the final day vanished.', }, @@ -145,6 +205,8 @@ export const TEMPORAL_CASES: readonly TemporalCase[] = [ field: 'on', kind: 'date', filter: { on: { $gte: '2026-04-29', $lte: '2026-07-28' } }, + tokenFilter: { on: { $gte: '{90_days_ago}', $lte: '{today}' } }, + dateRange: ['{90_days_ago}', '{today}'], expected: ['b_prev', 'c_open', 'd_mid', 'e_late'], note: 'A `date` column compares as calendar-day text, so `<= day` was always right there. The two rows must agree — that is the invariant.', }, @@ -153,7 +215,8 @@ export const TEMPORAL_CASES: readonly TemporalCase[] = [ field: 'at', kind: 'datetime', filter: { at: { $lte: '2026-07-28' } }, - expected: ['a_old', 'b_prev', 'c_open', 'd_mid', 'e_late', 'h_leap'], + tokenFilter: { at: { $lte: '{today}' } }, + expected: ['a_epoch', 'a_old', 'b_prev', 'c_open', 'd_mid', 'e_late', 'h_leap'], note: 'f_next sits exactly on the exclusive edge and must stay out — a half-open bound, never an inclusive 23:59:59.999.', }, { @@ -161,6 +224,7 @@ export const TEMPORAL_CASES: readonly TemporalCase[] = [ field: 'at', kind: 'datetime', filter: { at: { $between: ['2026-04-29', '2026-07-28'] } }, + tokenFilter: { at: { $between: ['{90_days_ago}', '{today}'] } }, expected: ['b_prev', 'c_open', 'd_mid', 'e_late'], note: 'knex whereBetween is inclusive on both ends, so it had the same midnight-anchored upper bound $lte had.', }, @@ -169,6 +233,8 @@ export const TEMPORAL_CASES: readonly TemporalCase[] = [ field: 'at', kind: 'datetime', filter: { at: { $between: ['2026-07-28', '2026-07-28'] } }, + tokenFilter: { at: { $between: ['{today}', '{today}'] } }, + dateRange: ['{today}', '{today}'], expected: ['c_open', 'd_mid', 'e_late'], note: 'The "today" preset degenerates to a single day: the min stays midnight-anchored while the max spans the day.', }, @@ -179,6 +245,7 @@ export const TEMPORAL_CASES: readonly TemporalCase[] = [ field: 'at', kind: 'datetime', filter: { at: { $gte: '2026-07-28' } }, + tokenFilter: { at: { $gte: '{today}' } }, expected: ['c_open', 'd_mid', 'e_late', 'f_next', 'g_eom'], note: 'A LOWER bound anchors to 00:00 — c_open is included precisely because the bound is inclusive of that instant.', }, @@ -187,7 +254,8 @@ export const TEMPORAL_CASES: readonly TemporalCase[] = [ field: 'at', kind: 'datetime', filter: { at: { $lt: '2026-07-28' } }, - expected: ['a_old', 'b_prev', 'h_leap'], + tokenFilter: { at: { $lt: '{today}' } }, + expected: ['a_epoch', 'a_old', 'b_prev', 'h_leap'], note: 'Strict-less keeps its anchoring: c_open is AT midnight, so it is excluded.', }, { @@ -197,6 +265,7 @@ export const TEMPORAL_CASES: readonly TemporalCase[] = [ field: 'on', kind: 'date', filter: { on: { $gt: '2026-07-28' } }, + tokenFilter: { on: { $gt: '{today}' } }, expected: ['f_next', 'g_eom'], note: 'The boundary day itself is out; the mirror-image of widening a lower bound.', }, @@ -207,7 +276,7 @@ export const TEMPORAL_CASES: readonly TemporalCase[] = [ field: 'at', kind: 'datetime', filter: { at: { $lte: '2026-07-28T12:00:00.000Z' } }, - expected: ['a_old', 'b_prev', 'c_open', 'd_mid', 'h_leap'], + expected: ['a_epoch', 'a_old', 'b_prev', 'c_open', 'd_mid', 'h_leap'], note: 'Only the day-granular STRING carries whole-day intent. e_late (21:40) is after noon and must stay out.', }, { @@ -225,6 +294,8 @@ export const TEMPORAL_CASES: readonly TemporalCase[] = [ field: 'at', kind: 'datetime', filter: { at: { $gte: '2026-07-29', $lte: '2026-07-31' } }, + tokenFilter: { at: { $gte: '{tomorrow}', $lte: '{current_month_end}' } }, + dateRange: ['{tomorrow}', '{current_month_end}'], expected: ['f_next', 'g_eom'], note: 'g_eom is 23:59:59.999 on the 31st — kept only if the bound rolled to 2026-08-01.', }, @@ -244,4 +315,32 @@ export const TEMPORAL_CASES: readonly TemporalCase[] = [ expected: [], note: 'Guards the opposite error: rolling 02-28 to 03-01 (skipping the 29th) would wrongly include h_leap.', }, + { + name: 'datetime: a pre-epoch day is an ordinary calendar day', + field: 'at', + kind: 'datetime', + filter: { at: { $gte: '1969-12-31', $lte: '1969-12-31' } }, + expected: ['a_epoch'], + note: 'Negative epoch ms. The #3773 family: any surface that assumes a datetime is a non-negative epoch, or reads one as a Julian day, breaks here first.', + }, + + // ── Equality on the `date` column — the ADR's original defect (#1874) ──── + { + name: 'date: equality against a resolved day matches the whole day', + field: 'on', + kind: 'date', + filter: { on: '2026-07-28' }, + tokenFilter: { on: '{today}' }, + expected: ['c_open', 'd_mid', 'e_late'], + note: '#1874: `date == today` silently matched nothing while dates were stored as instants. Equality on a date column is plain calendar-day text equality — Phase 1\'s whole point.', + }, + { + name: 'date: $in of two resolved days', + field: 'on', + kind: 'date', + filter: { on: { $in: ['2026-07-28', '2026-07-27'] } }, + tokenFilter: { on: { $in: ['{today}', '{yesterday}'] } }, + expected: ['b_prev', 'c_open', 'd_mid', 'e_late'], + note: 'The `expires_on: { $in: [daysFromNow(30)] }` template shape from the #1874 family — element-wise, order-independent.', + }, ] as const;