diff --git a/.changeset/filter-tokens-runtime-resolver.md b/.changeset/filter-tokens-runtime-resolver.md new file mode 100644 index 0000000000..dc29edc8ab --- /dev/null +++ b/.changeset/filter-tokens-runtime-resolver.md @@ -0,0 +1,66 @@ +--- +"@objectstack/core": minor +"@objectstack/objectql": minor +"@objectstack/service-analytics": minor +"@objectstack/service-automation": patch +"@objectstack/spec": patch +--- + +feat(filters): evaluate `{filter-token}` placeholders server-side (#3582) + +Filter values travel as JSON, so a time- or user-scoped slice writes a +placeholder instead of code: + +```ts +filter: { close_date: { $gte: '{current_year_start}' }, owner: '{current_user_id}' } +``` + +The vocabulary has been in `@objectstack/spec` for a while (`date-macros.zod.ts`, +`context-tokens.zod.ts`) and `objectstack build` rejects tokens outside it +(#3574). What was missing is the half that *substitutes a value*: **nothing on +the server ever did**. A placeholder reached the driver as the literal string +`'{current_year_start}'`, compared as text, and matched nothing. + +That failure is invisible — an empty widget looks exactly like a metric that is +legitimately zero — so apps worked around it by computing dates at module load, +which freezes "this year" into the built artifact and quietly goes stale. + +**New: `resolveFilterTokens()` in `@objectstack/core`**, wired into the two +server-side seams every filter passes through: + +- **ObjectQL read path** — `find` / `findOne` / `count` / `aggregate`, so REST + queries, related lists, saved-view filters and flow `find_records` all resolve. + It runs before the middleware chain, so only author-supplied filters are + inspected; RLS/sharing filters are injected downstream from concrete values. +- **Analytics dataset executor** — a dataset's intrinsic `filter`, a widget's + `runtimeFilter`, measure-scoped filters, and time-dimension `dateRange`s. + This path needs its own call: `NativeSQLStrategy` compiles raw SQL and binds + comparands directly, so a dashboard widget never passes through `engine.find()`. + +Behavioural notes: + +- Date tokens resolve to ISO strings (`YYYY-MM-DD`, or a full timestamp for + `{now}` / `{N_hours_ago}` / `{N_minutes_ago}`). Turning that into a column's + on-disk form stays the driver's job (`SqlDriver.temporalFilterValue`), so + there is still exactly one source of truth for the storage convention. +- Calendar boundaries follow `ExecutionContext.timezone`; one instant is pinned + per filter tree, so a `>= {current_month_start}` / `< {next_month_start}` pair + can never straddle a boundary. +- `{current_org_id}` reads `ExecutionContext.tenantId`; `{current_user_id}` reads + `userId`. A request carrying neither now **throws** instead of resolving to + `null` — a null comparand degrades to `IS NULL` on most drivers and would hand + back the rows the filter was written to exclude. +- An unrecognised placeholder **throws**, carrying the near-miss fix + (`{current_user}` → `{current_user_id}`, `{this_quarter_start}` → + `{current_quarter_start}`). This matches what `objectstack build` already + enforces. Consequence, previously implicit and now load-bearing: a filter value + that is *entirely* `{...}` is always read as a placeholder, so a literal value + of that shape is not expressible — rename the value. + +Also in this change: `notify` no longer sends the six-character string +`"undefined"` as an audience member. `to: ['{record.owner.manager}']` walks +`.manager` on a scalar foreign-key id, resolves to nothing, and `String(undefined)` +turned that into a phantom recipient — the emit "succeeded", addressed nobody, +and said nothing. Unresolved recipients are now dropped, and a node with no +recipient left fails naming the offending template and pointing at the start +node's `config.expand` (#3475), which does hydrate the relation. diff --git a/content/docs/data-modeling/analytics.mdx b/content/docs/data-modeling/analytics.mdx index b5528d5cd0..9659eed895 100644 --- a/content/docs/data-modeling/analytics.mdx +++ b/content/docs/data-modeling/analytics.mdx @@ -122,8 +122,15 @@ export const SalesByStageReport = { ``` `rows` are the pivot's down-axis dimensions; `values` are measure names. A matrix -report adds across-axis dimensions; `runtimeFilter` is the render-time scope -(`{date-macro}` placeholders are resolved by the renderer before querying). +report adds across-axis dimensions; `runtimeFilter` is the render-time scope. + +Placeholders inside a `filter` / `runtimeFilter` — `{date-macro}`s and the +session tokens `{current_user_id}` / `{current_org_id}` — are resolved on **both** +sides: by the renderer before it queries, and by the analytics dataset executor +server-side. The server pass is what makes a widget work at all, since the +native-SQL strategy compiles raw SQL and binds comparands directly without ever +passing through a renderer. An unrecognised placeholder is a build error and a +runtime error — never a literal string that quietly matches nothing. ## Cross-object joins diff --git a/content/docs/references/data/context-tokens.mdx b/content/docs/references/data/context-tokens.mdx index 7351264066..957a1e3057 100644 --- a/content/docs/references/data/context-tokens.mdx +++ b/content/docs/references/data/context-tokens.mdx @@ -21,13 +21,29 @@ slice cannot call `currentUser().id` inline — it writes a placeholder: [\{ field: 'owner', operator: 'equals', value: '\{current_user_id\}' \}] -Like date macros, the placeholders are expanded **client-side** by +Like date macros, the placeholders are expanded on **both** sides of -`resolveContextTokens()` in `@object-ui/core` immediately before the +the wire (framework#3582): `resolveContextTokens()` in -filter is handed to the data source. The data engine only ever sees +`@object-ui/core` before the filter leaves the browser, and -concrete ids, never `\{tokens\}`. +`resolveFilterTokens()` in `@objectstack/core` on the ObjectQL read + +path and the analytics dataset executor for filters that reach the + +database without passing through a renderer. The DRIVER only ever + +sees concrete ids, never `\{tokens\}`. + +The server resolver reads `ExecutionContext` — `\{current_user_id\}` is + +`userId`, `\{current_org_id\}` is `tenantId`. A request that carries + +neither is an ERROR, not a null comparand: resolving to `null` + +degrades to `IS NULL` on most drivers and would hand back the rows + +the filter was written to exclude. # Presentation scope, NOT a security boundary diff --git a/content/docs/references/data/date-macros.mdx b/content/docs/references/data/date-macros.mdx index 410cbf97d4..b1f68800a1 100644 --- a/content/docs/references/data/date-macros.mdx +++ b/content/docs/references/data/date-macros.mdx @@ -21,13 +21,41 @@ inline; they use a tiny placeholder grammar instead: \{ signal_at: \{ $gte: '\{30_days_ago\}' \} \} -The placeholders are expanded **client-side** by +The placeholders are expanded on **both** sides of the wire, so a -`resolveDateMacros()` in `@object-ui/core` immediately before the +filter behaves the same wherever it is executed (framework#3582): -filter is handed to the data source. The data engine itself only +- **Client** — `resolveDateMacros()` in `@object-ui/core`, just -sees ISO date / timestamp strings, never `\{tokens\}`. +before the filter is handed to the data source. + +- **Server** — `resolveFilterTokens()` in `@objectstack/core`, wired + +into the ObjectQL read path (`find`/`findOne`/`count`/`aggregate`) + +and the analytics dataset executor. Filters that reach the database + +WITHOUT passing through a renderer — dashboard widgets, dataset + +definitions, REST query params — need this: before it, the token + +compared as a literal string and matched nothing. + +Either way the DRIVER only ever sees ISO date / timestamp strings, + +never `\{tokens\}`. Translating an ISO comparand into a column's on-disk + +form (SQLite epoch-ms, `YYYY-MM-DD` text, native timestamp) is the + +driver's job — see `SqlDriver.temporalFilterValue`. + +A token OUTSIDE this vocabulary is rejected rather than passed + +through: `@objectstack/lint`'s `validate-filter-tokens` fails the + +build, and the runtime resolver throws. Silently matching nothing is + +the failure mode the vocabulary exists to prevent. AI agents and template authors author these placeholders directly, @@ -65,7 +93,15 @@ formula engine. They are unrelated to these placeholders; see calendars) are defined by the resolver implementation; spec only -freezes the **vocabulary**. +freezes the **vocabulary**. One property is worth stating here + +because it is authored against: a `*_end` token is the period's + +last calendar DAY (`\{current_year_end\}` → `2026-12-31`), so on a + +`datetime` column `<= \{current_year_end\}` stops at midnight on the + +31st. Filter a timestamp with the half-open `< \{next_year_start\}`. **Source:** `packages/spec/src/data/date-macros.zod.ts` diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index bc6b78035b..372885d11c 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -29,6 +29,9 @@ export * from './utils/datetime.js'; // Export the shared batched-write helper (framework#2678) export * from './utils/bulk-write.js'; +// Export the runtime filter-placeholder resolver (framework#3582) +export * from './utils/filter-tokens.js'; + // Export in-memory fallbacks for core-criticality services export * from './fallbacks/index.js'; diff --git a/packages/core/src/utils/filter-tokens.test.ts b/packages/core/src/utils/filter-tokens.test.ts new file mode 100644 index 0000000000..623d89cae9 --- /dev/null +++ b/packages/core/src/utils/filter-tokens.test.ts @@ -0,0 +1,247 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import { describe, it, expect } from 'vitest'; +import { + resolveFilterToken, + resolveFilterTokens, + filterTokenContextFrom, + UnknownFilterTokenError, + UnresolvedFilterTokenError, +} from './filter-tokens.js'; + +// A fixed instant so every expectation is deterministic: Wed 2026-07-15 18:30 UTC. +// Mid-month, mid-quarter (Q3), mid-week (Wednesday) — so week/month/quarter/year +// boundaries are all strictly inside the year and none coincide. +const NOW = new Date('2026-07-15T18:30:00.000Z'); + +describe('resolveFilterToken — date macros (framework#3582)', () => { + const at = (token: string, extra = {}) => resolveFilterToken(token, { now: NOW, ...extra }); + + it('resolves the instant tokens', () => { + expect(at('today')).toBe('2026-07-15'); + expect(at('yesterday')).toBe('2026-07-14'); + expect(at('tomorrow')).toBe('2026-07-16'); + expect(at('now')).toBe('2026-07-15T18:30:00.000Z'); + }); + + it('resolves current/last/next period starts', () => { + expect(at('current_week_start')).toBe('2026-07-13'); // Monday + expect(at('current_month_start')).toBe('2026-07-01'); + expect(at('current_quarter_start')).toBe('2026-07-01'); // Q3 + expect(at('current_year_start')).toBe('2026-01-01'); + + expect(at('last_month_start')).toBe('2026-06-01'); + expect(at('last_quarter_start')).toBe('2026-04-01'); + expect(at('last_year_start')).toBe('2025-01-01'); + + expect(at('next_month_start')).toBe('2026-08-01'); + expect(at('next_quarter_start')).toBe('2026-10-01'); + expect(at('next_year_start')).toBe('2027-01-01'); + }); + + it('resolves period ends to the LAST CALENDAR DAY of the period', () => { + // Documented half-open-range property: `_end` is a day, not the final + // instant. A `datetime` column wants `< {next_*_start}` instead. + expect(at('current_week_end')).toBe('2026-07-19'); // Sunday + expect(at('current_month_end')).toBe('2026-07-31'); + expect(at('current_quarter_end')).toBe('2026-09-30'); + expect(at('current_year_end')).toBe('2026-12-31'); + expect(at('last_month_end')).toBe('2026-06-30'); + expect(at('last_year_end')).toBe('2025-12-31'); + }); + + it('treats the bare aliases as the current period', () => { + expect(at('week_start')).toBe(at('current_week_start')); + expect(at('month_end')).toBe(at('current_month_end')); + expect(at('quarter_start')).toBe(at('current_quarter_start')); + expect(at('year_end')).toBe(at('current_year_end')); + }); + + it('resolves the parameterised grammar', () => { + expect(at('30_days_ago')).toBe('2026-06-15'); + expect(at('1_day_ago')).toBe('2026-07-14'); + expect(at('2_weeks_ago')).toBe('2026-07-01'); + expect(at('12_months_ago')).toBe('2025-07-15'); + expect(at('1_year_ago')).toBe('2025-07-15'); + expect(at('90_days_from_now')).toBe('2026-10-13'); + }); + + it('keeps sub-day units as full instants', () => { + // A `{15_minutes_ago}` truncated to a date would silently widen the window + // to the whole day — the exact class of bug this module exists to end. + expect(at('15_minutes_ago')).toBe('2026-07-15T18:15:00.000Z'); + expect(at('2_hours_from_now')).toBe('2026-07-15T20:30:00.000Z'); + }); + + it('anchors the calendar in the reference timezone', () => { + // 18:30 UTC is already 2026-07-16 in Tokyo (+09:00) but still the 15th in + // New York (-04:00). The token must follow the request's reference zone. + expect(at('today', { timezone: 'Asia/Tokyo' })).toBe('2026-07-16'); + expect(at('today', { timezone: 'America/New_York' })).toBe('2026-07-15'); + // …and the period boundaries move with it. + expect(at('current_week_start', { timezone: 'Asia/Tokyo' })).toBe('2026-07-13'); + }); + + it('falls back to the UTC calendar for an unknown zone', () => { + expect(at('today', { timezone: 'Mars/Olympus_Mons' })).toBe('2026-07-15'); + }); + + it('returns undefined for a token outside the vocabulary', () => { + expect(at('current_user')).toBeUndefined(); + expect(at('this_quarter_start')).toBeUndefined(); + }); + + describe('end-of-month reference dates (calendar overflow)', () => { + // The trap: naive `setUTCMonth(m - 1)` on the 31st rolls FORWARD ("Feb 31" + // = Mar 3), so "last month" reads back as the current month and + // `{1_month_ago}` lands AFTER today. Five days a year, silently. + const eom = (token: string) => + resolveFilterToken(token, { now: new Date('2026-03-31T12:00:00.000Z') }); + + it('last_month_* stays in February', () => { + expect(eom('last_month_start')).toBe('2026-02-01'); + expect(eom('last_month_end')).toBe('2026-02-28'); + }); + + it('current/next month boundaries are unaffected', () => { + expect(eom('current_month_start')).toBe('2026-03-01'); + expect(eom('current_month_end')).toBe('2026-03-31'); + expect(eom('next_month_start')).toBe('2026-04-01'); + expect(eom('next_month_end')).toBe('2026-04-30'); + }); + + it('quarter boundaries step by whole quarters', () => { + expect(eom('current_quarter_start')).toBe('2026-01-01'); + expect(eom('last_quarter_start')).toBe('2025-10-01'); + expect(eom('last_quarter_end')).toBe('2025-12-31'); + }); + + it('{1_month_ago} clamps to the last day of the shorter month', () => { + expect(eom('1_month_ago')).toBe('2026-02-28'); + expect(eom('1_month_from_now')).toBe('2026-04-30'); + }); + + it('a leap day one year out clamps rather than rolling into March', () => { + expect(resolveFilterToken('1_year_from_now', { now: new Date('2028-02-29T00:00:00.000Z') })) + .toBe('2029-02-28'); + }); + }); +}); + +describe('resolveFilterToken — context tokens', () => { + it('resolves the signed-in user and active org', () => { + const ctx = { now: NOW, userId: 'usr_1', orgId: 'org_9' }; + expect(resolveFilterToken('current_user_id', ctx)).toBe('usr_1'); + expect(resolveFilterToken('current_org_id', ctx)).toBe('org_9'); + }); + + it('throws rather than resolving to null when the request has no user', () => { + // Resolving to null/undefined degrades to `IS NULL` on most drivers, which + // hands back rows the filter was written to exclude. + expect(() => resolveFilterToken('current_user_id', { now: NOW })) + .toThrow(UnresolvedFilterTokenError); + expect(() => resolveFilterToken('current_org_id', { now: NOW })) + .toThrow(UnresolvedFilterTokenError); + }); +}); + +describe('resolveFilterTokens — tree walk', () => { + const ctx = { now: NOW, userId: 'usr_1', orgId: 'org_9' }; + + it('expands placeholders in a MongoDB-style filter', () => { + expect( + resolveFilterTokens( + { close_date: { $gte: '{current_year_start}' }, owner: '{current_user_id}' }, + ctx, + ), + ).toEqual({ close_date: { $gte: '2026-01-01' }, owner: 'usr_1' }); + }); + + it('expands placeholders in the array/rule filter shape', () => { + expect( + resolveFilterTokens( + [{ field: 'close_date', operator: 'greater_than', value: '{current_year_start}' }], + ctx, + ), + ).toEqual([{ field: 'close_date', operator: 'greater_than', value: '2026-01-01' }]); + }); + + it('descends logical branches', () => { + expect( + resolveFilterTokens( + { $and: [{ a: '{today}' }, { $or: [{ b: '{30_days_ago}' }, { c: 1 }] }] }, + ctx, + ), + ).toEqual({ $and: [{ a: '2026-07-15' }, { $or: [{ b: '2026-06-15' }, { c: 1 }] }] }); + }); + + it('leaves non-placeholder values alone, including embedded braces', () => { + const filter = { name: 'order {id}', qty: 3, flag: true, at: new Date(0) }; + expect(resolveFilterTokens(filter, ctx)).toBe(filter); + }); + + it('returns the input by reference when nothing resolves', () => { + // The no-placeholder path is every internal query; it must not allocate. + const filter = { status: 'open', $and: [{ qty: { $gt: 2 } }] }; + expect(resolveFilterTokens(filter, ctx)).toBe(filter); + }); + + it('never mutates the caller — metadata is shared across requests', () => { + const filter = { owner: '{current_user_id}' }; + const first = resolveFilterTokens(filter, { ...ctx, userId: 'usr_a' }); + const second = resolveFilterTokens(filter, { ...ctx, userId: 'usr_b' }); + expect(filter).toEqual({ owner: '{current_user_id}' }); + expect(first).toEqual({ owner: 'usr_a' }); + expect(second).toEqual({ owner: 'usr_b' }); + }); + + it('shares ONE instant across every token in a tree', () => { + // Straddling boundaries between two `new Date()` calls would silently drop + // rows from a `>= start` / `< next start` pair. + const out = resolveFilterTokens( + { $and: [{ a: '{now}' }, { b: '{now}' }] }, + { userId: 'usr_1' }, + ) as { $and: Array> }; + expect(out.$and[0].a).toBe(out.$and[1].b); + }); + + it('throws on an unknown placeholder and names the near miss', () => { + // The #3582 headline: `{current_user}` used to reach SQL as a literal. + let err: unknown; + try { + resolveFilterTokens({ owner: '{current_user}' }, ctx); + } catch (e) { + err = e; + } + expect(err).toBeInstanceOf(UnknownFilterTokenError); + expect((err as UnknownFilterTokenError).token).toBe('current_user'); + expect((err as Error).message).toContain('{current_user_id}'); + // 400, not 500 — a malformed filter is the caller's to fix, and the REST + // layer's generic 4xx passthrough keys on these two fields. + expect((err as UnknownFilterTokenError).status).toBe(400); + expect((err as UnknownFilterTokenError).code).toBe('FILTER_TOKEN_UNKNOWN'); + }); + + it('throws on a near-miss period spelling', () => { + expect(() => resolveFilterTokens({ d: '{this_quarter_start}' }, ctx)) + .toThrow(UnknownFilterTokenError); + }); + + it('accepts the `${token}` spelling', () => { + expect(resolveFilterTokens({ d: '${30_days_ago}' }, ctx)).toEqual({ d: '2026-06-15' }); + }); +}); + +describe('filterTokenContextFrom', () => { + it('maps ExecutionContext onto the resolver inputs', () => { + expect( + filterTokenContextFrom({ userId: 'usr_1', tenantId: 'org_9', timezone: 'Asia/Tokyo' }, NOW), + ).toEqual({ now: NOW, userId: 'usr_1', orgId: 'org_9', timezone: 'Asia/Tokyo' }); + }); + + it('tolerates a missing context', () => { + expect(filterTokenContextFrom(undefined)).toEqual({ + now: undefined, userId: undefined, orgId: undefined, timezone: undefined, + }); + }); +}); diff --git a/packages/core/src/utils/filter-tokens.ts b/packages/core/src/utils/filter-tokens.ts new file mode 100644 index 0000000000..b29acf199b --- /dev/null +++ b/packages/core/src/utils/filter-tokens.ts @@ -0,0 +1,408 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * Runtime resolution of filter placeholders — the server-side half of the + * `{token}` contract that `@objectstack/spec` declares (framework#3582). + * + * `date-macros.zod.ts` and `context-tokens.zod.ts` freeze the *vocabulary*; + * `@objectstack/lint`'s `validate-filter-tokens` rejects a token outside it at + * authoring time. Neither one ever substituted a value: every server-side + * consumer handed the literal `'{current_year_start}'` to the database, where + * it compared as a string and matched nothing. The failure was invisible — + * an empty widget, an unfiltered list — so apps worked around it by computing + * dates at module load, freezing "this year" into the built artifact. + * + * This module is the missing evaluator. It walks a filter tree and replaces + * every fully-wrapped placeholder with a concrete value: + * + * { close_date: { $gte: '{current_year_start}' } } + * → { close_date: { $gte: '2026-01-01' } } + * { owner: '{current_user_id}' } + * → { owner: 'usr_7f3a…' } + * + * # Output form: ISO strings, not driver-native values + * + * Date tokens resolve to `YYYY-MM-DD` (or a full ISO timestamp for the + * sub-day tokens `{now}` / `{N_hours_ago}` / `{N_minutes_ago}`), exactly the + * form the spec's module doc promises the data engine sees. Translating that + * to a column's on-disk form is the DRIVER's job and already exists — + * `SqlDriver.coerceFilterValue` / `temporalFilterValue` turn an ISO comparand + * into SQLite epoch-ms or leave it alone on native-timestamp dialects. Emitting + * a driver-native value here would fork that convention into a second source of + * truth and break the moment a query crosses datasources. + * + * # Period `_end` is the last calendar DAY, not the last instant + * + * `{current_year_end}` is `2026-12-31`, per the spec's own + * `DATE_MACRO_DESCRIPTIONS` ("Dec 31 of this year"). On a `datetime` column + * that means `<= {current_year_end}` excludes everything after midnight on the + * 31st — the classic half-open-range trap. Authors filtering a timestamp want + * `< {next_year_start}`. This is a documented property of the vocabulary, not + * something the resolver may quietly "fix": silently widening a bound would + * make the same token mean different things on different column types. + * + * # An unknown token throws + * + * A value that is entirely `{something}` is a placeholder by construction — no + * author means the literal six characters `{foo}`. Passing an unrecognised one + * through is precisely the silent-zero bug this module exists to end, so it is + * a hard error carrying the near-miss suggestion (`{current_user}` → + * `{current_user_id}`). Values that merely CONTAIN braces are left untouched. + */ + +import { + classifyFilterToken, + parseDateMacroParam, + type DateMacroUnit, +} from '@objectstack/spec/data'; +import { calendarPartsInTzOrUtc } from './datetime.js'; + +/** + * The slice of an execution context the resolver reads. Structural on purpose — + * see {@link filterTokenContextFrom}. + */ +export interface ExecutionContextLike { + readonly userId?: string; + readonly tenantId?: string; + readonly timezone?: string; +} + +/** + * The request-scoped values a placeholder can resolve against. + * + * `now` is captured ONCE per resolve call so every token in one filter shares + * an instant — otherwise a `$gte {current_month_start}` / `$lt + * {next_month_start}` pair evaluated microseconds apart could straddle a + * month boundary and silently drop a row. + */ +export interface FilterTokenResolutionContext { + /** Reference instant. Defaults to `new Date()` at call time. */ + now?: Date; + /** IANA reference timezone for calendar boundaries. Defaults to UTC. */ + timezone?: string; + /** Resolves `{current_user_id}`. */ + userId?: string; + /** Resolves `{current_org_id}`. */ + orgId?: string; +} + +/** + * Raised when a filter carries a placeholder outside the vocabulary. + * + * Carries `status`/`code` so the REST layer's generic 4xx passthrough maps it + * to a **400 with a fixable message** rather than a 500: the caller's filter is + * malformed, the server is fine. (Same convention plugin-sharing uses for its + * record-scope denial — no runtime dependency in either direction.) + */ +export class UnknownFilterTokenError extends Error { + readonly token: string; + readonly suggestion?: string; + readonly status = 400; + readonly code = 'FILTER_TOKEN_UNKNOWN'; + + constructor(token: string, suggestion?: string) { + super( + `Unresolvable filter placeholder "{${token}}". ` + + (suggestion + ? `Did you mean "{${suggestion}}"? ` + : 'Resolvable placeholders are the context tokens ({current_user_id}, ' + + '{current_org_id}) and the date macros ({today}, {current_quarter_start}, ' + + '{30_days_ago}, …). ') + + 'Sending it to the data engine verbatim would compare it as a literal ' + + 'string and match nothing, which is indistinguishable from an empty result.', + ); + this.name = 'UnknownFilterTokenError'; + this.token = token; + this.suggestion = suggestion; + } +} + +/** + * Raised when a token IS in the vocabulary but the request carries no value + * for it — an unauthenticated caller filtering on `{current_user_id}`. + * + * Distinct from {@link UnknownFilterTokenError} because the fix is different: + * the metadata is correct, the context is not. Never silently resolves to + * `null`/`undefined`, which on most drivers degrades to `IS NULL` and would + * quietly hand back rows the filter was written to exclude. + */ +export class UnresolvedFilterTokenError extends Error { + readonly token: string; + /** 400, not 500 — see {@link UnknownFilterTokenError}. */ + readonly status = 400; + readonly code = 'FILTER_TOKEN_UNRESOLVED'; + + constructor(token: string, detail: string) { + super(`Filter placeholder "{${token}}" cannot be resolved: ${detail}`); + this.name = 'UnresolvedFilterTokenError'; + this.token = token; + } +} + +/** `YYYY-MM-DD` for a calendar day, zero-padded. */ +function ymd(year: number, month: number, day: number): string { + const p = (n: number) => String(n).padStart(2, '0'); + return `${year}-${p(month)}-${p(day)}`; +} + +/** + * Calendar arithmetic is done on a UTC "proxy" date built from the reference + * timezone's calendar parts. Working in UTC keeps the math free of DST jumps + * (a local-midnight `Date` can shift by an hour when `setMonth` crosses a + * transition); the zone only decides WHICH calendar day "now" is, which + * {@link calendarPartsInTzOrUtc} answers from the platform tz database. + */ +function proxyDay(now: Date, timezone?: string): Date { + const { year, month, day } = calendarPartsInTzOrUtc(now, timezone); + return new Date(Date.UTC(year, month - 1, day)); +} + +const asYmd = (d: Date): string => ymd(d.getUTCFullYear(), d.getUTCMonth() + 1, d.getUTCDate()); + +type PeriodKind = 'week' | 'month' | 'quarter' | 'year'; + +/** Monday-based week start — matches the spec's "Monday 00:00 of this week". */ +function startOfPeriod(kind: PeriodKind, d: Date): Date { + const r = new Date(d.getTime()); + switch (kind) { + case 'week': { + const dow = (r.getUTCDay() + 6) % 7; // 0 = Monday + r.setUTCDate(r.getUTCDate() - dow); + return r; + } + case 'month': + return new Date(Date.UTC(r.getUTCFullYear(), r.getUTCMonth(), 1)); + case 'quarter': + return new Date(Date.UTC(r.getUTCFullYear(), Math.floor(r.getUTCMonth() / 3) * 3, 1)); + case 'year': + return new Date(Date.UTC(r.getUTCFullYear(), 0, 1)); + } +} + +/** Days in the given (0-based) month of `year`. */ +function daysInMonth(year: number, month: number): number { + return new Date(Date.UTC(year, month + 1, 0)).getUTCDate(); +} + +/** + * Shift `d` by `n` months, CLAMPING the day to the target month's length. + * + * Bare `setUTCMonth(m - 1)` on the 31st rolls FORWARD into the following month + * (Mar 31 minus one month = "Feb 31" = Mar 3), which would make + * `{1_month_ago}` land after `{today}` on five days of the year. Clamping to + * Feb 28 is what every calendar library does and the only answer an author + * would call correct. + */ +function addMonthsClamped(d: Date, n: number): Date { + const year = d.getUTCFullYear(); + const month = d.getUTCMonth() + n; + const targetYear = year + Math.floor(month / 12); + const targetMonth = ((month % 12) + 12) % 12; + const day = Math.min(d.getUTCDate(), daysInMonth(targetYear, targetMonth)); + return new Date(Date.UTC( + targetYear, targetMonth, day, + d.getUTCHours(), d.getUTCMinutes(), d.getUTCSeconds(), d.getUTCMilliseconds(), + )); +} + +/** Shift `d` by `n` whole periods of `kind` (negative shifts backwards). */ +function addPeriods(kind: PeriodKind, d: Date, n: number): Date { + switch (kind) { + case 'week': { + const r = new Date(d.getTime()); + r.setUTCDate(r.getUTCDate() + n * 7); + return r; + } + case 'month': return addMonthsClamped(d, n); + case 'quarter': return addMonthsClamped(d, n * 3); + case 'year': return addMonthsClamped(d, n * 12); + } +} + +/** Shift `d` by `n` units of the parameterised grammar. */ +function addUnits(unit: DateMacroUnit, d: Date, n: number): Date { + const r = new Date(d.getTime()); + switch (unit) { + case 'minute': r.setUTCMinutes(r.getUTCMinutes() + n); return r; + case 'hour': r.setUTCHours(r.getUTCHours() + n); return r; + case 'day': r.setUTCDate(r.getUTCDate() + n); return r; + case 'week': r.setUTCDate(r.getUTCDate() + n * 7); return r; + // Month/year steps clamp rather than overflow — see addMonthsClamped. + case 'month': return addMonthsClamped(d, n); + case 'year': return addMonthsClamped(d, n * 12); + } +} + +/** + * `current|last|next` × `week|month|quarter|year` × `start|end`, plus the bare + * `week_start`-style aliases (which mean `current_`). Returns `undefined` when + * the token is not a period token. + */ +const PERIOD_RE = /^(?:(current|last|next)_)?(week|month|quarter|year)_(start|end)$/; + +function resolvePeriodToken(token: string, today: Date): string | undefined { + const m = PERIOD_RE.exec(token); + if (!m) return undefined; + const rel = (m[1] ?? 'current') as 'current' | 'last' | 'next'; + const kind = m[2] as PeriodKind; + const bound = m[3] as 'start' | 'end'; + const offset = rel === 'last' ? -1 : rel === 'next' ? 1 : 0; + + // Normalize to the period's own start BEFORE stepping. Day 1 of a + // month/quarter/year (and a Monday) shifts exactly — no month-length clamping + // is involved at all — so the answer never depends on the clamp policy, and + // the arithmetic reads the same for every `kind`. + const periodStart = startOfPeriod(kind, addPeriods(kind, startOfPeriod(kind, today), offset)); + if (bound === 'start') return asYmd(periodStart); + // `_end` = the last calendar DAY of the period: the day before the next + // period begins. See the module doc on half-open ranges. + const next = addPeriods(kind, periodStart, 1); + next.setUTCDate(next.getUTCDate() - 1); + return asYmd(next); +} + +/** + * Resolve one token NAME (the bit inside the braces) to its concrete value. + * Throws {@link UnresolvedFilterTokenError} for a vocabulary token the request + * carries no value for. Returns `undefined` only when the token is outside the + * vocabulary — callers turn that into {@link UnknownFilterTokenError}. + */ +export function resolveFilterToken( + token: string, + ctx: FilterTokenResolutionContext = {}, +): unknown { + const now = ctx.now ?? new Date(); + + // ── Context tokens ──────────────────────────────────────────────────── + if (token === 'current_user_id') { + if (!ctx.userId) { + throw new UnresolvedFilterTokenError( + token, + 'the request has no authenticated user. A filter scoped to the signed-in ' + + 'user cannot run for an anonymous or system caller — gate the surface on ' + + 'authentication, or drop the token from the filter.', + ); + } + return ctx.userId; + } + if (token === 'current_org_id') { + if (!ctx.orgId) { + throw new UnresolvedFilterTokenError( + token, + 'the request carries no active organization (ExecutionContext.tenantId is ' + + 'unset). Set the active org on the request, or drop the token from the filter.', + ); + } + return ctx.orgId; + } + + // ── Date macros ─────────────────────────────────────────────────────── + const today = proxyDay(now, ctx.timezone); + + switch (token) { + case 'now': return now.toISOString(); + case 'today': return asYmd(today); + case 'yesterday': return asYmd(addUnits('day', today, -1)); + case 'tomorrow': return asYmd(addUnits('day', today, 1)); + } + + const period = resolvePeriodToken(token, today); + if (period !== undefined) return period; + + const param = parseDateMacroParam(token); + if (param) { + const sign = param.direction === 'ago' ? -1 : 1; + // Sub-day units are instants — they must keep their time-of-day, so they + // shift `now` and render as a full ISO timestamp. Day-and-coarser units are + // calendar quantities and render as `YYYY-MM-DD` off the reference day. + if (param.unit === 'minute' || param.unit === 'hour') { + return addUnits(param.unit, now, sign * param.n).toISOString(); + } + return asYmd(addUnits(param.unit, today, sign * param.n)); + } + + return undefined; +} + +/** + * Does this tree contain any fully-wrapped placeholder at all? + * + * A read-only pre-pass so the overwhelmingly common case — an internal query + * whose filter is entirely literal — costs one allocation-free walk instead of + * a full structural copy. This runs on every server-side read, so "no + * placeholders" must be close to free. + */ +function hasFilterToken(node: unknown): boolean { + if (typeof node === 'string') return classifyFilterToken(node) !== null; + if (Array.isArray(node)) return node.some(hasFilterToken); + if (node && typeof node === 'object' && !(node instanceof Date)) { + return Object.values(node as Record).some(hasFilterToken); + } + return false; +} + +/** + * Deep-replace every fully-wrapped placeholder in `filter` with its resolved + * value, returning a NEW tree (the caller's metadata is never mutated — a view + * or dataset definition is shared across requests, so resolving in place would + * bake one request's user id, and one day's dates, into every later render). + * + * Returns the input unchanged, by reference, when it holds no placeholders. + */ +export function resolveFilterTokens( + filter: T, + ctx: FilterTokenResolutionContext = {}, +): T { + if (filter == null) return filter; + if (!hasFilterToken(filter)) return filter; + + // One instant for the whole tree (see FilterTokenResolutionContext.now). + const pinned: FilterTokenResolutionContext = { ...ctx, now: ctx.now ?? new Date() }; + + const walk = (node: unknown): unknown => { + if (typeof node === 'string') { + const cls = classifyFilterToken(node); + if (!cls) return node; + if (cls.kind === 'unknown') throw new UnknownFilterTokenError(cls.token, cls.suggestion); + const resolved = resolveFilterToken(cls.token, pinned); + // `classifyFilterToken` already vouched for the token, so `undefined` + // here would mean the spec vocabulary and this resolver have drifted + // apart — surface it loudly rather than silently emitting `undefined`. + if (resolved === undefined) throw new UnknownFilterTokenError(cls.token); + return resolved; + } + if (Array.isArray(node)) return node.map(walk); + if (node && typeof node === 'object') { + // Dates and other class instances are comparands, not filter structure. + if (node instanceof Date) return node; + const out: Record = {}; + for (const [k, v] of Object.entries(node as Record)) out[k] = walk(v); + return out; + } + return node; + }; + + return walk(filter) as T; +} + +/** + * Convenience bridge from an execution context to the resolver's inputs. + * `{current_org_id}` reads `tenantId` — the active organization IS the tenant + * on the read path (same value the RLS compiler binds to + * `current_user.organization_id`). + * + * Typed structurally, not as `ExecutionContext`, so both the parsed context + * (defaults applied) and the pre-parse `ExecutionContextInput` a caller holds + * mid-pipeline satisfy it. The three fields read here are optional in both. + */ +export function filterTokenContextFrom( + execCtx: ExecutionContextLike | undefined, + now?: Date, +): FilterTokenResolutionContext { + return { + now, + timezone: execCtx?.timezone, + userId: execCtx?.userId, + orgId: execCtx?.tenantId, + }; +} diff --git a/packages/objectql/src/engine-filter-tokens.test.ts b/packages/objectql/src/engine-filter-tokens.test.ts new file mode 100644 index 0000000000..7816997458 --- /dev/null +++ b/packages/objectql/src/engine-filter-tokens.test.ts @@ -0,0 +1,189 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import { describe, it, expect, vi, beforeEach } from 'vitest'; +import { ObjectQL } from './engine'; +import { SchemaRegistry } from './registry'; + +/** + * framework#3582 — filter placeholders are expanded on the SERVER read path. + * + * Filters travel as JSON, so a time- or user-scoped slice authored in a view, + * a related list or a REST query writes `'{current_year_start}'` / + * `'{current_user_id}'`. Nothing on the server used to substitute them: the + * placeholder reached the driver as a literal string, compared as text, and + * matched nothing — an empty grid with no error anywhere, which apps worked + * around by freezing dates into their build artifact. + * + * These tests assert on what the DRIVER receives: no `{token}` may survive to + * the driver AST, and an unresolvable one must throw rather than pass through. + */ +vi.mock('./registry', () => { + const instance: any = { + getObject: vi.fn(), + resolveObject: vi.fn((n: string) => instance.getObject(n)), + registerObject: vi.fn(), + getObjectOwner: vi.fn(), + registerNamespace: vi.fn(), + registerKind: vi.fn(), + registerItem: vi.fn(), + registerApp: vi.fn(), + installPackage: vi.fn(), + reset: vi.fn(), + metadata: { get: vi.fn(() => new Map()) }, + }; + function SchemaRegistry() { + return instance; + } + Object.assign(SchemaRegistry, instance); + return { + SchemaRegistry, + computeFQN: (_ns: string | undefined, name: string) => name, + parseFQN: (fqn: string) => ({ namespace: undefined, shortName: fqn }), + RESERVED_NAMESPACES: new Set(['base', 'system']), + }; +}); + +const DEAL_SCHEMA = { + name: 'deal', + fields: { + title: { type: 'text' }, + owner: { type: 'text' }, + close_date: { type: 'date' }, + }, +}; + +function makeDriver() { + const seen: { findAst?: any; findOneAst?: any; countAst?: any; aggregateAst?: any } = {}; + const driver: any = { + name: 'memory', + supports: {}, + connect: vi.fn().mockResolvedValue(undefined), + disconnect: vi.fn().mockResolvedValue(undefined), + find: vi.fn(async (_o: string, ast: any) => { seen.findAst = ast; return []; }), + findOne: vi.fn(async (_o: string, ast: any) => { seen.findOneAst = ast; return null; }), + count: vi.fn(async (_o: string, ast: any) => { seen.countAst = ast; return 0; }), + aggregate: vi.fn(async (_o: string, ast: any) => { seen.aggregateAst = ast; return []; }), + create: vi.fn(), + update: vi.fn(), + delete: vi.fn(), + }; + return { driver, seen }; +} + +async function makeEngine(driver: any) { + vi.mocked((SchemaRegistry as any).getObject).mockImplementation((name: string) => + name === 'deal' ? DEAL_SCHEMA : undefined, + ); + const ql = new ObjectQL(); + ql.registerDriver(driver, true); + await ql.init(); + return ql; +} + +const CTX = { userId: 'usr_1', tenantId: 'org_9', timezone: 'UTC' } as any; +const THIS_YEAR_START = `${new Date().getUTCFullYear()}-01-01`; + +describe('engine filter placeholders (framework#3582)', () => { + beforeEach(() => { vi.clearAllMocks(); }); + + it('find(): a date macro reaches the driver as a concrete date', async () => { + const { driver, seen } = makeDriver(); + const ql = await makeEngine(driver); + + await ql.find('deal', { + where: { close_date: { $gte: '{current_year_start}' } }, + context: CTX, + }); + + expect(seen.findAst?.where).toEqual({ close_date: { $gte: THIS_YEAR_START } }); + }); + + it('find(): a context token reaches the driver as the signed-in user id', async () => { + const { driver, seen } = makeDriver(); + const ql = await makeEngine(driver); + + await ql.find('deal', { where: { owner: '{current_user_id}' }, context: CTX }); + + expect(seen.findAst?.where).toEqual({ owner: 'usr_1' }); + }); + + it('find(): expands inside logical branches and the `filter` alias', async () => { + const { driver, seen } = makeDriver(); + const ql = await makeEngine(driver); + + await ql.find('deal', { + filter: { $and: [{ owner: '{current_user_id}' }, { close_date: { $lt: '{today}' } }] }, + context: CTX, + } as any); + + const branches = seen.findAst?.where?.$and; + expect(branches?.[0]).toEqual({ owner: 'usr_1' }); + expect(branches?.[1].close_date.$lt).toMatch(/^\d{4}-\d{2}-\d{2}$/); + }); + + it('findOne(): placeholders are expanded too', async () => { + const { driver, seen } = makeDriver(); + const ql = await makeEngine(driver); + + await ql.findOne('deal', { where: { owner: '{current_user_id}' }, context: CTX }); + + expect(seen.findOneAst?.where).toEqual({ owner: 'usr_1' }); + }); + + it('count(): the driver counts the resolved filter, not the literal', async () => { + const { driver, seen } = makeDriver(); + const ql = await makeEngine(driver); + + await ql.count('deal', { where: { owner: '{current_user_id}' }, context: CTX }); + + expect(seen.countAst?.where).toEqual({ owner: 'usr_1' }); + }); + + it('aggregate(): placeholders are expanded before grouping', async () => { + const { driver, seen } = makeDriver(); + const ql = await makeEngine(driver); + + await ql.aggregate('deal', { + where: { close_date: { $gte: '{current_year_start}' } }, + groupBy: ['owner'], + aggregations: [{ func: 'count', field: 'id', alias: 'n' }], + context: CTX, + } as any); + + expect(seen.aggregateAst?.where).toEqual({ close_date: { $gte: THIS_YEAR_START } }); + expect(seen.aggregateAst?.groupBy).toEqual(['owner']); + }); + + it('throws on an unknown placeholder instead of matching nothing', async () => { + const { driver } = makeDriver(); + const ql = await makeEngine(driver); + + // `{current_user}` is the near-miss the issue was filed on: it is the RLS + // expression root, so authors reach for it in filters too. + await expect( + ql.find('deal', { where: { owner: '{current_user}' }, context: CTX }), + ).rejects.toThrow(/current_user_id/); + expect(driver.find).not.toHaveBeenCalled(); + }); + + it('leaves placeholder-free filters untouched', async () => { + const { driver, seen } = makeDriver(); + const ql = await makeEngine(driver); + + await ql.find('deal', { where: { title: 'acme {x} deal', owner: 'usr_2' }, context: CTX }); + + expect(seen.findAst?.where).toEqual({ title: 'acme {x} deal', owner: 'usr_2' }); + }); + + it('does not mutate the caller filter — view metadata is shared across requests', async () => { + const { driver } = makeDriver(); + const ql = await makeEngine(driver); + const viewFilter = { owner: '{current_user_id}' }; + + await ql.find('deal', { where: viewFilter, context: CTX }); + await ql.find('deal', { where: viewFilter, context: { ...CTX, userId: 'usr_2' } as any }); + + expect(viewFilter).toEqual({ owner: '{current_user_id}' }); + expect(driver.find.mock.calls[1][1].where).toEqual({ owner: 'usr_2' }); + }); +}); diff --git a/packages/objectql/src/engine.ts b/packages/objectql/src/engine.ts index c3ed1c629a..403da15d28 100644 --- a/packages/objectql/src/engine.ts +++ b/packages/objectql/src/engine.ts @@ -19,7 +19,16 @@ import { isDataMigrationFlagVerified, } from '@objectstack/spec/system'; import { ExecutionContext, ExecutionContextInput, ExecutionContextSchema } from '@objectstack/spec/kernel'; -import { IDataDriver, IDataEngine, Logger, createLogger, withTransientRetry, type RetryOptions } from '@objectstack/core'; +import { + IDataDriver, + IDataEngine, + Logger, + createLogger, + withTransientRetry, + type RetryOptions, + filterTokenContextFrom, + resolveFilterTokens, +} from '@objectstack/core'; import { SummaryRecomputeError, type SummaryRecomputeFailure } from './summary-errors.js'; import { DriverConnectError, emitDegradedBootBanner, type DriverConnectFailure, type DriverHealth } from './driver-connect-errors.js'; import { resolveAllowDriverConnectFailure } from '@objectstack/types'; @@ -2519,6 +2528,34 @@ export class ObjectQL implements IDataEngine { // Data Access Methods (IDataEngine Interface) // ============================================ + /** + * Expand `{filter-placeholder}` values in a read AST's `where` against the + * request (framework#3582). + * + * Filters travel as JSON, so a time- or user-scoped slice authored in a view, + * dashboard, related list or REST query writes `'{current_year_start}'` / + * `'{current_user_id}'` rather than a literal. Until now nothing on the server + * substituted them: the placeholder reached the driver as a string, compared + * as text, and matched nothing — an empty grid with no error anywhere. + * + * The engine is the right seam because it is the ONE gate every server-side + * read passes through (REST, SDK, related lists, flow `find_records`, sharing + * graph reads), so a surface that follows the filter contract works the day it + * ships instead of waiting for its own resolver. It runs BEFORE the middleware + * chain so only author-supplied filters are inspected; the RLS/sharing filters + * injected downstream are built from concrete context values and carry no + * placeholders. + * + * Cheap by construction: {@link resolveFilterTokens} returns the input by + * reference when the tree holds no placeholder, which is every internal query. + * An unresolvable placeholder throws (see the resolver's module doc) — the one + * outcome an author can act on. + */ + private resolveWhereTokens(ast: QueryAST | undefined, execCtx?: ExecutionContextInput): void { + if (!ast || ast.where == null) return; + ast.where = resolveFilterTokens(ast.where, filterTokenContextFrom(execCtx)); + } + async find(object: string, query?: EngineQueryOptions, options?: EngineReadOptions): Promise { object = this.resolveObjectName(object); this.logger.debug('Find operation starting', { object, query }); @@ -2610,6 +2647,7 @@ export class ObjectQL implements IDataEngine { options: query, context: mergeReadContext(query?.context, options?.context), }; + this.resolveWhereTokens(opCtx.ast as QueryAST, opCtx.context); await this.executeWithMiddleware(opCtx, async () => { const hookContext: HookContext = { @@ -2697,6 +2735,7 @@ export class ObjectQL implements IDataEngine { options: query, context: mergeReadContext(query?.context, options?.context), }; + this.resolveWhereTokens(opCtx.ast as QueryAST, opCtx.context); await this.executeWithMiddleware(opCtx, async () => { // [#3195] `findOne` fires the SAME `beforeFind`/`afterFind` hooks as @@ -3546,6 +3585,11 @@ export class ObjectQL implements IDataEngine { options: query, context: mergeReadContext(query?.context, options?.context), }; + this.resolveWhereTokens(opCtx.ast as QueryAST, opCtx.context); + // The caller's own `where`, placeholders expanded — captured BEFORE the + // middleware chain scopes `opCtx.ast.where`, so the find() fallback below + // still passes the unscoped filter (find() applies the read filters itself). + const callerWhere = (opCtx.ast as QueryAST).where; await this.executeWithMiddleware(opCtx, async () => { const countOpts = this.buildDriverOptions(object, opCtx.context); @@ -3554,7 +3598,7 @@ export class ObjectQL implements IDataEngine { } // Fallback to find().length — find() applies the read filters itself, // so pass the caller's original where, not the already-scoped ast. - const res = await this.find(object, { where: query?.where, fields: ['id'], context: opCtx.context }); + const res = await this.find(object, { where: callerWhere, fields: ['id'], context: opCtx.context }); return res.length; }); @@ -3622,6 +3666,7 @@ export class ObjectQL implements IDataEngine { options: query, context: mergeReadContext(query?.context, options?.context), }; + this.resolveWhereTokens(opCtx.ast as QueryAST, opCtx.context); await this.executeWithMiddleware(opCtx, async () => { const ast = opCtx.ast as QueryAST; diff --git a/packages/services/service-analytics/src/__tests__/dataset-filter-tokens.test.ts b/packages/services/service-analytics/src/__tests__/dataset-filter-tokens.test.ts new file mode 100644 index 0000000000..1bcc70cc19 --- /dev/null +++ b/packages/services/service-analytics/src/__tests__/dataset-filter-tokens.test.ts @@ -0,0 +1,142 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import { describe, it, expect } from 'vitest'; +import { DatasetSchema } from '@objectstack/spec/ui'; +import type { ExecutionContext } from '@objectstack/spec/kernel'; +import { AnalyticsService } from '../analytics-service.js'; + +/** + * framework#3582 — a dashboard widget's filter placeholders are expanded on the + * SERVER, before the strategy compiles SQL. + * + * The dashboard path needs its own resolution rather than inheriting the + * ObjectQL engine's: `NativeSQLStrategy` compiles a raw `SELECT … WHERE` and + * binds comparands directly, so a widget filtered on `{current_year_start}` + * never passes through `engine.find()`. That is precisely why such widgets + * rendered zero — the placeholder was bound as the literal text. + * + * The assertions read the BOUND PARAMS, i.e. what the database actually + * compares against. + */ +const dataset = DatasetSchema.parse({ + name: 'pipeline', + label: 'Pipeline', + object: 'opportunity', + dimensions: [{ name: 'stage', field: 'stage', type: 'string' }], + measures: [{ name: 'revenue', aggregate: 'sum', field: 'amount' }], +}); + +function service(captured: { sql: string; params: unknown[] }[]) { + return new AnalyticsService({ + queryCapabilities: () => ({ nativeSql: true, objectqlAggregate: false, inMemory: false }), + executeRawSql: async (_o, sql, params) => { + captured.push({ sql, params }); + return [{ stage: 'won', revenue: 100 }]; + }, + }); +} + +const CTX = { userId: 'usr_1', tenantId: 'org_9', timezone: 'UTC' } as ExecutionContext; +const THIS_YEAR_START = `${new Date().getUTCFullYear()}-01-01`; + +describe('dataset filter placeholders (framework#3582)', () => { + it('expands a date macro in a widget runtimeFilter into a bound date', async () => { + const captured: { sql: string; params: unknown[] }[] = []; + await service(captured).queryDataset( + dataset, + { + dimensions: ['stage'], + measures: ['revenue'], + runtimeFilter: { close_date: { $gte: '{current_year_start}' } }, + }, + CTX, + ); + + expect(captured[0].params).toContain(THIS_YEAR_START); + expect(captured[0].params).not.toContain('{current_year_start}'); + }); + + it('expands a context token in a widget runtimeFilter', async () => { + const captured: { sql: string; params: unknown[] }[] = []; + await service(captured).queryDataset( + dataset, + { dimensions: ['stage'], measures: ['revenue'], runtimeFilter: { owner: '{current_user_id}' } }, + CTX, + ); + + expect(captured[0].params).toContain('usr_1'); + }); + + it("expands the dataset's own intrinsic filter", async () => { + const scoped = DatasetSchema.parse({ + ...dataset, + name: 'my_pipeline', + filter: { owner: '{current_user_id}' }, + }); + const captured: { sql: string; params: unknown[] }[] = []; + await service(captured).queryDataset( + scoped, + { dimensions: ['stage'], measures: ['revenue'] }, + CTX, + ); + + expect(captured[0].params).toContain('usr_1'); + }); + + it('expands a measure-scoped filter', async () => { + const scoped = DatasetSchema.parse({ + ...dataset, + name: 'ytd_pipeline', + measures: [ + { + name: 'revenue_ytd', + aggregate: 'sum', + field: 'amount', + filter: { close_date: { $gte: '{current_year_start}' } }, + }, + ], + }); + const captured: { sql: string; params: unknown[] }[] = []; + await service(captured).queryDataset( + scoped, + { dimensions: ['stage'], measures: ['revenue_ytd'] }, + CTX, + ); + + expect(captured.some((c) => c.params.includes(THIS_YEAR_START))).toBe(true); + }); + + it('never mutates the registered dataset — it is reused across requests', async () => { + const scoped = DatasetSchema.parse({ + ...dataset, + name: 'shared_pipeline', + filter: { owner: '{current_user_id}' }, + }); + const captured: { sql: string; params: unknown[] }[] = []; + const svc = service(captured); + + await svc.queryDataset(scoped, { dimensions: ['stage'], measures: ['revenue'] }, CTX); + await svc.queryDataset( + scoped, + { dimensions: ['stage'], measures: ['revenue'] }, + { ...CTX, userId: 'usr_2' } as ExecutionContext, + ); + + // Second render must scope to the SECOND user, not a baked-in first one. + expect(captured[1].params).toContain('usr_2'); + expect(captured[1].params).not.toContain('usr_1'); + expect(scoped.filter).toEqual({ owner: '{current_user_id}' }); + }); + + it('fails loudly on an unresolvable placeholder instead of charting zero', async () => { + const captured: { sql: string; params: unknown[] }[] = []; + await expect( + service(captured).queryDataset( + dataset, + { dimensions: ['stage'], measures: ['revenue'], runtimeFilter: { owner: '{current_user}' } }, + CTX, + ), + ).rejects.toThrow(/current_user_id/); + expect(captured).toHaveLength(0); + }); +}); diff --git a/packages/services/service-analytics/src/dataset-executor.ts b/packages/services/service-analytics/src/dataset-executor.ts index b4d11a7cd8..fe64d19fd3 100644 --- a/packages/services/service-analytics/src/dataset-executor.ts +++ b/packages/services/service-analytics/src/dataset-executor.ts @@ -9,6 +9,7 @@ import type { } from '@objectstack/spec/contracts'; import type { FilterCondition } from '@objectstack/spec/data'; import type { ExecutionContext } from '@objectstack/spec/kernel'; +import { filterTokenContextFrom, resolveFilterTokens } from '@objectstack/core'; import type { CompiledDataset, DerivedMeasureSpec } from './dataset-compiler.js'; import type { OrderLabelResolver } from './dimension-labels.js'; @@ -71,6 +72,56 @@ export type CompareTo = DatasetCompareTo; * engine dependency. */ +/** + * Expand `{filter-placeholder}` values across everything a dataset query + * compares on (framework#3582): the dataset's intrinsic `filter`, the + * presentation's `runtimeFilter` (a dashboard widget's own scope), every + * measure-scoped filter, and the `dateRange` bounds of the selection's time + * dimensions. + * + * The dashboard path needs its own call rather than inheriting the ObjectQL + * engine's: `NativeSQLStrategy` compiles a raw `SELECT … WHERE` and binds + * comparands directly, so a widget filtered on `{current_year_start}` never + * passes through `engine.find()` at all — which is exactly why the token + * reached SQLite as the literal text and every such widget rendered zero. + * + * Inputs are treated as immutable: a `CompiledDataset` lives in the service's + * registry across requests, so resolving in place would bake one request's + * user id (and one day's dates) into every later render. New objects are + * allocated only when the tree actually held a placeholder. + */ +function resolveSelectionTokens( + compiled: CompiledDataset, + selection: DatasetSelection, + context?: ExecutionContext, +): { compiled: CompiledDataset; selection: DatasetSelection } { + // One instant for the whole call: the intrinsic filter, the runtime filter + // and each measure filter are resolved in separate passes, and a query whose + // pieces disagreed about "now" could straddle a period boundary — the primary + // grid scoped to this month while a measure-scoped sub-query saw the next. + const tokenCtx = filterTokenContextFrom(context, new Date()); + const resolve = (v: T): T => resolveFilterTokens(v, tokenCtx); + + const filter = resolve(compiled.filter); + const measureFilters = resolve(compiled.measureFilters); + const runtimeFilter = resolve(selection.runtimeFilter); + const timeDimensions = selection.timeDimensions?.map((td) => + td.dateRange == null ? td : { ...td, dateRange: resolve(td.dateRange) }, + ); + + const compiledChanged = + filter !== compiled.filter || measureFilters !== compiled.measureFilters; + const selectionChanged = + runtimeFilter !== selection.runtimeFilter || + (timeDimensions !== undefined && + timeDimensions.some((td, i) => td !== selection.timeDimensions![i])); + + return { + compiled: compiledChanged ? { ...compiled, filter, measureFilters } : compiled, + selection: selectionChanged ? { ...selection, runtimeFilter, timeDimensions } : selection, + }; +} + /** AND two optional FilterConditions into one (MongoDB-style). */ export function combineFilters( a?: FilterCondition, @@ -350,10 +401,15 @@ export class DatasetExecutor { * applied per request (ADR-0021 D-C). */ async execute( - compiled: CompiledDataset, - selection: DatasetSelection, + compiledInput: CompiledDataset, + selectionInput: DatasetSelection, context?: ExecutionContext, ): Promise { + // framework#3582 — expand `{current_quarter_start}` / `{current_user_id}` + // placeholders BEFORE any query is shaped, once for the whole call so every + // sub-query (measure-scoped, totals, compareTo) shares one instant. + const { compiled, selection } = resolveSelectionTokens(compiledInput, selectionInput, context); + const result = await this.executeSelection(compiled, selection, context); // Server-side totals (#1753) — re-run the selection grouped by each diff --git a/packages/services/service-automation/src/builtin/notify-node.test.ts b/packages/services/service-automation/src/builtin/notify-node.test.ts index fe7a584c35..fa7fdd3464 100644 --- a/packages/services/service-automation/src/builtin/notify-node.test.ts +++ b/packages/services/service-automation/src/builtin/notify-node.test.ts @@ -206,6 +206,56 @@ describe('notify (baseline node)', () => { expect(result.success).toBe(false); expect(result.error).toContain('recipient'); }); + + // ── framework#3582 — unresolved recipient templates ────────────────── + // `{record.owner.manager}` walks `.manager` on a scalar foreign-key id + // and resolves to nothing. `String(undefined)` used to make that the + // six-character audience member "undefined": the emit succeeded, the + // notification addressed nobody, and nothing anywhere said so. + it('never sends the literal "undefined" as an audience member', async () => { + engine.registerFlow('notify_flow', notifyFlow({ + title: 'Escalated', + recipients: ['user_1', '{record.owner.manager}'], + })); + + const result = await engine.execute('notify_flow', { + record: { id: 'case_1', owner: 'usr_7' }, + } as any); + + expect(result.success).toBe(true); + expect(messaging.emitted[0].audience).toEqual(['user_1']); + }); + + it('fails the step, naming the template, when every recipient resolves to nothing', async () => { + engine.registerFlow('notify_flow', notifyFlow({ + title: 'Escalated', + recipients: ['{record.owner.manager}'], + })); + + const result = await engine.execute('notify_flow', { + record: { id: 'case_1', owner: 'usr_7' }, + } as any); + + expect(result.success).toBe(false); + expect(result.error).toContain('{record.owner.manager}'); + expect(result.error).toContain('expand'); + expect(messaging.emitted).toHaveLength(0); + }); + + it('resolves a recipient hop when the start node expanded the relation (#3475)', async () => { + engine.registerFlow('notify_flow', notifyFlow({ + title: 'Escalated', + recipients: ['{record.owner.manager}'], + })); + + // An expanded relation arrives on the record as a nested object. + const result = await engine.execute('notify_flow', { + record: { id: 'case_1', owner: { id: 'usr_7', manager: 'usr_9' } }, + } as any); + + expect(result.success).toBe(true); + expect(messaging.emitted[0].audience).toEqual(['usr_9']); + }); }); describe('without a messaging service', () => { diff --git a/packages/services/service-automation/src/builtin/notify-node.ts b/packages/services/service-automation/src/builtin/notify-node.ts index cb341194b4..924e133ef3 100644 --- a/packages/services/service-automation/src/builtin/notify-node.ts +++ b/packages/services/service-automation/src/builtin/notify-node.ts @@ -27,13 +27,47 @@ export interface MessagingServiceSurface { }): Promise<{ notificationId: string; delivered: number; failed: number }>; } -/** Coerce a config value (string | string[]) into a clean string[]. */ +/** + * Coerce a config value (string | string[]) into a clean string[]. + * + * Entries that resolved to nothing are DROPPED, not stringified (framework#3582). + * `String(undefined)` is the six-character string `"undefined"`, which survives + * `.filter(Boolean)` and was handed to the messaging service as an audience + * member — so `to: ['{record.owner.manager}']` addressed a user id literally + * named "undefined": no delivery, no error, and a `sys_notification` row + * pointing at nobody. Dropping the entry instead lets the empty-recipients + * guard below report the real problem. + */ function toStringList(value: unknown): string[] { - if (Array.isArray(value)) return value.map((v) => String(v)).filter(Boolean); + if (Array.isArray(value)) { + return value + .filter((v) => v != null) + .map((v) => String(v).trim()) + .filter(Boolean); + } if (typeof value === 'string' && value.trim()) return [value.trim()]; return []; } +/** + * The `{…}` templates an author wrote in the recipient config, for the + * diagnostic when none of them resolved. Reading them off the RAW config (not + * the interpolated result) is the whole point: after interpolation an + * unresolved token is indistinguishable from an author who configured nothing. + */ +function recipientTemplates(raw: unknown): string[] { + const out: string[] = []; + const walk = (v: unknown): void => { + if (typeof v === 'string') { + for (const m of v.matchAll(/\{[^{}]+\}/g)) out.push(m[0]); + return; + } + if (Array.isArray(v)) v.forEach(walk); + }; + walk(raw); + return [...new Set(out)]; +} + /** Coerce an interpolated config value to a non-empty trimmed string, else undefined. */ function toStr(value: unknown): string | undefined { if (value == null) return undefined; @@ -140,7 +174,8 @@ export function registerNotifyNode(engine: AutomationEngine, ctx: PluginContext) async execute(node, variables, context) { const cfg = (node.config ?? {}) as Record; - const recipients = toStringList(interpolate(cfg.recipients ?? cfg.to ?? [], variables, context)); + const recipientCfg = cfg.recipients ?? cfg.to ?? []; + const recipients = toStringList(interpolate(recipientCfg, variables, context)); // stringifyForTemplate (not String()): a sole-token `{$error}` resolves // to the engine's error OBJECT, which String() would render as the // useless `[object Object]` (#3450). Serialize it readably instead. @@ -166,7 +201,22 @@ export function registerNotifyNode(engine: AutomationEngine, ctx: PluginContext) if (!title) return { success: false, error: 'notify: title (or subject) is required' }; if (recipients.length === 0) { - return { success: false, error: 'notify: at least one recipient is required' }; + // Name the templates that came up empty (framework#3582). The + // dominant cause is a cross-object hop — `{record.owner.manager}` + // walks `.manager` on a scalar foreign-key id — which used to + // deliver to a phantom "undefined" audience instead of failing. + const templates = recipientTemplates(recipientCfg); + return { + success: false, + error: templates.length > 0 + ? `notify: at least one recipient is required, but every recipient template ` + + `resolved to nothing: ${templates.join(', ')}. A flow template reads the ` + + `trigger record as it was written — a relation field holds a scalar id, not ` + + `an expanded record, so \`{record..}\` resolves to nothing. ` + + `Add the relation to the start node's \`config.expand\` so the engine ` + + `hydrates it, or address the id directly (\`{record.}\`).` + : 'notify: at least one recipient is required', + }; } const messaging = getMessaging(); diff --git a/packages/spec/src/data/context-tokens.zod.ts b/packages/spec/src/data/context-tokens.zod.ts index 1873d03787..e7bf24db64 100644 --- a/packages/spec/src/data/context-tokens.zod.ts +++ b/packages/spec/src/data/context-tokens.zod.ts @@ -16,10 +16,19 @@ import { DATE_MACRO_WRAPPED_RE, isDateMacroToken } from './date-macros.zod.js'; * { owner_id: '{current_user_id}' } * [{ field: 'owner', operator: 'equals', value: '{current_user_id}' }] * - * Like date macros, the placeholders are expanded **client-side** by - * `resolveContextTokens()` in `@object-ui/core` immediately before the - * filter is handed to the data source. The data engine only ever sees - * concrete ids, never `{tokens}`. + * Like date macros, the placeholders are expanded on **both** sides of + * the wire (framework#3582): `resolveContextTokens()` in + * `@object-ui/core` before the filter leaves the browser, and + * `resolveFilterTokens()` in `@objectstack/core` on the ObjectQL read + * path and the analytics dataset executor for filters that reach the + * database without passing through a renderer. The DRIVER only ever + * sees concrete ids, never `{tokens}`. + * + * The server resolver reads `ExecutionContext` — `{current_user_id}` is + * `userId`, `{current_org_id}` is `tenantId`. A request that carries + * neither is an ERROR, not a null comparand: resolving to `null` + * degrades to `IS NULL` on most drivers and would hand back the rows + * the filter was written to exclude. * * # Presentation scope, NOT a security boundary * diff --git a/packages/spec/src/data/date-macros.zod.ts b/packages/spec/src/data/date-macros.zod.ts index 2a038eef56..5c071cff24 100644 --- a/packages/spec/src/data/date-macros.zod.ts +++ b/packages/spec/src/data/date-macros.zod.ts @@ -15,10 +15,27 @@ import { z } from 'zod'; * { published_at: { $gte: '{last_quarter_start}' } } * { signal_at: { $gte: '{30_days_ago}' } } * - * The placeholders are expanded **client-side** by - * `resolveDateMacros()` in `@object-ui/core` immediately before the - * filter is handed to the data source. The data engine itself only - * sees ISO date / timestamp strings, never `{tokens}`. + * The placeholders are expanded on **both** sides of the wire, so a + * filter behaves the same wherever it is executed (framework#3582): + * + * - **Client** — `resolveDateMacros()` in `@object-ui/core`, just + * before the filter is handed to the data source. + * - **Server** — `resolveFilterTokens()` in `@objectstack/core`, wired + * into the ObjectQL read path (`find`/`findOne`/`count`/`aggregate`) + * and the analytics dataset executor. Filters that reach the database + * WITHOUT passing through a renderer — dashboard widgets, dataset + * definitions, REST query params — need this: before it, the token + * compared as a literal string and matched nothing. + * + * Either way the DRIVER only ever sees ISO date / timestamp strings, + * never `{tokens}`. Translating an ISO comparand into a column's on-disk + * form (SQLite epoch-ms, `YYYY-MM-DD` text, native timestamp) is the + * driver's job — see `SqlDriver.temporalFilterValue`. + * + * A token OUTSIDE this vocabulary is rejected rather than passed + * through: `@objectstack/lint`'s `validate-filter-tokens` fails the + * build, and the runtime resolver throws. Silently matching nothing is + * the failure mode the vocabulary exists to prevent. * * AI agents and template authors author these placeholders directly, * so the **set of recognised tokens is part of the platform contract** @@ -43,7 +60,11 @@ import { z } from 'zod'; * `@objectstack/formula`. * - Token resolution semantics (week-start day, timezone, fiscal * calendars) are defined by the resolver implementation; spec only - * freezes the **vocabulary**. + * freezes the **vocabulary**. One property is worth stating here + * because it is authored against: a `*_end` token is the period's + * last calendar DAY (`{current_year_end}` → `2026-12-31`), so on a + * `datetime` column `<= {current_year_end}` stops at midnight on the + * 31st. Filter a timestamp with the half-open `< {next_year_start}`. */ /** Single-point tokens — moments in time. */ diff --git a/scripts/analytics-reconcile/macros.ts b/scripts/analytics-reconcile/macros.ts index 3784892304..45aa4bc0f1 100644 --- a/scripts/analytics-reconcile/macros.ts +++ b/scripts/analytics-reconcile/macros.ts @@ -2,14 +2,18 @@ // // ADR-0021 Phase 2 — date-macro resolver for the reconciliation harness. // -// In production the renderer (@object-ui/core `resolveDateMacros()`) resolves -// `{today}` / `{current_quarter_start}` / `{30_days_ago}` to concrete dates -// BEFORE issuing the query — identically for the legacy and dataset forms. This -// repo has no runtime resolver, so the harness must resolve them itself, or the -// two paths diverge on the unparseable placeholder (the dataset filter-normalizer -// drops an unparseable date filter; engine.aggregate keeps it). The resolved -// VALUE is arbitrary — what matters is that BOTH paths receive the identical -// concrete filter, so equality holds iff the two query paths agree semantically. +// In production `{today}` / `{current_quarter_start}` / `{30_days_ago}` resolve +// to concrete dates BEFORE the query runs — in the renderer (@object-ui/core +// `resolveDateMacros()`) or, since framework#3582, server-side in +// `resolveFilterTokens()` (@objectstack/core). The harness pre-resolves them +// itself for a different reason: it must hand BOTH compared paths the byte-identical +// concrete filter, so any difference in the result is a semantic disagreement +// between the paths rather than a difference in when each resolved the clock. +// +// This is also why it does NOT call `resolveFilterTokens()`: that emits ISO +// strings (the driver then coerces per column), while the harness compares raw +// stored values and wants epoch-millis directly. The resolved VALUE is arbitrary +// here — only its identity across the two paths matters. // // Token grammar mirrors packages/spec/src/data/date-macros.zod.ts. diff --git a/skills/objectstack-query/rules/filters.md b/skills/objectstack-query/rules/filters.md index b74a791928..212f1bde40 100644 --- a/skills/objectstack-query/rules/filters.md +++ b/skills/objectstack-query/rules/filters.md @@ -241,8 +241,33 @@ where: { units `minute|hour|day|week|month|year` — e.g. `{30_days_ago}`, `{2_weeks_from_now}`. -**Scope:** the tokens are expanded **client-side** (by `resolveDateMacros()` -in `@object-ui/core`) immediately before the filter reaches the data -engine — the engine only ever sees ISO date/timestamp strings. Use macros in -UI-rendered filter metadata; for queries you issue directly against the -engine, keep computing dates in application code as above. +**Session tokens:** the same value positions accept `{current_user_id}` and +`{current_org_id}` (defined in `data/context-tokens.zod.ts`) — the signed-in +user's id and the active organization id. + +```typescript +where: { owner: '{current_user_id}', close_date: { $gte: '{current_year_start}' } } +``` + +**Scope:** tokens are expanded on **both** sides of the wire, so the same +filter behaves the same wherever it runs — client-side by `resolveDateMacros()` +/ `resolveContextTokens()` in `@object-ui/core`, and server-side by +`resolveFilterTokens()` in `@objectstack/core` (wired into the ObjectQL read +path and the analytics dataset executor). The driver only ever sees ISO +date/timestamp strings and concrete ids, never `{tokens}`. You may therefore +use tokens in a query issued directly against the engine — and you should: +computing "today" at module load freezes the date into the built artifact. + +One exception: a **flow node's** `config.filter` is interpolated by the flow +template engine first, which owns `{…}` in that position and blanks anything +that is not a flow variable. Compute the bound in an earlier node instead. + +**Unknown tokens are rejected, not ignored.** `{current_user}` (the RLS +expression root) and `{this_quarter_start}` are near-misses, not tokens: +`objectstack build` fails on them and the runtime resolver throws. A filter +value that is entirely `{...}` is always read as a placeholder, so a literal +value of that shape is not expressible. + +**`*_end` is a calendar DAY.** `{current_year_end}` is `2026-12-31`, so on a +`datetime` column `<= {current_year_end}` stops at midnight on the 31st. Use +the half-open `< {next_year_start}` for timestamps. diff --git a/skills/objectstack-ui/SKILL.md b/skills/objectstack-ui/SKILL.md index 61e69ee841..bab2cf6441 100644 --- a/skills/objectstack-ui/SKILL.md +++ b/skills/objectstack-ui/SKILL.md @@ -1479,11 +1479,19 @@ Dashboard and report drill are unified. ## Date Macros — Filter Placeholders Dashboards, reports, list-view filters, and other UI metadata can embed -relative-date placeholders that are resolved on the client just before -the request leaves the browser. The canonical contract is published as +relative-date placeholders. The canonical contract is published as `DATE_MACRO_TOKENS` in `@objectstack/spec/data` (source: -`node_modules/@objectstack/spec/src/data/date-macros.zod.ts`); the resolver -lives in `@object-ui/core` (`resolveDateMacros`). Keep the two in lockstep. +`node_modules/@objectstack/spec/src/data/date-macros.zod.ts`); two resolvers +consume it and must stay in lockstep with it — `resolveDateMacros` in +`@object-ui/core` (before the request leaves the browser) and +`resolveFilterTokens` in `@objectstack/core` (framework#3582: the ObjectQL +read path and the analytics dataset executor, which is what a dashboard +widget's `filter` actually travels through — it never passes a renderer). + +An unrecognised placeholder is a build error (`validate-filter-tokens`) and a +runtime throw, never a silent literal. Note `*_end` is the period's last +calendar DAY, so a `datetime` column wants `< {next_*_start}` rather than +`<= {current_*_end}`. Both `{token}` and `${token}` forms are accepted.