fix(formula,spec,core): the RLS write-side check evaluator honours calendar-day upper bounds (ADR-0053 D-D)

.changeset/calendar-day-primitive-to-spec.md

@@ -0,0 +1,32 @@
+---
+"@objectstack/spec": minor
+"@objectstack/formula": patch
+"@objectstack/core": patch
+---
+
+fix(formula,spec,core): the RLS write-side `check` evaluator honours calendar-day upper bounds (ADR-0053 D-D)
+
+`@objectstack/formula`'s `matchesFilterCondition` — the evaluator behind RLS
+write-side `check` policies (ADR-0058 D4) — compared a bare `YYYY-MM-DD` `$lte`
+bound literally. On a `datetime` post-image that meant a policy of the shape
+`{ signed_on: { $lte: '{today}' } }` **denied every write made after 00:00**:
+the write-side twin of the read-side data loss #3777 fixed, and the last of the
+platform's filter backends that disagreed about what a bare day means as a
+bound.
+
+`$lte` and a `$between` max now evaluate half-open against the next calendar
+day, matching the SQL compiler, the memory and mongo drivers, and the analytics
+preview evaluator. Unchanged, per the same semantics table: full-ISO bounds keep
+exact-instant semantics, `$gte`/`$gt`/`$lt` keep their midnight anchoring, and a
+plain `YYYY-MM-DD` value compares identically (string ordering makes the two
+forms equivalent). The evaluator stays fail-closed on a null bound.
+
+**Where the rule now lives.** `nextUtcCalendarDay` moved from
+`@objectstack/core` to `@objectstack/spec/data` — beside `date-macros.zod.ts`,
+whose vocabulary it interprets. `formula` cannot depend on `core`, and a second
+copy of the rule is exactly the divergence #3777 catalogued; `spec` is the one
+package all six consumers already depend on, so this adds no dependency edge.
+
+No import changes are required: `@objectstack/core` re-exports the symbol, so
+existing `import { nextUtcCalendarDay } from '@objectstack/core'` keeps working.
+New code should prefer `@objectstack/spec/data`.

content/docs/data-modeling/queries.mdx

@@ -83,7 +83,7 @@ disagree about shape:
 | Field type | Canonical comparand | Semantics |
 |:---|:---|:---|
 | `date` | `YYYY-MM-DD` | Timezone-naive calendar day. A `Date` collapses to its UTC day; a longer ISO string is truncated to its leading date. |
-| `datetime` | `YYYY-MM-DDTHH:MM:SS.sssZ` | A UTC instant. A `Date`, an epoch-millisecond number, a bare `YYYY-MM-DD` (→ midnight UTC) and a zone-naive `YYYY-MM-DD HH:MM:SS` (→ read as UTC) all fold to this one form. |
+| `datetime` | `YYYY-MM-DDTHH:MM:SS.sssZ` | A UTC instant. A `Date`, an epoch-millisecond number, a bare `YYYY-MM-DD` (→ midnight UTC, but see the whole-day rule below) and a zone-naive `YYYY-MM-DD HH:MM:SS` (→ read as UTC) all fold to this one form. |
 | `time` | `HH:MM:SS`, with `.fff` **only** when the milliseconds are non-zero | Timezone-naive wall clock. `'14:30'` and `'14:30:00'` canonicalise identically, so they are the same filter. |
 
 ```typescript
@@ -97,6 +97,32 @@ disagree about shape:
 }
 ```
 
+### A bare calendar day as an UPPER bound means the whole day
+
+Canonicalisation settles a comparand's *shape*; which **instant** a bare
+`YYYY-MM-DD` denotes additionally depends on the side of the comparison it sits on:
+
+| Operator | A bare `YYYY-MM-DD` means |
+|:---|:---|
+| `$gte` / `$gt` / `$lt` | that day's `00:00:00.000` |
+| `$lte`, and the max of a `$between` | the **whole** day — compiled half-open, as `< next day` |
+
+So `{ created_at: { $gte: '2026-01-01', $lte: '2026-03-31' } }` includes
+everything recorded *on* March 31st, not just the instant it began. Without
+that rule a dashboard window ending "today" silently dropped every row created
+after midnight (#3777). Pass a full ISO timestamp when you want an exact
+instant — only the day-granular *string* carries whole-day intent:
+
+```typescript
+{ created_at: { $lte: '2026-03-31' } }              // through all of March 31st
+{ created_at: { $lte: '2026-03-31T12:00:00.000Z' } } // up to noon exactly
+```
+
+On a `date` field the two forms are equivalent (`< next day` and `<= day` order
+identically over `YYYY-MM-DD` text), so the rule needs no special handling
+there. It applies uniformly across the SQL drivers, the in-memory and MongoDB
+drivers, analytics windows, and the RLS write-side `check` evaluator.
+
 <Callout type="info">
 Canonicalisation runs on **every SQL dialect**, not just SQLite — a zone-naive string bound
 into a Postgres `timestamptz` would otherwise be read in the *server's* timezone. MySQL

docs/adr/0053-date-and-datetime-semantics.md

@@ -782,8 +782,36 @@ cover lives in `mongodb-datetime-storage.test.ts` (against a real MongoDB via
 `mongodb-memory-server`) and `memory-datetime-storage.test.ts`. Still open
 under D-A3: the relative-token × live-driver × timezone combinations.
 
-One evaluator remains unaligned on both the D-D and D-E rules:
+~~One evaluator remains unaligned on both the D-D and D-E rules:
 `@objectstack/formula`'s `matchesFilterCondition` (the RLS write-side `check`
 path), which cannot depend on `@objectstack/core` for the shared primitive.
 Deciding its home — a private copy, as `formula` already keeps for `today()`,
-or lowering the utility into `spec`/`types` — is the remaining piece.
+or lowering the utility into `spec`/`types` — is the remaining piece.~~
+Closed 2026-07-30 — see D-D2 below.
+
+### D-D2 — The primitive lives in `spec`, beside the vocabulary it interprets
+
+`nextUtcCalendarDay` moved from `@objectstack/core` to
+`@objectstack/spec/data` (`calendar-day.ts`), next to `date-macros.zod.ts`.
+The two are halves of one contract: the macros define which bare days an author
+can *name*, D-D defines what such a day *denotes* as a bound. That is protocol,
+not business logic — the same species as the pure helpers `spec/data` already
+owns (`parseDateMacroParam`, `canonicalAstOperator`, `parseAutonumberFormat`),
+so Prime Directive #2 is satisfied.
+
+Chosen over `@objectstack/types` because **it adds no dependency edge**: every
+consumer already depends on `spec`, whereas `core` does not depend on `types`
+and would have needed a new one. `core` re-exports the symbol, so the published
+`@objectstack/core` surface — which the drivers and analytics strategies import
+from — is unchanged.
+
+Rejected: a private copy in `formula` (the precedent it would cite, `formula`'s
+own `today()`, is debt rather than a pattern). Six backends now share one
+definition: the SQL compiler, the memory and mongo drivers, the analytics
+raw-SQL strategy, the dataset preview evaluator, and `matchesFilterCondition`.
+
+The write-side gap this closed was not cosmetic. A `check` policy of the shape
+`{ signed_on: { $lte: '{today}' } }` evaluated against a `datetime` post-image
+**denied every write made after 00:00** — the write-side twin of the read-side
+loss #3777 fixed, on the one surface where the failure direction is a rejected
+write rather than a missing row.

packages/core/src/utils/datetime.test.ts

@@ -61,28 +61,14 @@ describe('zonedDateStartToUtcMs — round-trips to the day boundary in the zone'
   }
 });
 
-describe('nextUtcCalendarDay — the exclusive upper bound of a bare calendar day (#3777)', () => {
-  it('advances one day, rolling month, year and leap boundaries', () => {
+describe('nextUtcCalendarDay — re-exported from @objectstack/spec (ADR-0053 D-D)', () => {
+  it('is still reachable from this package, with the same semantics', () => {
+    // The rule itself now lives in spec (six backends share it, and
+    // `@objectstack/formula` cannot depend on core) — its thorough coverage is
+    // `packages/spec/src/data/calendar-day.test.ts`. What this asserts is the
+    // re-export the drivers and analytics strategies import from HERE, so
+    // dropping it would break them loudly rather than at their call sites.
     expect(nextUtcCalendarDay('2026-07-28')).toBe('2026-07-29');
-    expect(nextUtcCalendarDay('2026-07-31')).toBe('2026-08-01');
-    expect(nextUtcCalendarDay('2026-12-31')).toBe('2027-01-01');
-    expect(nextUtcCalendarDay('2024-02-28')).toBe('2024-02-29'); // leap year
-    expect(nextUtcCalendarDay('2025-02-28')).toBe('2025-03-01');
-    expect(nextUtcCalendarDay(' 2026-07-28 ')).toBe('2026-07-29'); // trimmed
-  });
-
-  it('returns null for anything that is not a valid bare calendar day', () => {
-    // Instants keep instant semantics — never widened.
     expect(nextUtcCalendarDay('2026-07-28T12:00:00Z')).toBeNull();
-    expect(nextUtcCalendarDay(new Date('2026-07-28T00:00:00Z'))).toBeNull();
-    // Impossible days are rejected, not rolled into an invented bound.
-    expect(nextUtcCalendarDay('2026-02-30')).toBeNull();
-    expect(nextUtcCalendarDay('2026-13-01')).toBeNull();
-    // Non-strings / junk.
-    expect(nextUtcCalendarDay(1753660800000)).toBeNull();
-    expect(nextUtcCalendarDay(null)).toBeNull();
-    expect(nextUtcCalendarDay(undefined)).toBeNull();
-    expect(nextUtcCalendarDay('7/28/2026')).toBeNull();
-    expect(nextUtcCalendarDay('2026-7-28')).toBeNull(); // not zero-padded → not the canonical shape
   });
 });

packages/core/src/utils/datetime.ts

@@ -103,37 +103,16 @@ export function zonedDateStartToUtcMs(ymd: string, tz?: string): number {
 }
 
 /**
- * The calendar day after a bare `YYYY-MM-DD` string — the exclusive upper
- * bound of that day, for compiling "through day X" into the half-open
- * `[X, X+1)` a `datetime` column needs (#3777).
+ * Calendar-day bound semantics (ADR-0053 D-D) now live in `@objectstack/spec`,
+ * beside the date-macro vocabulary they give meaning to — the fifth consumer
+ * (`@objectstack/formula`'s RLS write-side `check` evaluator) cannot depend on
+ * this package, and a second copy of the rule is exactly the divergence #3777
+ * catalogued.
  *
- * A bare calendar day used as an upper bound (`$lte`, a `dateRange` end, a
- * `{current_month_end}` token) always carries whole-day intent — the author
- * means "including everything that happened on X", never "up to the stroke of
- * midnight that begins X". On a `date` column `<= X` already says that; on a
- * `datetime` column it silently drops every instant after 00:00. The correct
- * translation is the half-open pair `>= start AND < nextUtcCalendarDay(end)`
- * — the same shape the analytics drill ranges already emit — rather than an
- * inclusive `23:59:59.999` constant, which re-opens the gap at whatever
- * precision the dialect stores beyond milliseconds.
- *
- * `< nextUtcCalendarDay(X)` is also *equivalent* to `<= X` for a `date` column
- * (plain `YYYY-MM-DD` text ordering), so emitters that cannot see the column
- * type (raw-SQL strategies, the dataset preview evaluator) can apply it
- * unconditionally to a bare-day bound and be right on both column types.
- *
- * Returns `null` for anything that is not a valid bare calendar day — full
- * ISO timestamps keep instant semantics and must NOT be widened, and an
- * impossible day (`2026-02-30`) is rejected the same way
- * {@link bucketKeyToCalendarRange} rejects it, so a caller falls back to the
- * untranslated comparand instead of inventing a bound.
+ * Re-exported here so the published `@objectstack/core` surface is unchanged
+ * for the drivers and analytics strategies that already import it from here.
  */
-export function nextUtcCalendarDay(value: unknown): string | null {
-  if (typeof value !== 'string') return null;
-  const day = value.trim();
-  if (!/^\d{4}-\d{2}-\d{2}$/.test(day)) return null;
-  return bucketKeyToCalendarRange(day, 'day')?.end ?? null;
-}
+export { nextUtcCalendarDay } from '@objectstack/spec/data';
 
 /**
  * Granularity of a canonical date-bucket key. Mirrors `@objectstack/spec`'s

packages/formula/src/matches-filter.test.ts

@@ -108,3 +108,55 @@ describe('matchesFilterCondition — FAIL CLOSED', () => {
     expect(m(rec, 'nope' as never)).toBe(false);
   });
 });
+
+describe('matchesFilterCondition — calendar-day upper bounds (ADR-0053 D-D, #3777)', () => {
+  // A `check` policy of this shape on a datetime post-image used to DENY every
+  // write made after 00:00 of the bound day — the write-side twin of the
+  // read-side data loss #3777 fixed. The four other filter backends already
+  // share this rule; this evaluator was the last one that disagreed.
+  const at = (created_at: string) => ({ created_at });
+
+  it('a bare-day $lte admits the whole day', () => {
+    expect(m(at('2026-07-28T00:00:00.000Z'), { created_at: { $lte: '2026-07-28' } })).toBe(true);
+    expect(m(at('2026-07-28T09:15:00.000Z'), { created_at: { $lte: '2026-07-28' } })).toBe(true);
+    expect(m(at('2026-07-28T23:59:59.999Z'), { created_at: { $lte: '2026-07-28' } })).toBe(true);
+  });
+
+  it('…and stops at the next day', () => {
+    expect(m(at('2026-07-29T00:00:00.000Z'), { created_at: { $lte: '2026-07-28' } })).toBe(false);
+  });
+
+  it('a plain calendar-day value is unchanged (string ordering is equivalent)', () => {
+    expect(m({ signed_on: '2026-07-28' }, { signed_on: { $lte: '2026-07-28' } })).toBe(true);
+    expect(m({ signed_on: '2026-07-29' }, { signed_on: { $lte: '2026-07-28' } })).toBe(false);
+  });
+
+  it('a full-ISO bound keeps exact-instant semantics — never widened', () => {
+    expect(m(at('2026-07-28T09:15:00.000Z'), { created_at: { $lte: '2026-07-28T12:00:00.000Z' } })).toBe(true);
+    expect(m(at('2026-07-28T21:40:00.000Z'), { created_at: { $lte: '2026-07-28T12:00:00.000Z' } })).toBe(false);
+  });
+
+  it('$gte / $gt / $lt keep their midnight anchoring', () => {
+    expect(m(at('2026-07-28T09:15:00.000Z'), { created_at: { $gte: '2026-07-28' } })).toBe(true);
+    expect(m(at('2026-07-27T23:59:59.999Z'), { created_at: { $gte: '2026-07-28' } })).toBe(false);
+    expect(m(at('2026-07-28T09:15:00.000Z'), { created_at: { $lt: '2026-07-28' } })).toBe(false);
+  });
+
+  it('$between inherits the rule on its max, and still bounds the min', () => {
+    expect(m(at('2026-07-28T21:40:00.000Z'), { created_at: { $between: ['2026-04-29', '2026-07-28'] } })).toBe(true);
+    expect(m(at('2026-07-29T00:00:00.000Z'), { created_at: { $between: ['2026-04-29', '2026-07-28'] } })).toBe(false);
+    expect(m(at('2026-04-28T23:00:00.000Z'), { created_at: { $between: ['2026-04-29', '2026-07-28'] } })).toBe(false);
+  });
+
+  it('an impossible day is not rolled over — the bound stays as written', () => {
+    // `2026-02-30` is rejected by the primitive, so the comparison falls back
+    // to the literal `<=` rather than silently querying March 2nd.
+    expect(m({ signed_on: '2026-02-30' }, { signed_on: { $lte: '2026-02-30' } })).toBe(true);
+    expect(m({ signed_on: '2026-03-01' }, { signed_on: { $lte: '2026-02-30' } })).toBe(false);
+  });
+
+  it('stays fail-closed on a null bound', () => {
+    expect(m(at('2026-07-28T09:15:00.000Z'), { created_at: { $lte: null } as never })).toBe(false);
+    expect(m(at('2026-07-28T09:15:00.000Z'), { created_at: { $between: ['2026-04-29', null] } as never })).toBe(false);
+  });
+});

packages/formula/src/matches-filter.ts

@@ -17,6 +17,7 @@
  */
 
 import type { FilterCondition } from '@objectstack/spec/data';
+import { nextUtcCalendarDay } from '@objectstack/spec/data';
 
 /** True iff `record` satisfies `filter`. A null/empty filter matches everything. */
 export function matchesFilterCondition(record: Record<string, unknown>, filter: FilterCondition | null | undefined): boolean {
@@ -72,12 +73,12 @@ function evalOp(actual: unknown, op: string, raw: unknown, record: Record<string
     case '$gt': return actual != null && v != null && (actual as never) > (v as never);
     case '$gte': return actual != null && v != null && (actual as never) >= (v as never);
     case '$lt': return actual != null && v != null && (actual as never) < (v as never);
-    case '$lte': return actual != null && v != null && (actual as never) <= (v as never);
+    case '$lte': return actual != null && v != null && lteBound(actual, v);
     case '$in': return Array.isArray(v) && v.some((x) => looseEq(actual, x));
     case '$nin': return Array.isArray(v) && !v.some((x) => looseEq(actual, x));
     case '$between':
-      return Array.isArray(v) && v.length === 2 && actual != null
-        && (actual as never) >= (v[0] as never) && (actual as never) <= (v[1] as never);
+      return Array.isArray(v) && v.length === 2 && actual != null && v[0] != null && v[1] != null
+        && (actual as never) >= (v[0] as never) && lteBound(actual, v[1]);
     case '$contains': return typeof actual === 'string' && typeof v === 'string' && actual.includes(v);
     case '$notContains': return !(typeof actual === 'string' && typeof v === 'string' && actual.includes(v));
     case '$startsWith': return typeof actual === 'string' && typeof v === 'string' && actual.startsWith(v);
@@ -88,6 +89,31 @@ function evalOp(actual: unknown, op: string, raw: unknown, record: Record<string
   }
 }
 
+/**
+ * The inclusive-upper-bound comparison, with the calendar-day rule the rest of
+ * the platform applies (ADR-0053 D-D, #3777): a bare `YYYY-MM-DD` bound means
+ * "through that whole day", so it is evaluated half-open against the next day
+ * rather than against that day's midnight.
+ *
+ * Without this, a `check` policy of the shape `{ signed_on: { $lte: '{today}' } }`
+ * evaluated on a `datetime` post-image **denied every write made after 00:00** —
+ * the write-side twin of the read-side data loss #3777 fixed, and the reason
+ * this evaluator had to stop being the one backend that disagreed. The four
+ * other backends (SQL compiler, memory, mongo, the analytics preview) already
+ * share this rule via the same primitive.
+ *
+ * String ordering makes `< nextDay` equivalent to `<= day` for a plain
+ * `YYYY-MM-DD` value, so no field-type lookup is needed — which matters here,
+ * because this evaluator sees a bare record and has no schema to consult.
+ * A full-ISO or non-string bound keeps exact-instant semantics.
+ */
+function lteBound(actual: unknown, bound: unknown): boolean {
+  if (bound == null) return false;
+  const nextDay = nextUtcCalendarDay(bound);
+  if (nextDay != null) return (actual as never) < (nextDay as never);
+  return (actual as never) <= (bound as never);
+}
+
 /** Resolve a `{ $field: 'path' }` reference against the record; else passthrough. */
 function resolveValue(raw: unknown, record: Record<string, unknown>): unknown {
   if (raw && typeof raw === 'object' && !Array.isArray(raw) && '$field' in (raw as Record<string, unknown>)) {

packages/spec/api-surface.json

@@ -582,6 +582,7 @@
     "isTitleEligible (function)",
     "isUniqueDeclared (function)",
     "missingFieldValues (function)",
+    "nextUtcCalendarDay (function)",
     "objectForm (const)",
     "objectTitleCompleteness (function)",
     "parseAutonumberFormat (function)",