diff --git a/.changeset/sqlite-canonical-audit-timestamp.md b/.changeset/sqlite-canonical-audit-timestamp.md new file mode 100644 index 0000000000..17bf848fb3 --- /dev/null +++ b/.changeset/sqlite-canonical-audit-timestamp.md @@ -0,0 +1,27 @@ +--- +'@objectstack/driver-sql': patch +--- + +Fix: store SQLite `created_at`/`updated_at` in one canonical, timezone-explicit format (ADR-0074) + +The two SQLite write paths disagreed on the audit-timestamp format. INSERT fell +back to the column default `CURRENT_TIMESTAMP` (`'YYYY-MM-DD HH:MM:SS'`) while +UPDATE stamped `toISOString().replace('T',' ').replace('Z','')` +(`'YYYY-MM-DD HH:MM:SS.mmm'`) — both **timezone-naive**, space-separated strings +that `Date.parse` reads as *local* time. On a non-UTC runtime a stored UTC +wall-clock silently shifted by the host offset; e.g. the objectos kernel +freshness probe compared a shifted `updated_at` against an absolute `builtAtMs` +and never evicted (publishes/installs/config toggles didn't take effect until the +LRU TTL expired). + +`create` / `bulkCreate` / `upsert` / `update` now stamp a single canonical +ISO-8601 instant with an explicit `Z` (`new Date().toISOString()`) — matching the +caller-stamped paths (`sys_metadata`, the service outboxes) and Postgres/MySQL's +native `now()`. Because the stamp is applied app-side (not via the column +default), **existing** tenant databases are fixed immediately, not just freshly +created tables. `formatOutput` additionally repairs any legacy/raw zone-naive +audit timestamp to the same format on read (idempotent), so old rows read back +unambiguously without a data migration. `upsert` now treats `created_at` as +insert-only — a conflicting merge never overwrites it. + +Postgres/MySQL are unaffected (they store a real zone-aware `TIMESTAMP`). diff --git a/docs/adr/0074-canonical-audit-timestamp-storage-on-sqlite.md b/docs/adr/0074-canonical-audit-timestamp-storage-on-sqlite.md new file mode 100644 index 0000000000..b32ab26828 --- /dev/null +++ b/docs/adr/0074-canonical-audit-timestamp-storage-on-sqlite.md @@ -0,0 +1,94 @@ +# ADR-0074: Audit timestamps are stored in one canonical, timezone-explicit format on SQLite + +**Status**: Accepted (2026-06-26) +**Deciders**: ObjectStack Protocol Architects +**Builds on**: [ADR-0053](./0053-date-and-datetime-semantics.md) (`datetime` is an instant stored as UTC) +**Consumers**: `@objectstack/driver-sql` (`create`/`bulkCreate`/`upsert`/`update`, `formatOutput`), `@objectstack/objectql` (optimistic locking via `updated_at`, `sys_metadata` writes), report/analytics date bucketing, and any out-of-tree consumer of `created_at`/`updated_at` (notably the objectos kernel freshness probe). +**Surfaced by**: a self-hosted objectos-ee on a UTC+8 host where the per-environment kernel never evicted after a publish/install/config toggle — `updated_at` parsed as local time landed *before* the kernel's `builtAtMs`, so the freshness probe always reported "fresh". + +--- + +## TL;DR + +SQLite has no native timestamp type. The driver's two write paths disagreed on +how they wrote the builtin `created_at`/`updated_at` audit columns: + +- **INSERT** relied on the column default `defaultTo(knex.fn.now())`, which on + SQLite is `CURRENT_TIMESTAMP` → `'2026-06-26 10:23:40'` (space-separated, no + millis, **no zone**). +- **UPDATE** stamped `new Date().toISOString().replace('T',' ').replace('Z','')` + → `'2026-06-26 10:23:40.246'` (space-separated, with millis, **no zone**). + +Both are **timezone-naive**. A zone-less, space-separated string is not +ISO-8601, so `Date.parse` (V8) interprets it as **local** time. On any non-UTC +runtime a stored UTC wall-clock therefore shifts by the host offset when read +back as an instant — silently corrupting every `new Date(updated_at)` comparison, +not just the freshness probe that surfaced it. + +**Decision.** On SQLite, every driver write path stamps audit timestamps in a +single canonical format — **full ISO-8601 with an explicit `Z`** +(`new Date().toISOString()`, e.g. `'2026-06-26T10:23:40.246Z'`) — and the read +path repairs any legacy/raw zone-naive value to that same format. INSERT and +UPDATE now agree, and the stored instant is unambiguous. Postgres/MySQL are +unchanged: they store a real zone-aware `TIMESTAMP` via native `now()` and never +had the ambiguity. + +This extends ADR-0053's "`datetime` is an instant stored as UTC" to the builtin +audit columns: an instant must be stored zone-explicitly, never as a bare +wall-clock string. + +## Decision detail + +1. **Write — app-side stamp, not the column default.** `create`, `bulkCreate`, + `upsert` and `update` stamp `created_at`/`updated_at` to + `new Date().toISOString()` (SQLite only; gated on `tablesWithTimestamps`). + A caller-provided value (a seed fixture, the `sys_metadata` writer, a service + outbox) is preserved on insert — only an empty slot is filled. + + Stamping **app-side** (rather than changing the column DDL default to a + `strftime('%Y-%m-%dT%H:%M:%fZ','now')` expression) is deliberate: a DDL + default only applies to **newly created** tables, so existing tenant databases + — exactly the ones exhibiting the bug — would keep emitting the naive + `CURRENT_TIMESTAMP` on insert and continue to mix formats with the now-fixed + UPDATE. App-side stamping fixes every database immediately and needs no + schema migration. The legacy `CURRENT_TIMESTAMP` column default is left in + place as a harmless fallback for raw inserts that bypass the driver. + +2. **`created_at` is insert-only.** `upsert`'s conflict `merge()` now excludes + `created_at`, so a merge that updates an existing row advances `updated_at` + but never rewrites the original `created_at`. + +3. **Read — tolerant reader for legacy/mixed rows.** `formatOutput` repairs a + zone-naive `created_at`/`updated_at` string to canonical ISO-8601-`Z`, + interpreting the stored wall-clock as UTC (what `CURRENT_TIMESTAMP` and the + old UPDATE stamp both wrote). The repair is **idempotent** (an + already-zone-explicit value is returned unchanged) and **total** (a + `Field.datetime`-typed audit column stored as epoch-ms INTEGER, and any + unrecognised shape, pass through untouched). This mirrors the existing + read-repair the `Field.date`/numeric-scalar paths already perform, and means + existing rows read back unambiguously **without a data migration**. + +## Consequences + +- **Unambiguous instants.** `new Date(created_at|updated_at)` yields the correct + UTC instant on every host timezone. The objectos freshness probe (already + hardened defensively on the consumer side) now also receives a correct value + at the source. +- **No on-disk format mixing for new writes.** Because INSERT and UPDATE share + one format, a SQL-level `ORDER BY updated_at` (e.g. objectql metadata loads) + sorts chronologically. Lexicographic order of ISO-8601-`Z` equals chronological + order. +- **Optimistic locking stays stable.** objectql `assertVersionMatch` compares the + `updated_at` token app-side, on both sides through `formatOutput`; the + idempotent reader keeps the token deterministic across reads, including for + legacy rows. +- **Legacy rows at rest stay naive until rewritten.** Like ADR-0053's date-only + repair, the on-disk value is only normalized when a row is next written through + the driver; reads are repaired transparently. A raw SQL `ORDER BY` over rows + that straddle the fix (some naive, some `Z`) can mis-order until those rows are + rewritten — an accepted, self-healing residual, not a regression of the common + driver-mediated path. +- **Scope.** Only the builtin `created_at`/`updated_at` audit columns are + covered. A user-declared `datetime` field with a `defaultValue: 'NOW()'` still + takes the naive `CURRENT_TIMESTAMP` default on SQLite; aligning those is a + possible follow-up but was out of scope for this fix. diff --git a/packages/plugins/driver-sql/src/sql-driver-timestamp-format.test.ts b/packages/plugins/driver-sql/src/sql-driver-timestamp-format.test.ts new file mode 100644 index 0000000000..fdacc4e62c --- /dev/null +++ b/packages/plugins/driver-sql/src/sql-driver-timestamp-format.test.ts @@ -0,0 +1,154 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * Canonical audit-timestamp format on SQLite. + * + * SQLite has no native timestamp type. The two write paths used to disagree: + * INSERT fell back to the column default `CURRENT_TIMESTAMP` + * (`'YYYY-MM-DD HH:MM:SS'`) while UPDATE stamped + * `toISOString().replace('T',' ').replace('Z','')` + * (`'YYYY-MM-DD HH:MM:SS.mmm'`). BOTH were timezone-NAIVE: `Date.parse` reads a + * zone-less, space-separated string as LOCAL time, so a UTC wall-clock value + * silently shifted by the host offset on a non-UTC runtime — the bug that made + * the objectos kernel freshness probe never evict (it compared a shifted + * `updated_at` against an absolute `builtAtMs`). + * + * These tests pin the fix: every driver write path stamps a single canonical + * ISO-8601-with-`Z` instant, INSERT and UPDATE agree on that one format, and + * legacy/raw zone-naive rows are repaired to the same format on read. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { SqlDriver } from '../src/index.js'; + +const ISO_Z = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/; + +describe('SqlDriver canonical audit-timestamp format (SQLite)', () => { + let driver: SqlDriver; + let raw: any; + + beforeEach(async () => { + driver = new SqlDriver({ + client: 'better-sqlite3', + connection: { filename: ':memory:' }, + useNullAsDefault: true, + }); + raw = (driver as any).knex; + await driver.initObjects([ + { name: 'thing', fields: { name: { type: 'string' } } }, + ]); + }); + + afterEach(async () => { + await driver.disconnect(); + }); + + it('create() stamps created_at AND updated_at as canonical ISO-8601 Z (raw on disk)', async () => { + await driver.create('thing', { id: 't1', name: 'A' }, { bypassTenantAudit: true }); + const row = await raw('thing').where('id', 't1').first(); + expect(row.created_at).toMatch(ISO_Z); + expect(row.updated_at).toMatch(ISO_Z); + // both stamped from one instant on insert + expect(row.created_at).toBe(row.updated_at); + }); + + it('update() stamps updated_at canonical, INSERT and UPDATE agree on the format, created_at is preserved', async () => { + await driver.create('thing', { id: 't2', name: 'A' }, { bypassTenantAudit: true }); + const inserted = await raw('thing').where('id', 't2').first(); + await new Promise((r) => setTimeout(r, 5)); + await driver.update('thing', 't2', { name: 'B' }, { bypassTenantAudit: true }); + const updated = await raw('thing').where('id', 't2').first(); + + expect(updated.updated_at).toMatch(ISO_Z); // same canonical shape as insert + expect(inserted.updated_at).toMatch(ISO_Z); + expect(updated.created_at).toBe(inserted.created_at); // created_at immutable + // Lexicographic == chronological for ISO-8601-Z, so a SQL ORDER BY is correct. + expect(updated.updated_at >= updated.created_at).toBe(true); + }); + + it('no on-disk format mixing: an inserted-only row and an updated row share one format (SQL ORDER BY safe)', async () => { + await driver.create('thing', { id: 'never', name: 'x' }, { bypassTenantAudit: true }); + await driver.create('thing', { id: 'edited', name: 'y' }, { bypassTenantAudit: true }); + await driver.update('thing', 'edited', { name: 'y2' }, { bypassTenantAudit: true }); + const rows = await raw('thing').select('updated_at'); + for (const r of rows) expect(r.updated_at).toMatch(ISO_Z); + }); + + it('preserves a caller-provided created_at, still stamps a missing updated_at', async () => { + const provided = '2025-03-01T12:00:00.000Z'; + await driver.create('thing', { id: 't3', name: 'A', created_at: provided }, { bypassTenantAudit: true }); + const row = await raw('thing').where('id', 't3').first(); + expect(row.created_at).toBe(provided); + expect(row.updated_at).toMatch(ISO_Z); + }); + + it('bulkCreate stamps canonical timestamps', async () => { + await driver.bulkCreate('thing', [ + { id: 'b1', name: '1' }, + { id: 'b2', name: '2' }, + ], { bypassTenantAudit: true }); + const rows = await raw('thing').whereIn('id', ['b1', 'b2']).select('created_at', 'updated_at'); + for (const r of rows) { + expect(r.created_at).toMatch(ISO_Z); + expect(r.updated_at).toMatch(ISO_Z); + } + }); + + it('upsert: insert stamps canonical; a conflicting merge preserves created_at and advances updated_at', async () => { + await driver.upsert('thing', { id: 'u1', name: 'first' }, ['id'], { bypassTenantAudit: true }); + const afterInsert = await raw('thing').where('id', 'u1').first(); + expect(afterInsert.created_at).toMatch(ISO_Z); + expect(afterInsert.updated_at).toMatch(ISO_Z); + + await new Promise((r) => setTimeout(r, 5)); + await driver.upsert('thing', { id: 'u1', name: 'second' }, ['id'], { bypassTenantAudit: true }); + const afterMerge = await raw('thing').where('id', 'u1').first(); + expect(afterMerge.name).toBe('second'); + expect(afterMerge.created_at).toBe(afterInsert.created_at); // created_at immutable on merge + expect(afterMerge.updated_at).toMatch(ISO_Z); + expect(afterMerge.updated_at >= afterInsert.updated_at).toBe(true); + }); + + // ── Read-side tolerant reader (legacy / raw zone-naive rows) ──────────────── + + it('repairs a legacy space-separated row to canonical ISO-Z on read, interpreting it as UTC', async () => { + // Simulate a row written by the OLD update stamp / CURRENT_TIMESTAMP default, + // bypassing the driver write path entirely. + await raw('thing').insert({ id: 'legacy', name: 'L', created_at: '2026-01-15 08:30:00', updated_at: '2026-01-15 08:30:00.246' }); + const row: any = await driver.findOne('thing', 'legacy', { bypassTenantAudit: true }); + expect(row.created_at).toBe('2026-01-15T08:30:00.000Z'); + expect(row.updated_at).toBe('2026-01-15T08:30:00.246Z'); + }); + + it('REGRESSION (freshness probe): the repaired instant equals the UTC wall-clock, host-timezone-independent', async () => { + // The zone-naive '2026-01-15 08:30:00' must mean 08:30 UTC, NOT 08:30 local. + await raw('thing').insert({ id: 'fr', name: 'F', created_at: '2026-01-15 08:30:00', updated_at: '2026-01-15 08:30:00' }); + const row: any = await driver.findOne('thing', 'fr', { bypassTenantAudit: true }); + expect(new Date(row.updated_at as string).getTime()).toBe(Date.parse('2026-01-15T08:30:00.000Z')); + }); + + it('read-repair is idempotent: an already-canonical value is returned unchanged', async () => { + const canonical = '2026-02-02T02:02:02.222Z'; + await raw('thing').insert({ id: 'canon', name: 'C', created_at: canonical, updated_at: canonical }); + const row: any = await driver.findOne('thing', 'canon', { bypassTenantAudit: true }); + expect(row.created_at).toBe(canonical); + expect(row.updated_at).toBe(canonical); + }); + + it('does not mangle a Field.datetime-typed audit column stored as epoch ms', async () => { + // When created_at is declared `datetime`, better-sqlite3 stores a JS Date as + // INTEGER ms; the repair must leave that number alone (Field.datetime owns it). + const dd = new SqlDriver({ + client: 'better-sqlite3', connection: { filename: ':memory:' }, useNullAsDefault: true, + }); + try { + await dd.initObjects([{ name: 'evt', fields: { created_at: { type: 'datetime' }, label: { type: 'string' } } }]); + await dd.create('evt', { id: 'e1', label: 'x', created_at: new Date('2026-04-04T04:04:04.004Z') }, { bypassTenantAudit: true }); + const row: any = await dd.findOne('evt', 'e1', { bypassTenantAudit: true }); + expect(typeof row.created_at).toBe('number'); + expect(row.created_at).toBe(Date.parse('2026-04-04T04:04:04.004Z')); + } finally { + await dd.disconnect(); + } + }); +}); diff --git a/packages/plugins/driver-sql/src/sql-driver.ts b/packages/plugins/driver-sql/src/sql-driver.ts index 9bf0431a3f..f343635f15 100644 --- a/packages/plugins/driver-sql/src/sql-driver.ts +++ b/packages/plugins/driver-sql/src/sql-driver.ts @@ -82,6 +82,47 @@ const NUMERIC_SCALAR_TYPES = new Set([ 'rating', 'slider', 'progress', ]); +/** + * The builtin audit-timestamp columns every managed object carries. They are + * stamped to a single canonical instant format on SQLite (see + * `stampInsertTimestamps`/`update`) and read-repaired by + * `repairNaiveUtcAuditTimestamp`. + */ +const AUDIT_TIMESTAMP_COLUMNS = ['created_at', 'updated_at'] as const; + +/** + * Read-side repair for the builtin audit timestamps on SQLite. + * + * SQLite has no native timestamp type. Rows written before the canonical-format + * fix — or by a raw insert that fell back to the `CURRENT_TIMESTAMP` column + * default — hold a timezone-NAIVE, space-separated string + * (`'YYYY-MM-DD HH:MM:SS[.fff]'`). `Date.parse` reads such a zone-less string as + * LOCAL time, so a stored UTC wall-clock silently shifts by the host offset on + * any non-UTC runtime — the bug that made the objectos freshness probe never + * evict. Re-emit those values as canonical ISO-8601 with an explicit `Z`, + * interpreting the stored wall-clock as UTC (exactly what `CURRENT_TIMESTAMP` + * and the legacy UPDATE stamp both wrote). + * + * Idempotent and total: a value that already carries an explicit zone (`…Z` or + * `±HH:MM`) is returned unchanged, so re-reading a normalised row is a no-op + * (this keeps optimistic-lock `updated_at` tokens stable — see + * objectql `assertVersionMatch`). Non-strings (e.g. a `Field.datetime`-typed + * audit column stored as epoch-ms INTEGER) and unrecognised shapes pass + * through untouched. + */ +function repairNaiveUtcAuditTimestamp(value: unknown): unknown { + if (typeof value !== 'string') return value; + const s = value.trim(); + if (s === '') return value; + // Already zone-explicit (`…Z` or `±HH:MM`) — leave as-is (idempotent). + if (/[Zz]$/.test(s) || /[+-]\d{2}:?\d{2}$/.test(s)) return value; + // Zone-naive `YYYY-MM-DD[ T]HH:MM:SS[.fff]` → interpret the wall-clock as UTC. + const m = /^(\d{4}-\d{2}-\d{2})[ T](\d{2}:\d{2}:\d{2}(?:\.\d+)?)$/.exec(s); + if (!m) return value; + const d = new Date(`${m[1]}T${m[2]}Z`); + return Number.isNaN(d.getTime()) ? value : d.toISOString(); +} + // ── Introspection Types ────────────────────────────────────────────────────── export interface IntrospectedColumn { @@ -612,6 +653,7 @@ export class SqlDriver implements IDataDriver { const builder = this.getBuilder(object, options); const formatted = this.applyWriteColumnMap(object, this.formatInput(object, toInsert)); + this.stampInsertTimestamps(object, formatted); const result = await builder.insert(formatted).returning('*'); return this.formatOutput(object, result[0]); @@ -921,6 +963,33 @@ export class SqlDriver implements IDataDriver { } } + /** + * Stamp the builtin audit timestamps to one canonical ISO-8601-with-`Z` + * instant on the SQLite write paths (`create`/`bulkCreate`/`upsert`), so + * INSERT and UPDATE agree on a single zone-explicit format. + * + * Without this, an insert that omits `created_at`/`updated_at` falls back to + * the column's `CURRENT_TIMESTAMP` default, which on SQLite renders a + * zone-NAIVE, space-separated `'YYYY-MM-DD HH:MM:SS'` (no millis, no zone) — + * the same ambiguity the old UPDATE stamp had. Stamping app-side (rather than + * changing the column default) fixes this for EXISTING tenant databases + * immediately, since their tables keep the legacy default. Legacy/raw rows + * still written zone-naive are repaired on read by + * `repairNaiveUtcAuditTimestamp`. + * + * Only fills a slot the caller left empty — an explicit value (a seed fixture, + * the sys_metadata writer, a service outbox) is preserved. No-op for + * timestamp-less objects and for Postgres/MySQL, whose native `now()` column + * default already stores a zone-aware TIMESTAMP. + */ + protected stampInsertTimestamps(object: string, formatted: Record): void { + if (!this.isSqlite || !this.tablesWithTimestamps.has(object)) return; + const iso = new Date().toISOString(); + for (const col of AUDIT_TIMESTAMP_COLUMNS) { + if (formatted[col] === undefined || formatted[col] === null) formatted[col] = iso; + } + } + async update(object: string, id: string | number, data: Record, options?: DriverOptions): Promise { this.auditMissingTenant(object, 'update', options); const builder = this.getBuilder(object, options).where('id', id); @@ -928,12 +997,15 @@ export class SqlDriver implements IDataDriver { const formatted = this.applyWriteColumnMap(object, this.formatInput(object, data)); if (this.tablesWithTimestamps.has(object)) { - if (this.isSqlite) { - const now = new Date(); - formatted.updated_at = now.toISOString().replace('T', ' ').replace('Z', ''); - } else { - formatted.updated_at = this.knex.fn.now(); - } + // Canonical instant format. On SQLite (no native timestamp type) stamp + // full ISO-8601 WITH an explicit `Z` — matching the insert paths + // (`stampInsertTimestamps`) so create and update agree on one + // zone-explicit format. The previous `…replace('T',' ').replace('Z','')` + // wrote a zone-NAIVE, space-separated string that `Date.parse` reads as + // LOCAL time, silently shifting the instant by the host offset on a + // non-UTC runtime (the objectos freshness-probe miss). Postgres/MySQL keep + // native `now()` — a real zone-aware TIMESTAMP that never had the issue. + formatted.updated_at = this.isSqlite ? new Date().toISOString() : this.knex.fn.now(); } await builder.update(formatted); @@ -959,10 +1031,17 @@ export class SqlDriver implements IDataDriver { await this.fillAutoNumberFields(object, toUpsert, options); const formatted = this.applyWriteColumnMap(object, this.formatInput(object, toUpsert)); + this.stampInsertTimestamps(object, formatted); const mergeKeys = conflictKeys && conflictKeys.length > 0 ? conflictKeys : ['id']; const builder = this.getBuilder(object, options); - await builder.insert(formatted).onConflict(mergeKeys).merge(); + // `created_at` is insert-only — never overwrite it when an existing row is + // merged on conflict (the stamped/seeded value belongs to the original + // insert). Everything else (incl. `updated_at`) merges as before, so an + // upsert that updates a row still advances `updated_at`. + const mergeColumns = Object.keys(formatted).filter((c) => c !== 'created_at'); + const insertion = builder.insert(formatted).onConflict(mergeKeys); + await (mergeColumns.length > 0 ? insertion.merge(mergeColumns) : insertion.merge()); const readback = this.getBuilder(object, options).where('id', toUpsert.id); this.applyTenantScope(readback, object, options); @@ -990,6 +1069,9 @@ export class SqlDriver implements IDataDriver { // Reserve a persistent sequence value for each row's autonumber // field(s) — the engine no longer pre-fills these (see #1603). await this.fillAutoNumberFields(object, row, options); + // Same canonical-instant stamp as create()/upsert() so bulk-inserted + // rows don't fall back to the zone-naive CURRENT_TIMESTAMP default. + this.stampInsertTimestamps(object, row); } } const builder = this.getBuilder(object, options); @@ -2871,6 +2953,16 @@ export class SqlDriver implements IDataDriver { } } } + + // Builtin audit timestamps: repair any legacy/raw row stored as a + // zone-naive, space-separated string (CURRENT_TIMESTAMP or the pre-fix + // UPDATE stamp) to canonical ISO-8601 with `Z`, so reads are unambiguous + // and uniform regardless of when/how the row was written. Idempotent on + // already-canonical values; mirrors the legacy-row read-repair the + // `Field.date`/numeric paths already do. See `repairNaiveUtcAuditTimestamp`. + for (const col of AUDIT_TIMESTAMP_COLUMNS) { + if (data[col] !== undefined) data[col] = repairNaiveUtcAuditTimestamp(data[col]); + } } // ADR-0053 Phase 1: present `Field.date` as a timezone-naive `YYYY-MM-DD`