objectstack-ai
diff --git a/‎.changeset/regex-retirement-icontains-drivers.md‎
Lines changed: 53 additions & 0 deletions b/‎.changeset/regex-retirement-icontains-drivers.md‎
Lines changed: 53 additions & 0 deletions
diff --git a/‎docs/protocol-upgrade-guide.md‎
Lines changed: 3 additions & 0 deletions b/‎docs/protocol-upgrade-guide.md‎
Lines changed: 3 additions & 0 deletions
@@ -0,0 +1,53 @@
+---
+"@objectstack/driver-sql": minor
+"@objectstack/driver-sqlite-wasm": minor
+"@objectstack/driver-turso": minor
+"@objectstack/driver-memory": minor
+"@objectstack/driver-mongodb": minor
+"@objectstack/objectql": minor
+---
+
+feat(drivers,objectql): `$regex` / `$options` are refused everywhere, and `$icontains` is implemented on the SQL family (#5702)
+
+The driver half of the #4706 ruling. #5701 landed the contract (the vocabulary,
+the `RETIRED_FILTER_OPERATORS` prescriptions, the shared text case-set) and
+#5710 flipped the last live producer — `plugin-auth`'s ObjectQL adapter, which
+emitted `$regex` on the authentication path — so the refusal can now land
+without breaking sign-in.
+
+**BREAKING for anyone writing `$regex` or `$options` in a filter.** Both are
+refused on every backend with `INVALID_FILTER` / 400 and a message that names
+the replacement. `$regex` was never a declared operator: `driver-sql` compiled
+it to a LIKE-escaped substring (so `a.b` matched only the literal `a.b`),
+`driver-memory` ran it as a real `RegExp` (so the same filter also matched
+`axb`, and an *invalid* pattern was caught and answered `false` — zero rows, in
+silence), and `objectql`'s `having` did the same. Write `$icontains` for the
+case-insensitive substring search this was almost always used for, `$contains`
+for a case-sensitive one; a pattern that genuinely needs a regex has no
+filter-level replacement.
+
+**`$icontains` now runs on the SQL family** — `driver-sql`, `driver-sqlite-wasm`,
+and both of `driver-turso`'s transports (the remote one does not go through
+knex, so it needed its own). It compiles to `LOWER(col) LIKE LOWER(?) ESCAPE ?`
+through the same `applyLike` / `pushLike` that carries the `%` / `_` / `\`
+escaping, as a `fold` parameter rather than a second emitter — a copied emitter
+is where the escape class would have been dropped, and an unescaped `%` matches
+every row. An empty or non-string comparand is refused on the validating walk
+(an empty one matches every row, which widens rather than narrows). On SQLite
+`lower()` folds ASCII only, which IS the contract (#4706 Q1 = A): `$icontains:
+'café'` does not match `CAFÉ`.
+
+<!-- adr-0087: registered filter-regex-options-retired -->
+
+`driver-mongodb`'s unknown-operator arm was throwing a bare `Error` with no
+`code` and no `status`, three lines from the helper in its own file that sets
+`INVALID_FILTER` / 400 — a 500-shaped body for a 400-class client mistake. It
+now speaks the same envelope as its three siblings.
+
+Two parts of the ruling are deliberately NOT in this change and stay tracked in
+`scripts/check-driver-conformance.mjs`'s ledger: the `$contains` family's
+case-sensitivity (#4706 Q2 = A) needs SQLite's `LIKE` replaced by a case-exact
+construct in the driver, the RLS lowering and the analytics lowering together,
+or one permission rule compiles to two row sets (#6518); and `$icontains` on the
+JS evaluation faces needs the spec vocabulary to take the operator, which cannot
+happen before `driver-memory` has an arm for it (#6520).
@@ -375,6 +375,9 @@ Finally it narrows the aggregation vocabulary: `array_agg` and `string_agg` leav
 - **`driver-sql-distinct-bare-filter-typed`** — `SqlDriver.distinct() third argument — any value` → a bare FilterCondition (@objectstack/spec/data) — the same value find() carries under query.where, never a query envelope
   - Why not automatic: This entry records a TYPE being added, not a surface being withdrawn, and it says so up front because the distinction decides who has to do anything. `distinct` is not declared on `IDataDriver`, so #5181 / #6075 never reached it and it kept `filters?: any` while its body said something far more specific — `applyFilters(builder, filters)` is handed the ARGUMENT ITSELF, never a `.where` off it. ⚠️ RUNTIME BEHAVIOUR IS UNCHANGED by this entry's change: not one statement moved, so no upgrade breaks at run time and nothing that answered correctly stops. What the annotation removes is a compile-time hole, measured rather than assumed: a truthy NON-OBJECT third argument — `distinct('orders', 'product', 'completed')` — used to type-check and resolve the UNFILTERED set, because `applyFilters` emits no predicate at all for a truthy non-object, non-array filter. A call meaning "which products among completed orders" answered with EVERY product, silently. That spelling is now TS2345 at the call site. This is a driver CALL ARGUMENT — code, never stack metadata — so there is no source for the D2 chain to rewrite and deliberately no schema tombstone, the disposition `data-driver-find-stream-retired` (#4484), `storage-service-list-retired` (#5540), `actor-user-roles-to-positions` (#6011) and `driver-aggregate-undeclared-key-aliases-removed` (#6321) already carry. ⚠️ It differs from those four in ONE measured way a reader should not have to infer: because nothing changed at run time, an untyped JS caller is not affected BY THE UPGRADE at all. The entry is here for a different reason — such a caller is exactly the one tsc can never reach, and the silent widening above is a defect they may ALREADY be sitting on, before and after this major. The generated upgrade guide is the only channel that reaches them, which is why the fix is written down rather than left to the compiler. ⛔ The reverse mismatch is NOT closed and no type can close it: `FilterCondition` is an open map (`[key: string]: any`) because a filter key IS a field name, so a query envelope `{ object, where }` is structurally a valid filter — one constraining columns named `object` and `where` — and so is a FilterArray. Both reach `distinct` type-checked and are refused at run time, loudly, with INVALID_FILTER / 400. `driver-memory`'s opposite half — where the BARE spelling returns the unfiltered set in silence — stays open under the #5499 freeze (#6320). ADR-0087, #6320.
   - Done when: No caller passes a non-object to `distinct()`'s third argument. A scalar there is now a compile error (`TS2345: Argument of type 'string' is not assignable to parameter of type 'FilterCondition'`); rewrite it as the bare filter it was always meant to be — `'completed'` becomes `{ status: 'completed' }`. ⚠️ That is NOT an equivalent rewrite: the old spelling returned the UNFILTERED set, so the answer changes once fixed, and the changed answer is the one the call always meant. An untyped JS caller gets no compile error and no behaviour change — for them this entry is the only notice that the spelling never filtered anything. A query envelope or a FilterArray in that slot still compiles and is rejected at run time with INVALID_FILTER / 400.
+- **`filter-regex-options-retired`** — `data.filter $regex / $options — in a STORED filter (dashboard widget filter and globalFilters, report runtimeFilter, page and component filter, solution-blueprint filter), and equally in the where clause of a query request` → $icontains for the case-insensitive substring match this was almost always used for, or $contains for a case-sensitive one — a pattern that genuinely needs a regular expression has no filter-level replacement
+  - Why not automatic: Like `driver-aggregate-undeclared-key-aliases-removed` and `driver-sql-distinct-bare-filter-typed`, this entry records a LENIENCY being withdrawn rather than a declared surface: `$regex` was never in `FILTER_OPERATORS` and never a key on `StringOperatorSchema`. That is measured, not assumed — `git log -S'$regex'` over `packages/spec/src` returns only doc comments describing how `$contains` LOWERS to MongoDB (`Contains substring - SQL: LIKE %?% | MongoDB: $regex`), plus #5701 itself, which added the name solely as `RETIRED_FILTER_OPERATORS` prescription data. ⚠️ But it differs from those two in the one way that decides the disposition, so a reader should not have to infer it: those were driver CALL ARGUMENTS, code and never stack metadata, whereas a filter IS stored metadata. `FilterConditionSchema` is an OPEN RECORD (`z.record(z.string(), z.unknown())`) because a filter key is a field name, so a stored `{ name: { $regex: 'acme.*' } }` parses GREEN and always will — a `retiredKey()` tombstone cannot exist on an open map, which is exactly why the ledger has to carry this. What such a stack used to get was four different answers from four backends: `driver-sql` and Turso's remote transport compiled it to a LIKE-escaped SUBSTRING (so `a.b` matched only the literal `a.b` and the regex was silently never a regex), `driver-memory` and objectql's `having` ran it as a real `RegExp` (so the same filter also matched `axb`, and an INVALID pattern was caught and answered `false` — zero rows, in silence), and `driver-mongodb` refused it with a bare `Error` carrying no `code` and no `status`. It is now refused everywhere with INVALID_FILTER / 400 naming the replacement. There is deliberately NO D2 conversion and this sits in `semantic` rather than among the mechanical transforms: rewriting `$regex` to `$icontains` is NOT lossless in either direction — a regex metacharacter becomes a literal — so an auto-applied rewrite would silently change which rows a dashboard, report or permission filter selects, a wrong number rather than a missing one. Choosing the substring the pattern MEANT is a judgment about the query, not a transform. ⚠️ This entry covers BOTH HALVES of the #4706 ruling (B), not just the driver one: the contract half (#5701 — the `$icontains` declaration, the `$contains` family pinned case-sensitive, and the `RETIRED_FILTER_OPERATORS` prescriptions) landed before the ADR-0087 disposition gate (#6148) existed and so was never asked for a ledger entry; the driver half (#5702) is where the refusal became executable. One surface, one entry, registered from the half that made it observable. ADR-0049 / ADR-0087, #4706 / #5701 / #5702.
+  - Done when: No stored filter and no request `where` spells `$regex` or `$options` — grep the stack for both. Each one is rewritten by asking what the pattern MEANT, not by transliterating it: a bare substring pattern becomes `$icontains` (or `$contains` when the match must stay case-sensitive), and its metacharacters are dropped rather than escaped, because they were never honoured as a regex on the SQL family in the first place. ⚠️ Expect the answer to CHANGE on any stack that ran on `driver-memory`, `driver-mongodb` or objectql `having`, where the pattern really was evaluated as a regular expression; on the SQL family the rewritten filter returns what it always returned. A pattern that genuinely needs alternation, anchoring or character classes has no filter-level replacement — move that predicate into a formula field or a server-side view, or open an issue for it. Verify by loading the stack: a surviving `$regex` or `$options` is answered INVALID_FILTER / 400 with a message naming the replacement, on every backend.
 - **`http-server-runtime-vocabulary-retired`** — `system.serverEvent / system.serverEventType / system.serverCapabilities / system.serverStatus (the lifecycle-event, capability-report and status vocabulary of system/http-server.zod.ts — 4 defs, 8 exported names)` → (removed — there is no replacement key, because there was never a key. Server lifecycle is the transport plugin's own start/stop seam; per-request and per-server observability is `system/metrics.zod.ts` and `system/logging.zod.ts` (plus `OS_SERVER_TIMING` for timings), and liveness is the `/health` endpoint. What a transport plugin can DO it states by implementing the kernel plugin contract — the seams it registers are the capability statement, and a self-described capability record can only disagree with them. Server-level configuration that IS authorable lives on `defineStack({ server })` / `StackServerConfigSchema`, which is unaffected)
   - Why not automatic: The second and final ADR-0049 pass over `system/http-server.zod.ts`. #4938 removed the CONFIG half (`HttpServerConfigSchema`, nine keys, zero readers, zero authoring entry); this removes the RUNTIME half — a 7-member lifecycle event union with a timestamped envelope, an eight-boolean capability report, and a five-state status record with connection and request counters. Nothing ever emitted, consumed or parsed any of them. This card was HELD for four days rather than queued, on a specific and legitimate doubt: a response/capability vocabulary can be a REFERENCE surface for host implementers, so "zero consumers in this repo" is weaker evidence for one of those than for an authorable key (the CSS-variable rebuttal). The hold was lifted by measuring the reference reader itself rather than by re-running the same grep: `plugin-hono-server`, the one in-tree host implementation, neither implements nor reports any of the three — it names no capability record, no status shape and no event union, and what it registers is routes and middleware through the kernel plugin contract. A declaration-site grep put every declaration in this one file, a quoted-name sweep across objectstack and objectui found no reader outside it, and the control passed in the SAME run: `MiddlewareConfig`, declared twelve lines away, resolves to `packages/runtime/src/middleware.ts`. So the sweep could see a reader in this file when there was one. With no carrier key there is nothing to tombstone, and with no author there is no source or `sys_metadata` row for a D2 conversion to rewrite: RETIRED_DEFS_BY_MAJOR plus this entry are the declaration — route 3, the same shape as #4938 in this very file, #4834, #4988 and #5055. If host-implementer conformance becomes a real requirement it returns through the ENFORCE route: an adapter contract with a checker behind it, vocabulary second. ADR-0049, #5295.
   - Done when: No source imports `ServerEvent`, `ServerEventType`, `ServerEventSchema`, `ServerCapabilities`, `ServerCapabilitiesSchema`, `ServerCapabilitiesParsed`, `ServerStatus` or `ServerStatusSchema` from `@objectstack/spec/system` — a grep over consumer code resolves none of them, and `tsc` reports TS2724/TS2305 on any that survives. The route-registration half of the same module still resolves (`RouteHandlerMetadataSchema`, `MiddlewareType`, `MiddlewareConfigSchema`, `MiddlewareConfig`), and `StackServerConfigSchema` — the one authorable server surface — is untouched: a stack declaring `server: { trustProxy, security }` parses exactly as it did in 16.x.