Merge remote-tracking branch 'origin/main' into claude/issue-6437-dropped-fields-reason-vocabulary

# Conflicts:
#	packages/spec/src/contracts/data-engine.ts

Author: claude

.changeset/grouping-notify-describe-align.md

@@ -0,0 +1,8 @@
+---
+'@objectstack/spec': patch
+---
+
+Align two schema `.describe()` strings with their measured acceptance faces (docs-only; no acceptance change — every previously-valid input is judged byte-identically):
+
+- `GroupingConfigSchema.fields` no longer claims "(supports up to 3 levels)". The gate is `.min(1)` with no upper bound, nothing downstream enforces a cap, and the grid renderer recurses over all configured levels — the describe now states the shape instead: array order is nesting order (first entry outermost), at least one field. (#7084)
+- `NotifyConfigSchema.sourceObject` / `sourceId` no longer say "Requires sourceId." / "Requires sourceObject.". The schema deliberately accepts the half-specified pair — the executor drops it at execute time so the inbox never renders a dead link (the module JSDoc's recorded contract) — and the describes now state that tolerance. (#7085)

.changeset/hook-ctx-previous-stash-limb-removed.md

@@ -0,0 +1,32 @@
+---
+'@objectstack/trigger-record-change': patch
+---
+
+record-change trigger: drop the `ctx.__previous` stash fallback — read the engine's declared `ctx.previous` only
+
+Behaviour is unchanged: the limb guarded against a producer that no longer
+exists. `plugin-audit`'s `captureBefore` was the **only** writer of
+`ctx.__previous` in the repo, and #6656 retired it, so `buildContext`'s
+
+```ts
+ctx.previous ?? (ctx as { __previous? }).__previous
+```
+
+had a second operand nothing could ever bind. The engine is the single producer
+of the pre-image and it binds the declared key ahead of every dispatch — by-id
+update (`engine.ts:7010`, immediately before the `beforeUpdate` dispatch at
+`:7012`), by-id delete (`bindPreImage`, `engine.ts:7869`, called at `:7897`
+before the `beforeDelete` dispatch at `:7899`), and each per-row context of a
+predicate write (`engine.ts:1746` after-phase, `:1825` before-phase) —
+#5272 / #5574 / #5846.
+
+Removed rather than kept "for safety", under ADR-0049 enforce-or-remove and
+PD #12: a fallback with zero producers is a second de-facto contract waiting to
+be rediscovered. The consequence is deliberate and stated here rather than left
+to be found — **a future producer of `ctx.__previous` is now silently ignored**;
+the declared way to hand this consumer a pre-image is `ctx.previous`.
+
+The test that fed the limb synthesised `__previous` in its own body, which is
+what kept the dead limb looking live (#4984). It is replaced by the inverted pin
+the deletion actually needs — the same treatment the `doc` alias got in #5671 —
+so restoring the limb goes red instead of unnoticed.

.changeset/icontains-reaches-every-js-face.md

@@ -0,0 +1,64 @@
+---
+"@objectstack/spec": minor
+"@objectstack/driver-memory": minor
+"@objectstack/driver-mongodb": minor
+"@objectstack/objectql": minor
+"@objectstack/formula": minor
+"@objectstack/service-analytics": minor
+---
+
+feat(spec,drivers,objectql,analytics,formula): `$icontains` reaches every JS evaluation face (#6520)
+
+The other half of #5702. That change implemented `$icontains` on the SQL family
+and correctly left the spec's `FILTER_OPERATORS` alone; this one adds the
+operator to that array and gives every remaining evaluation face an arm, in ONE
+change, because those two steps cannot be separated.
+
+**Why one PR.** `FILTER_OPERATORS` is not a word list, it is a runtime allowlist:
+`driver-memory`'s shape gate derives from it, and its matcher's `default:` arm
+assumes the gate already refused anything unimplemented. Measured on a branch
+that added the name early (#5701): the gate stopped refusing, the matcher fell
+through, and `match({ name: 'zzz' }, { name: { $icontains: 'acme' } })` returned
+`true` — the predicate silently dropped, every row matched. A dropped predicate
+does not narrow a query, it WIDENS it, and on an RLS read scope that is a
+permission bypass rather than a degraded feature (#3948). So the word list
+travels with the evaluators or not at all.
+
+**What now answers it**, all folding the same domain: `driver-memory` (query
+path, reference matcher, and the analytics/cube face), `driver-mongodb`,
+`objectql`'s `having`, `@objectstack/formula`'s `matchesFilterCondition` (the RLS
+write-side `check`), and `service-analytics`' three SQL compilers (the RLS
+lowering, the native-SQL strategy, and the `/analytics/sql` echo).
+
+**The fold is ASCII-only, and that is the contract, not an implementation
+detail** (#4706 Q1 = A). `$icontains: 'café'` does not match `CAFÉ`. Every face
+reads one shared definition — `foldAsciiCase` /
+`asciiCaseInsensitiveContains` / `asciiCaseInsensitiveRegexSource`, new exports
+on `@objectstack/spec/data` — because the two obvious per-package spellings are
+both wrong in the same direction: `toLowerCase()` folds the whole Unicode range,
+and so does a `RegExp` built with the `i` flag. SQLite folds ASCII only and three
+of the five drivers are SQLite underneath, so a Unicode fold on a JS face would
+re-open exactly the divergence the ruling closed. The pattern-binding faces
+(mingo, mongo) therefore emit one `[Aa]` character class per ASCII letter and
+pass NO flags; mongo's `$icontains` is the one arm in its family that does not
+set `$options: 'i'`.
+
+The comparand keeps the rules its SQL twin has: matched LITERALLY (`%`, `_` and
+regex metacharacters are ordinary characters), and refused when empty or
+non-string — an empty comparand matches every row, which is a predicate that
+constrains nothing.
+
+**User-visible effect.** A filter using `$icontains` now behaves the same on the
+in-memory double and on SQL, so an app whose tests run on one and whose
+production runs the other stops getting two answers from one filter. Downstream,
+#5814 (better-auth `Where.mode: 'insensitive'`) no longer hits a 400 on the
+memory double.
+
+Not changed, and still tracked: the `$contains` family still folds Unicode on
+`driver-memory`'s query path and `driver-mongodb` (#6682) — both remain DEBT rows
+in `scripts/check-driver-conformance.mjs`, now naming one open requirement each
+instead of two. `formula`'s unknown-operator posture stays a silent, fail-closed
+`false` (it governs a write-side check, where an unevaluable condition denies
+rather than widens); the decision and its limits are documented on
+`matches-filter.ts`, and no operator the spec DECLARES is answered that way any
+more.

.changeset/scim-case-insensitive-identifier.md

@@ -0,0 +1,47 @@
+---
+"@objectstack/plugin-auth": patch
+---
+
+fix(plugin-auth): honour better-auth's `Where.mode`, and normalise the identifier SCIM matches on (#5814)
+
+better-auth's `Where` carries a fourth field — `mode?: "sensitive" | "insensitive"`,
+`@default "sensitive"` — and `convertWhere()` in the ObjectQL adapter read `field` /
+`operator` / `value` and nothing else. The default covers almost every caller, so the
+drop was invisible; the caller it is not invisible for is the one that explicitly asked.
+
+`@better-auth/scim` is that caller. SCIM's `userName` is case-insensitive by RFC 7643
+(`caseExact: false`), so a `filter=userName eq "Alice@example.com"` reaches this adapter
+as `{ field: 'email', operator: 'eq', mode: 'insensitive' }`. With `mode` unread, whether
+it matched a user stored as `alice@example.com` came down to how the driver under the
+auth path happens to compare strings — and because SCIM provisioning is "look up, create
+if absent", a missed match did not raise an error, it provisioned a **second user**.
+Only deployments that turned SCIM on (`OS_SCIM_ENABLED`, off by default) were exposed.
+
+Both halves of the fix, per the maintainer's ruling on #5814:
+
+- **Normalisation, not new vocabulary.** `sys_user.email` — the field SCIM's `userName`
+  maps onto — is now stored lower-cased and compared lower-cased by this adapter. An
+  insensitive lookup lower-cases its comparand, which is an *exact* match against the
+  stored form, so nothing in the query vocabulary changes. The set is a declared table
+  (`NORMALISED_IDENTIFIER_FIELDS`), not a name heuristic, and it drives the read and
+  write halves from one place so a field cannot be added to one of them only.
+- **The silent drop ends.** `convertWhere()` handles `mode` explicitly. On a normalised
+  identifier the request is satisfied by construction. On **any other** field, a
+  `mode: 'insensitive'` clause now emits a loud warning naming the model, the field and
+  the operator, and stating that the query is being answered case-sensitively — instead
+  of answering a different question and looking fine doing it. It deliberately does not
+  throw: refusing here would turn an occasional duplicate user into "`userName` queries
+  entirely unavailable", which is the worse trade on an authentication path.
+
+No migration ships and none is needed. Every existing write path already lower-cased
+`user.email` before reaching the adapter (better-auth's own `internalAdapter` does it on
+`createUser` / `createOAuthUser` / `updateUser` / `updateUserByEmail`, and SCIM's create
+path does it again), so the write half changes no existing behaviour — it moves the
+invariant the read half depends on into the layer that depends on it, instead of
+inheriting it from an internal of a prerelease dependency. Queries that do not set
+`mode`, or set it to `"sensitive"`, keep their comparand byte-for-byte: folding case
+unasked would be the same failure in the opposite direction.
+
+Adding a case-insensitive equality operator (`$ieq`) was deferred until there is
+demonstrated pull for it, and downgrading `eq + insensitive` to `$icontains` was
+rejected — containment is not equality.

.changeset/searchable-fields-formula-refused.md

@@ -0,0 +1,68 @@
+---
+"@objectstack/metadata-protocol": patch
+"@objectstack/spec": patch
+"@objectstack/lint": patch
+---
+
+fix(spec,lint): a virtual `formula` field in `searchableFields` is refused loudly, not admitted verbatim (#6674)
+
+#4254 closed the fail-open on the unknown-name axis: a `$searchFields` entry the
+engine would not scan is `400 INVALID_FIELD`, never a silently widened search.
+The same shape survived one axis over, on names that are perfectly real.
+
+The declared branch of `resolveSearchFieldResolution` filtered entries by
+EXISTENCE only, so a `formula` field declared in `searchableFields` entered the
+allowed set — and the ingress gate, which reads that same set, accepted it for
+exactly that reason. Measured on `origin/main`:
+
+```
+AUTO:          {"allowed":["name","project_name"],"source":"auto"}                formula excluded
+DECL-FORMULA:  {"allowed":["name","project_name_formula"],"source":"declared"}    admitted verbatim
+?search=Apollo&searchFields=project_name_formula  ->  200, 0 rows                 silent
+```
+
+Zero rows is the defect. A formula value is computed on read and no driver
+materializes a column for it (`driver-sql` `fieldHasColumn`, driver-turso's
+"Virtual — no column"), so the `$contains` the engine expands `$search` into has
+nothing to scan: 0 rows on driver-memory (the property is absent from the stored
+row) and 0 rows WITH NO ERROR on driver-sql/better-sqlite3. The declaration read
+as search coverage and delivered none.
+
+- **`@objectstack/spec` — the deciding face.** The declared branch now filters on
+  existence AND scannability: an entry naming a virtual field is not admitted.
+  New exports `SEARCH_VIRTUAL_TYPES` (exactly `formula`, pinned) and
+  `isVirtualSearchField` — one judgment, so the resolution, the gate and the
+  linter cannot drift about which types have a column. The resolution itself
+  stays non-throwing: it is consulted on every search by internal callers that
+  never pass an ingress, which is why #4254 put the loudness at the ingress.
+- **`@objectstack/metadata-protocol` — `400 INVALID_FIELD` with its own reason.**
+  Split out before the declared/auto branch, because both of those messages are
+  wrong for it: "outside the declared set" is false when the entry IS in the
+  list, and the auto-default's "declare `searchableFields` to choose the
+  searchable set" would instruct the author to write the declaration being
+  refused. The new message names the field, its type, that the value is computed
+  on read and never stored, and the fix (mirror onto a stored text field).
+- **`@objectstack/lint` — a build error at authoring time**, on the object's own
+  `searchableFields` as well as a view's narrowing, under the existing
+  `searchable-field-unsearchable` rule (no new rule id). This narrows the
+  canonical surface, which #4830 had deliberately left existence-only.
+
+The carve-out that made canonical existence-only is deliberately KEPT and pinned
+by controls in all three packages: the dividing line is STORAGE, not search
+quality. A `json` or `lookup` column declared in `searchableFields` is still the
+author's choice and still executed — a `$contains` over the stored JSON text or
+the stored foreign key. Narrow and rarely useful, but a scan that CAN match, so
+it is neither a 400 nor a finding. Only "there is no column at all" is refused.
+
+**Compatibility.** A corpus sweep of this repo plus `objectui` and `cloud` found
+ZERO authored `searchableFields` naming a formula-typed field, so nothing in the
+tree changes verdict. For an already-published object that does carry one:
+loading is unaffected (no schema-parse change — `searchableFields` is still
+`z.array(z.string())`, this is a resolution and enforcement rule); a plain
+`?search=` keeps returning the SAME rows, because the dropped entry matched none
+of them; only a request that NAMES the formula field flips from `200` with no
+rows to `400 INVALID_FIELD` — including objectui's list search, which echoes the
+declaration verbatim. An object whose `searchableFields` is ENTIRELY formula
+entries filters to empty and falls through to the auto-default, exactly as an
+all-stale declaration has since #4254; the linter reports the declaration rather
+than leaving that swap silent.

.changeset/sort-formula-field-refusal.md

@@ -0,0 +1,59 @@
+---
+"@objectstack/metadata-protocol": minor
+---
+
+fix(metadata-protocol): refuse a sort naming a `formula` field instead of dropping it silently (#6994)
+
+The list path's SORT gate (`assertSortFieldsExist`) refuses a sort naming a field
+the object does not have (#4226) and a dotted path that would have to cross into
+a related record (#4256). It did **not** refuse a name that is a real,
+non-dotted field of the object whose **type** materialises no column — a
+`formula` field is in the object's field map, so it passed the unknown check,
+and it carries no dot, so it passed the dotted check.
+
+It then reached a driver that has no column for it. Re-measured on a real
+`SqlDriver` (better-sqlite3, on-disk) driving a real `ObjectQL` engine with this
+protocol on top, over five rows inserted `C A E B D` and a formula field
+`sort_key` whose expression is `record.title`:
+
+```
+CONTROL   orderBy title asc     -> ["A","B","C","D","E"]   a real column really sorts
+BASELINE  no sort               -> ["C","A","E","B","D"]   insertion order
+
+FORMULA   orderBy sort_key asc  -> ["C","A","E","B","D"]   5 rows, 200
+            its sort_key values -> ["C","A","E","B","D"]
+FORMULA   orderBy sort_key desc -> ["C","A","E","B","D"]   byte-identical to asc
+
+RAW SQL   order by sort_key     -> sqlite: no such column: sort_key
+```
+
+`asc` and `desc` coming back identical is what makes this a dropped sort rather
+than a coincidence: `SqlDriver.createColumn` returns early for `formula` (it is
+virtual — computed on read, after `driver.find` has already returned), sqlite
+answers `no such column`, and the #3821 unknown-column backstop retries the
+query **without** the `ORDER BY`. The response even carries the values it was
+asked to order by, out of order, under a 200 — so it contradicts the request in
+plain view and still reports success. `sort` + `top` is how a caller asks for
+"the latest N", which this turned into an arbitrary N.
+
+**Now:** `400 INVALID_SORT`, naming the field and its type, and prescribing the
+same remedy in the same words as the dotted refusal (#6924) and the SEARCH axis
+(#6673) — denormalise onto a **stored field, written when the source changes**.
+Precedence on this axis is `unknown` > `dotted` > unmaterializable, so both
+older verdicts answer exactly what they answered before.
+
+**`summary` / `rollup` is not affected** and deliberately not in the refused
+set: a summary field gets a real, maintained `float` column and genuinely sorts.
+The spec's `COMPUTED_VALUE_TYPES` (`formula`/`summary`/`autonumber`) is the
+WRITE contract and is the wrong set to gate a sort with — it would refuse two
+types that work.
+
+**Scope.** This is an ingress gate, so it covers what reaches `findData`: the
+REST list route, `POST /data/:object/query`, the export route, and the RPC
+dispatcher. An internal caller that reaches `engine.find()` directly (hooks,
+flows, reports) still gets the silent drop — closing that half means deciding
+whether the engine refuses or keeps its documented internal-caller tolerance,
+which is a separate contract decision and is tracked separately.
+
+If you were sorting a list by a formula field, that sort was never applied; the
+call now fails loudly instead of returning rows in an arbitrary order.

.changeset/strict-insert-contract-tsdoc.md

@@ -0,0 +1,23 @@
+---
+"@objectstack/spec": patch
+---
+
+docs(spec): `WriteObservabilityOptions.strictReadonlyWrites` no longer claims INSERT ignores it (#7064)
+
+The contract's closing paragraph still said "INSERT ignores it … insert is
+exempt from both strips, so there is nothing to refuse" — true when #5126
+shipped the option, false since #5503 wired `engine.insert` to REFUSE a
+payload carrying a runtime-owned value (`RUNTIME_OWNED_FIELD_TYPES`, today
+`autonumber`) under `strictReadonlyWrites: true`, throwing
+`ReadonlyFieldRejectedError` (`ERR_READONLY_FIELD_REJECTED`,
+`operation: 'insert'`) and writing nothing.
+
+The TSDoc now states, measured against the engine: insert stays exempt from
+the two author-declared strips at this seam (#3413 — an in-process create may
+seed a `readonly: true` field's initial value; `readonlyWhen` cannot lock a
+create), while the runtime-owned strip runs on insert and is exactly what
+strict refuses; the exempt writers are the ones the error message names
+(`isSystem`, and `preserveAudit` for a #3493 historical import), explicitly
+scoped to this in-process seam so the DataProtocol ingress policy
+(#3043/#6640, `FieldSchema.readonly`) stays a distinct layer. Prose only — no
+key, type, or behaviour changes.

.github/workflows/lint.yml

@@ -117,8 +117,13 @@ jobs:
       - name: "@objectstack/verify stand-in erasure guard"
         run: pnpm check:verify-stand-in
 
-      # Raw control-byte guard (#3127 / #4890 / #5157 / #5460). Scans every
-      # tracked TEXT file for a raw ASCII control byte and fails on any hit.
+      # Raw control-byte guard (#3127 / #4890 / #5157 / #5460 / #6984). Scans
+      # every TEXT file git knows about for a raw ASCII control byte and fails on
+      # any hit. Since #6984 the scan set is the index PLUS untracked-but-not-
+      # ignored working-tree files, so a locally-authored file is covered before
+      # it is staged; here that widening is a no-op, because a workflow checks out
+      # a commit and has no untracked files at all (the step's summary line says
+      # so — it names both halves, and the untracked one reads 0 in CI).
       # WHICH bytes are in the set and WHY each is rejected are stated and argued
       # once, in the gate script's header — `scripts/check-nul-bytes.mjs`. That
       # header is authoritative and this comment cites it rather than restating it