fix(driver-sql): $not 取反前先把操作数编译成全域谓词,NULL 行不再被静默排除 (#5146) (#5296)

SQL 是三值逻辑:`NULL = 'won'` 是 UNKNOWN,`NOT UNKNOWN` 仍是 UNKNOWN,而 `WHERE`
只保留 TRUE。于是 `applyFilterCondition` 发出的裸 `not (stage = 'won')` 把「该列没有
值」的行整批丢掉,而 `driver-memory` 的 `match()` 与 `formula` 的
`matchesFilterCondition` 用普通两值 JS 求值(`undefined !== 'won'` → 行匹配),把同
一批行**全部返回**。一个 spec 声明的算子,答案取决于跑它的是哪个驱动 —— 而权限规则
里的 CEL `!expr` 经 `cel-to-filter.ts` 正是降解成 `{ $not: {…} }`,所以同一条 read
scope 在不同后端准入的行集不同。#5146 判定以 JS 家族的答案为准(2:1 多数派),本次
把 SQL 侧对齐。

`$not` 的操作数在取反前经 `nullSafeNegationOperand` 改写,使每个叶子都编译成**全域
(total)**谓词 —— 永远 TRUE 或 FALSE,不会 UNKNOWN:

    -- 之前:not (`stage` = 'won')
    -- 现在:not ((`stage` is not null) and (`stage` = 'won'))

对 issue 给出的扁平形状,这与 `NOT (…) OR col IS NULL` 等价。守卫下推到**叶子**而非
挂在 `NOT` 旁边,是为了在操作数嵌套时仍然正确:`$not` 内含 `$or` 时,顶层的
`OR col IS NULL` 会把 JS 家族排除的行(某列为 NULL、但另一分支成立)重新放进来。

守卫方向按算子逐个判定,不是一刀切:`$ne` / `$nin` / `$notContains` 走
`col IS NULL OR (…)`,`$eq` / `$in` / `$gt` / `$contains` 一族走
`col IS NOT NULL AND (…)`,`$null` / `$exists` / `$eq: null` / `$ne: null` 本就是全域
谓词、一个字节不加。无条件加 `OR col IS NULL` 会把 `{$not: {a: {$ne: 5}}}`(语义是
「a 就是 5」)静默放松成也返回 NULL 行 —— 正是 #2704 / #5134 那一族缺陷。

只有 `$not` 路径被改写:普通比较的 SQL 逐字符不变,没有非否定谓词因此失去索引。
#5134 / PR #5243 的布尔单位元与 ADR-0112 形状拒收全部保持;`{ field: {} }`(#5240)
刻意不在此裁定,编译结果与之前完全一致。

`driver-memory` 与 `formula` 无需改动 —— 三家各补一组 pin 测试,把「值缺失行在
`$not` 下的去留」钉在一起,并把两个 JS 后端彼此不一致的角落(`$notContains`、
`$exists`、缺键 `$nin`)按实测钉住而不是假装一致。

Claude-Session: https://claude.ai/code/session_01Pbu27iNUfQCHeuS551Rqo7

Co-authored-by: Claude <noreply@anthropic.com>

Author: os-zhuang

.changeset/sql-driver-not-null-safe.md

@@ -0,0 +1,54 @@
+---
+"@objectstack/driver-sql": patch
+---
+
+fix(driver-sql): `$not` 改为 NULL-safe —— 被比较列为 NULL 的行不再被否定条件静默排除
+
+**这是一处可观察的查询行为变更,且直接关系到 RLS 的可见集合。**
+`{ $not: { stage: 'won' } }` 以前**不返回** `stage IS NULL` 的行,现在**返回**它们。
+如果你的规则依赖了旧行为,它依赖的是「同一条规则在不同后端给出不同可见集合」。
+
+SQL 是三值逻辑:`NULL = 'won'` 是 UNKNOWN,`NOT UNKNOWN` 仍是 UNKNOWN,而 `WHERE`
+只保留 TRUE。于是 `applyFilterCondition` 编译出的裸 `NOT (stage = 'won')` 会把
+「该列没有值」的行整批丢掉;同一条 filter 在 `driver-memory` 与 `formula` 的
+`matchesFilterCondition` 上是普通的两值 JS 求值(`undefined !== 'won'` → 行匹配),
+两边把这些行**都返回**。一个 spec 声明的算子,答案取决于跑它的是哪个驱动。
+
+这不是「数目对不上」而已:权限规则里的 CEL `!expr` 经 `cel-to-filter.ts` 正是降解成
+`{ $not: {…} }`,所以同一条 read scope 在 SQL 数据源与内存数据源上准入的行集不同。
+#5146 判定以 JS 家族的答案为准(2:1 的多数派;写 `!(stage == 'won')` 的人不会预期
+「stage 为空的行被隐藏」),本次把 SQL 侧对齐过去。
+
+**编译出来的形状。** `$not` 的操作数在取反之前先被改写成**全域(total)谓词** ——
+永远是 TRUE 或 FALSE,不会是 UNKNOWN:
+
+```sql
+-- 之前
+not (`stage` = 'won')
+-- 现在
+not ((`stage` is not null) and (`stage` = 'won'))
+```
+
+对 issue 里给出的扁平形状,这与 `NOT (…) OR col IS NULL` 完全等价。把守卫下推到
+**每个叶子**而不是挂在 `NOT` 旁边,是为了在操作数嵌套时仍然正确:`$not` 里套一个
+`$or` 时,顶层的 `OR col IS NULL` 会把 JS 家族排除的行重新放进来(某一列为 NULL、
+但另一个析取分支成立的行)。
+
+**守卫方向按算子逐个判定,不是一刀切。** `{ $not: { a: { $ne: 5 } } }` 的语义是
+「a 就是 5」,两个 JS 后端都把 NULL 行排除在外;无条件加 `OR a IS NULL` 会把这些行
+交回去 —— 正是本驱动反复付过学费的静默放松(#2704 / #5134)。因此
+`$ne` / `$nin` / `$notContains` 用的是 `col IS NULL OR (…)`,`$eq` / `$in` /
+`$gt` / `$contains` 一族用 `col IS NOT NULL AND (…)`,而 `$null` / `$exists` /
+`$eq: null` / `$ne: null` 本来就是全域谓词,一个字节都不加。
+
+**只有 `$not` 路径被改写。** 普通比较的 SQL 逐字符不变(`{ a: 1 }` 仍然是
+`a = 1`),因此没有任何非否定谓词因此失去索引;`$not` 路径上的 `IS NOT NULL` 守卫
+本身处在一个原本就不可 sargable 的 `NOT (…)` 里。
+
+`#5134` / PR #5243 定下的布尔单位元(`{ $not: {} }` → 零行、`$not` of FALSE →
+全部行、非 filter 节点的操作数按 ADR-0112 响亮拒收)全部保持不变;`{ field: {} }`
+(#5240)也刻意不在此裁定 —— 它编译出的 SQL 与之前完全一致。
+
+`driver-memory` 与 `formula` 无需改动,本次为三家各补了一组 pin 测试,把「值缺失
+行在 `$not` 下的去留」钉在一起。跨驱动 conformance case(`FILTER_LOGIC_CASES`)与
+契约 TSDoc 归 spec 车道,随 #5239 落地。

packages/formula/src/matches-filter-not-null-safe.test.ts

@@ -0,0 +1,154 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * [#5146] `$not` and absent values — the answers this evaluator gives, pinned.
+ *
+ * `matchesFilterCondition` needed no change for #5146: it already negates in
+ * ordinary two-valued JS, so `{ $not: { stage: 'won' } }` matches a record whose
+ * `stage` is null or missing. `driver-sql` used to disagree (SQL's
+ * `NOT (stage = 'won')` is UNKNOWN for a NULL column, and a `WHERE` drops it),
+ * so the SAME rule admitted different rows per backend — and this evaluator is
+ * the RLS write-side `check`, i.e. the half that decides whether a write is
+ * allowed, against read scopes compiled elsewhere. #5146 ruled this answer
+ * canonical and rewrote the SQL compiler to match it.
+ *
+ * These cases are therefore a PIN on the reference behaviour, mirrored id-for-id
+ * by `driver-sql`'s `sql-driver-not-null-safe.test.ts` and `driver-memory`'s
+ * `memory-matcher-not-null-safe.test.ts`. Moving an expectation here silently
+ * re-opens the divergence.
+ *
+ * `cel-to-filter.ts` is why this matters in practice: a CEL `!expr` in a
+ * permission rule lowers to exactly these `$not` shapes.
+ *
+ * Home for these eventually: `FILTER_LOGIC_CASES` in `@objectstack/spec/data`
+ * (spec lane, with #5239).
+ */
+
+import { describe, it, expect } from 'vitest';
+import { matchesFilterCondition } from './matches-filter.js';
+import type { FilterCondition } from '@objectstack/spec/data';
+
+/** Fields present but null — how a SQL NULL round-trips into a record. */
+const NULLED: Array<Record<string, unknown>> = [
+  { id: '1', stage: 'won', owner: 'u1', amount: 10 },
+  { id: '2', stage: 'lost', owner: 'u2', amount: 20 },
+  { id: '3', stage: null, owner: 'u1', amount: null },
+  { id: '4', stage: null, owner: null, amount: 40 },
+];
+
+/** The same rows with the null fields ABSENT — a partial write's post-image. */
+const MISSING: Array<Record<string, unknown>> = [
+  { id: '1', stage: 'won', owner: 'u1', amount: 10 },
+  { id: '2', stage: 'lost', owner: 'u2', amount: 20 },
+  { id: '3', owner: 'u1' },
+  { id: '4', amount: 40 },
+];
+
+const ALL = ['1', '2', '3', '4'];
+
+const ids = (rows: Array<Record<string, unknown>>, filter: unknown): string[] =>
+  rows.filter((r) => matchesFilterCondition(r, filter as FilterCondition)).map((r) => String(r.id));
+
+/** Both readings of "no value" must give the same answer unless noted. */
+const matched = (filter: unknown): string[] => {
+  const nulled = ids(NULLED, filter);
+  expect(ids(MISSING, filter), 'a null field and an absent field must match alike').toEqual(nulled);
+  return nulled;
+};
+
+describe('[#5146] matchesFilterCondition — $not over records with no value', () => {
+  describe('a record with no value does not satisfy the negated condition', () => {
+    it('$not on an implicit equality matches the value-less records', () => {
+      expect(matched({ $not: { stage: 'won' } })).toEqual(['2', '3', '4']);
+    });
+
+    it('$not over multiple keys matches a record missing EITHER', () => {
+      expect(matched({ $not: { stage: 'won', owner: 'u1' } })).toEqual(['2', '3', '4']);
+    });
+
+    it('the RLS shape: a CEL `!(stage == "won")` check keeps stage-less records', () => {
+      expect(matched({ $not: { stage: 'won' } })).toHaveLength(3);
+    });
+  });
+
+  describe('nesting', () => {
+    it('$not of a $or rejects a value-less record whose OTHER branch matches', () => {
+      // Record 3 has no stage but owner = 'u1', so the $or holds and the
+      // negation rejects it. This is the case that forced `driver-sql` to put
+      // its NULL guard on each leaf rather than beside the `NOT`.
+      expect(matched({ $not: { $or: [{ stage: 'won' }, { owner: 'u1' }] } })).toEqual(['2', '4']);
+    });
+
+    it('$not of a $and matches every record failing either conjunct', () => {
+      expect(matched({ $not: { $and: [{ stage: 'won' }, { owner: 'u1' }] } })).toEqual(['2', '3', '4']);
+    });
+
+    it('a double negation is the positive filter again', () => {
+      expect(matched({ $not: { $not: { stage: 'won' } } })).toEqual(['1']);
+      expect(matched({ $not: { $not: { stage: 'won' } } })).toEqual(matched({ stage: 'won' }));
+    });
+
+    it('$not ANDs with its sibling keys', () => {
+      expect(matched({ $not: { stage: 'won' }, owner: 'u1' })).toEqual(['3']);
+    });
+  });
+
+  describe('operator polarity — a negation is not a blanket "and also the empty ones"', () => {
+    it('$not of $ne still means "the field IS that value"', () => {
+      expect(matched({ $not: { stage: { $ne: 'won' } } })).toEqual(['1']);
+    });
+
+    it('$not of $nin still means "the field IS among them"', () => {
+      expect(matched({ $not: { stage: { $nin: ['won'] } } })).toEqual(['1']);
+    });
+
+    it('$not of $in matches the value-less records', () => {
+      expect(matched({ $not: { stage: { $in: ['won'] } } })).toEqual(['2', '3', '4']);
+    });
+
+    it('$not of an ordering comparison matches the value-less records', () => {
+      expect(matched({ $not: { amount: { $gt: 15 } } })).toEqual(['1', '3']);
+    });
+
+    it('$not of $contains matches the value-less records', () => {
+      expect(matched({ $not: { stage: { $contains: 'w' } } })).toEqual(['2', '3', '4']);
+    });
+
+    it('$not of $notContains does NOT match them — the mirror case', () => {
+      // A value-less field satisfies `$notContains` here, so the negation
+      // rejects it. `driver-sql` follows this answer; `driver-memory` answers
+      // the opposite for a null-valued field, which is filed on its own.
+      expect(matched({ $not: { stage: { $notContains: 'w' } } })).toEqual(['1']);
+    });
+
+    it('$not of a null predicate', () => {
+      expect(matched({ $not: { stage: { $null: true } } })).toEqual(['1', '2']);
+      expect(matched({ $not: { stage: { $null: false } } })).toEqual(['3', '4']);
+      expect(matched({ $not: { stage: null } })).toEqual(['1', '2']);
+      expect(matched({ $not: { stage: { $eq: null } } })).toEqual(['1', '2']);
+    });
+  });
+
+  describe('the boolean identities still hold here too (#5134)', () => {
+    it('$not: {} matches nothing — NOT TRUE ≡ FALSE', () => {
+      expect(matched({ $not: {} })).toEqual([]);
+    });
+
+    it('$not of an empty $or matches everything', () => {
+      expect(matched({ $not: { $or: [] } })).toEqual(ALL);
+    });
+  });
+
+  // ── Where this evaluator and `driver-memory` disagree — pinned, not fixed ──
+
+  describe('known disagreement with driver-memory (NOT ruled on by #5146)', () => {
+    it('$exists reads "the key is present", so a null value EXISTS', () => {
+      // `driver-memory` reads `$exists` as "has a value", so it answers the
+      // opposite for a present-but-null field. `driver-sql` cannot tell the two
+      // apart at all (a NULL column is a NULL column) and keeps its existing
+      // `IS NOT NULL` compilation.
+      expect(ids(NULLED, { $not: { stage: { $exists: true } } })).toEqual([]);
+      expect(ids(MISSING, { $not: { stage: { $exists: true } } })).toEqual(['3', '4']);
+    });
+  });
+});

packages/plugins/driver-memory/src/memory-matcher-not-null-safe.test.ts

@@ -0,0 +1,157 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * [#5146] `$not` and absent values — the answers this matcher gives, pinned.
+ *
+ * This backend needed no change for #5146: it already evaluates a negation in
+ * ordinary two-valued JS, so `{ $not: { stage: 'won' } }` matches a record whose
+ * `stage` is null or missing (`undefined !== 'won'`). `driver-sql` used to
+ * DISAGREE — SQL's `NOT (stage = 'won')` is UNKNOWN for a NULL column and a
+ * `WHERE` drops it — which meant one CEL `!expr` permission rule admitted a
+ * different set of rows depending on which driver ran it. #5146 ruled this
+ * backend's answer canonical (it is the 2:1 majority with `formula`) and
+ * `driver-sql` was rewritten to match.
+ *
+ * So these cases are a PIN, not a change: they are the reference the SQL
+ * compiler was aligned to, and `sql-driver-not-null-safe.test.ts` asserts the
+ * same ids over the same fixture. Changing an expectation here silently
+ * re-opens the divergence — the point of writing them down is that the next
+ * edit has to move both files, deliberately.
+ *
+ * Where this matcher and `formula`'s `matchesFilterCondition` disagree, the case
+ * says so and pins what each actually answers rather than pretending to a
+ * consensus; those disagreements are filed separately and are NOT what #5146
+ * ruled on.
+ *
+ * Home for these eventually: `FILTER_LOGIC_CASES` in `@objectstack/spec/data`,
+ * so all five backends are held to one table (spec lane, with #5239).
+ */
+
+import { describe, it, expect } from 'vitest';
+import { match } from './memory-matcher.js';
+
+/** Fields present but null — how a SQL NULL round-trips into a record. */
+const NULLED: Array<Record<string, unknown>> = [
+  { id: '1', stage: 'won', owner: 'u1', amount: 10 },
+  { id: '2', stage: 'lost', owner: 'u2', amount: 20 },
+  { id: '3', stage: null, owner: 'u1', amount: null },
+  { id: '4', stage: null, owner: null, amount: 40 },
+];
+
+/** The same rows with the null fields ABSENT — the shape a partial write leaves. */
+const MISSING: Array<Record<string, unknown>> = [
+  { id: '1', stage: 'won', owner: 'u1', amount: 10 },
+  { id: '2', stage: 'lost', owner: 'u2', amount: 20 },
+  { id: '3', owner: 'u1' },
+  { id: '4', amount: 40 },
+];
+
+const ALL = ['1', '2', '3', '4'];
+
+const ids = (rows: Array<Record<string, unknown>>, filter: unknown): string[] =>
+  rows.filter((r) => match(r, filter)).map((r) => String(r.id));
+
+/** Both readings of "no value" must give the same answer unless noted. */
+const matched = (filter: unknown): string[] => {
+  const nulled = ids(NULLED, filter);
+  expect(ids(MISSING, filter), 'a null field and an absent field must match alike').toEqual(nulled);
+  return nulled;
+};
+
+describe('[#5146] memory-matcher — $not over records with no value', () => {
+  describe('a record with no value does not satisfy the negated condition', () => {
+    it('$not on an implicit equality matches the value-less records', () => {
+      expect(matched({ $not: { stage: 'won' } })).toEqual(['2', '3', '4']);
+    });
+
+    it('$not over multiple keys matches a record missing EITHER', () => {
+      expect(matched({ $not: { stage: 'won', owner: 'u1' } })).toEqual(['2', '3', '4']);
+    });
+
+    it('the RLS shape: a CEL `!(stage == "won")` scope keeps stage-less records', () => {
+      expect(matched({ $not: { stage: 'won' } })).toHaveLength(3);
+    });
+  });
+
+  describe('nesting', () => {
+    it('$not of a $or rejects a value-less record whose OTHER branch matches', () => {
+      // Record 3 has no stage but owner = 'u1', so the $or holds and the
+      // negation must reject it. This is the case that forced `driver-sql` to
+      // compile its NULL guard onto each leaf instead of beside the `NOT`.
+      expect(matched({ $not: { $or: [{ stage: 'won' }, { owner: 'u1' }] } })).toEqual(['2', '4']);
+    });
+
+    it('$not of a $and matches every record failing either conjunct', () => {
+      expect(matched({ $not: { $and: [{ stage: 'won' }, { owner: 'u1' }] } })).toEqual(['2', '3', '4']);
+    });
+
+    it('a double negation is the positive filter again', () => {
+      expect(matched({ $not: { $not: { stage: 'won' } } })).toEqual(['1']);
+      expect(matched({ $not: { $not: { stage: 'won' } } })).toEqual(matched({ stage: 'won' }));
+    });
+
+    it('$not ANDs with its sibling keys', () => {
+      expect(matched({ $not: { stage: 'won' }, owner: 'u1' })).toEqual(['3']);
+    });
+  });
+
+  describe('operator polarity — a negation is not a blanket "and also the empty ones"', () => {
+    it('$not of $ne still means "the field IS that value"', () => {
+      expect(matched({ $not: { stage: { $ne: 'won' } } })).toEqual(['1']);
+    });
+
+    it('$not of $in matches the value-less records', () => {
+      expect(matched({ $not: { stage: { $in: ['won'] } } })).toEqual(['2', '3', '4']);
+    });
+
+    it('$not of an ordering comparison matches the value-less records', () => {
+      expect(matched({ $not: { amount: { $gt: 15 } } })).toEqual(['1', '3']);
+    });
+
+    it('$not of $contains matches the value-less records', () => {
+      expect(matched({ $not: { stage: { $contains: 'w' } } })).toEqual(['2', '3', '4']);
+    });
+
+    it('$not of a null predicate', () => {
+      expect(matched({ $not: { stage: { $null: true } } })).toEqual(['1', '2']);
+      expect(matched({ $not: { stage: { $null: false } } })).toEqual(['3', '4']);
+    });
+  });
+
+  describe('the boolean identities still hold here too (#5134)', () => {
+    it('$not: {} matches nothing — NOT TRUE ≡ FALSE', () => {
+      expect(matched({ $not: {} })).toEqual([]);
+    });
+
+    it('$not of an empty $or matches everything', () => {
+      expect(matched({ $not: { $or: [] } })).toEqual(ALL);
+    });
+  });
+
+  // ── Where this matcher and `formula` disagree — pinned, not harmonised ─────
+
+  describe('known disagreements with formula.matchesFilterCondition (NOT ruled on by #5146)', () => {
+    it('$nin: an ABSENT field is treated differently from a null one', () => {
+      // The early `value === undefined` guard in `checkCondition` exempts only
+      // `$exists` / `$ne` / `$null`, so an absent field fails `$nin` outright
+      // while a null field passes it. `formula` answers "not among" for both.
+      // Pinned as measured; the ruling belongs to the issue that records it.
+      expect(ids(NULLED, { $not: { stage: { $nin: ['won'] } } })).toEqual(['1']);
+      expect(ids(MISSING, { $not: { stage: { $nin: ['won'] } } })).toEqual(['1', '3', '4']);
+    });
+
+    it('$notContains: a value-less field does NOT satisfy it here', () => {
+      // `typeof null !== 'string'` → false, so the negation matches. `formula`
+      // answers true for the same record, and `driver-sql` follows `formula`.
+      expect(matched({ $not: { stage: { $notContains: 'w' } } })).toEqual(['1', '3', '4']);
+    });
+
+    it('$exists: a present-but-null field counts as NOT existing here', () => {
+      // `formula` reads `$exists` as "the key is present" (a null value exists);
+      // this matcher reads it as "has a value". Same answer for an absent field,
+      // different for a null one.
+      expect(ids(NULLED, { $not: { stage: { $exists: true } } })).toEqual(['3', '4']);
+      expect(ids(MISSING, { $not: { stage: { $exists: true } } })).toEqual(['3', '4']);
+    });
+  });
+});