Unbounded recursion in `minusWherePredicates` / `isWhereSubsetInternal` overflows the stack under rapid on-demand subset churn

[simonstaffgenius]

• I've validated the bug against the latest version of DB packages

Describe the bug
Rapidly fetching on-demand subsets (e.g. while scrolling a virtualized view that re-windows on every scroll) throws RangeError: Maximum call stack size exceeded. The stack points into query/predicate-utils.js — minusWherePredicates and isWhereSubsetInternal, which are mutually recursive and have no depth bound.

To Reproduce
Steps to reproduce the behavior:

• Trigger is rapid window churn against an on-demand collection whose subset where carries an IN/= ANY clause.
• A Node harness importing the real DeduplicatedLoadSubset reproduces the predicate accumulation, but the overflow needs the join-derived IN clause plus realistic async live-query timing to manifest.

Suggested fix

A depth guard on the mutually recursive walkers that bails conservatively (minus → null, isWhereSubset → false) so a too-deep case just costs a redundant re-request rather than crashing. Better still would be flattening the in→or handling so the subtraction doesn't recurse per disjunct.

Desktop:

• OS: macOS 26 Tahoe
• Browser chrome, safari, firefox
• Version 0.6.8

[kevin-dp]

Hi @simonstaffgenius,

Thanks for the report! I tried to reproduce this and couldn't trigger it through the public API. Specifically, I ran DeduplicatedLoadSubset through several churn scenarios (overlapping IN windows, sliding windows, thousands of eq calls on rotating fields, etc.) and the accumulated unlimitedWhere always stayed shallow — combineWherePredicates and unionSameFieldPredicates flatten same-operation nodes and merge IN/eq on the same field, so the predicate tree doesn't grow deeper with churn.

I can only reproduce the RangeError by hand-constructing a left-nested OR of depth ~2000 (or(or(or(eq, eq), eq), ...)), which doesn't look like a shape the dedupe pipeline can actually produce.

Do you have a minimal repro — ideally collection setup + the query/subset being loaded + the loop that triggers it? A stack trace from the real crash would also help a lot. Without that it's hard to tell whether there's a code path we're missing or whether the recursion bound is purely theoretical.

[kevin-dp]

Closing this, feel free to reopen if you have more information to reproduce this.

[chrishoermann]

Hi @kevin-dp — I hit this issue as well.
since I'm not capable of debugging such a thing myself i had fable do it. maybe you can get something out of the issue description it came up with:

You were right that flat churn shapes can't produce it; the missing ingredient is a nested and, and the pipeline does build one through public API.

Where the nested and comes from

1. A live query puts two .where() clauses on an alias (e.g. eq(role.buildingId, X) + isNull(role.validTo)), so that alias's subscription whereExpression is an and(...).
2. That alias is joined lazily (on-demand/progressive collection, initial state not loaded). Lazy join loading requests partial snapshots keyed by join keys: requestSnapshot({ where: inArray(joinRef, joinKeys) }) (query/compiler/joins.ts, also query/compiler/index.ts).
3. CollectionSubscription.requestSnapshot combines that with the subscription's own where: and(whereExpression, snapshotWhere) (collection/subscription.ts) → and( and(eq, isUndefined), in(partyId, [...]) ) — nested.
4. DeduplicatedLoadSubset tracks the first batch as unlimitedWhere; when the join-key set grows on a later tick, minusWherePredicates(newNested, oldNested) runs (query/subset-dedupe.ts L40) and never terminates.

Mechanism

Asymmetry in predicate-utils.ts: findCommonConditions → extractAllConditions flattens nested ands recursively, but removeConditions only filters top-level and args. When the common conditions sit inside a nested and node, they're found but can't be removed, so minusWherePredicates recurses with byte-identical inputs.

Repro (stock 0.6.14 and 0.6.16, deterministic)

import { DeduplicatedLoadSubset } from "@tanstack/db"

const ref = (...path) => ({ type: "ref", path })
const val = (v) => ({ type: "val", value: v })
const fn = (name, ...args) => ({ type: "func", name, args })

// The shape requestSnapshot builds for a lazy join load on an alias
// with a two-condition whereExpression:
const whereExpression = fn(
  "and",
  fn("eq", ref("role", "buildingId"), val("b-1")),
  fn("isUndefined", ref("role", "validTo"))
)
const lazyJoinLoadWhere = (joinKeys) =>
  fn("and", whereExpression, fn("in", ref("role", "partyId"), val(joinKeys)))

const dedupe = new DeduplicatedLoadSubset({ loadSubset: async () => {} })

await dedupe.loadSubset({ where: lazyJoinLoadWhere(["p-1"]) })
await dedupe.loadSubset({ where: lazyJoinLoadWhere(["p-1", "p-2"]) })
// ^ RangeError: Maximum call stack size exceeded

Production stack (Vite dev, @tanstack/db@0.6.14, electric collection, role ⋈ party live query while the collection was mid-initial-sync):

RangeError: Maximum call stack size exceeded
    at areExpressionsEqual (predicate-utils.js:536:10)
    at isWhereSubsetInternal (predicate-utils.js:33:7)
    at predicate-utils.js:38:16
    at Array.every (<anonymous>)
    at isWhereSubsetInternal (predicate-utils.js:37:26)
    ...
    at isWhereSubset (predicate-utils.js:12:10)
    at minusWherePredicates (predicate-utils.js:139:7)

Suggested fix

Make removeConditions symmetric with extractAllConditions — filter the flattened condition set:

function removeConditions(predicate: BasicExpression, conditionsToRemove: Array<BasicExpression>) {
  if (predicate.type === `func` && (predicate as Func).name === `and`) {
    const remainingArgs = extractAllConditions(predicate).filter(
      (arg) => !conditionsToRemove.some((cond) => areExpressionsEqual(arg, cond)),
    )
    // ... unchanged tail (0 → undefined, 1 → arg, else new and)
  }
  return predicate
}

and is associative/commutative so flattening preserves semantics, and it guarantees progress: every found common condition exists in the flattened set, so the recursion input strictly shrinks. We're running this as a one-line pnpm patch in production; the repro then returns the expected difference (and(eq(buildingId), isUndefined(validTo), eq(partyId, "p-2"))) and our collection test suites stay green.

If this makes sense to you could you reopen this?