`.build_range_filter()`'s `Reduce(call("|", ...))` has no depth cap -> node stack overflow on a real (non-random) key subset with a few thousand contiguous ranges

[baj12]

Summary

.filter_tblconn() (R/DuckDBTable-class.R) picks between two strategies for
filtering a DuckDBTable down to a key subset: a temp-table SEMI/ANTI JOIN
(.apply_key_filter(), used above .KEY_FILTER_JOIN_THRESHOLD keys), or an
inline BETWEEN/== filter built by .build_range_filter() when the subset
compresses into relatively few contiguous ranges:

ranges <- .find_contiguous_ranges(set)
if (length(ranges) <= max(10L, k / 10L)) {
    range_filter <- .build_range_filter(conn, i, ranges)
}

.build_range_filter() combines the per-range expressions with an unbounded
Reduce():

exprs <- lapply(ranges, function(r) { ... call("between", col_sym, left, right) ... })
if (length(exprs) == 1L) exprs[[1L]]
else Reduce(function(a, b) call("|", a, b), exprs)

The length(ranges) <= max(10L, k / 10L) guard is a ratio check ("are
ranges efficient relative to an IN list of this size"), not an absolute cap.
For a subset with a few thousand contiguous ranges, this guard is satisfied
happily (ranges well under k/10), but Reduce() still builds a call tree
thousands of nodes deep. When that expression is later walked (dbplyr's SQL
translation, or deparse()), it overflows R's C-level node/evaluation stack
with a raw crash:

<nodeStackOverflowError: node stack overflow>

— not a normal, catchable R-level error in every context. Under
callr::r_bg() specifically, the child process dies before it can write a
result or error file, so the caller only ever sees callr's own generic
wrapper message ("could not start R, exited with non-zero status, has
crashed or was killed"), with no indication this came from inside
DuckDBDataFrame at all — which made this very hard to track down from the
caller's side.

Reproduction

Against a real ~294k-row/18k-column sparse count store (via the DuckDBArray
package, which calls into this filtering path through
DuckDBArraySeed/DuckDBTable), filtering down to a ~139k-cell subset with
real biological structure (not a random sample — see why that matters below)
reliably triggers the crash. Root-caused precisely, not just observed:

set <- <the real 139483-length integer key subset>
ranges <- .find_contiguous_ranges(set)
length(ranges)          # 6504
length(set) / 10        # 13948.3  -- 6504 <= 13948.3, so .build_range_filter() IS used
# .build_range_filter(conn, "cell_key_col", ranges) then builds a
# 6504-deep nested `|`/`between` call tree via Reduce(), and evaluating/
# translating it overflows the node stack.

Why a size threshold alone won't reproduce it: a purely random sample
of the same absolute size (e.g. sample.int(293885, 50000)) does NOT trigger
this, because a random scatter produces close to as many ranges as elements
(ratio ≈ 1, far above the k/10 guard), so it correctly falls through to the
safe join path instead. It's specifically a moderately-clustered real-world
key subset — few enough ranges to pass the ratio guard, but still far too
many for an unbounded Reduce()-built call tree to survive evaluation —
that hits this. A minimal repro that doesn't require the full package stack:

# Minimal illustration of the underlying Reduce() problem, independent of
# DuckDBDataFrame/DBI/dplyr:
ranges <- lapply(seq_len(6504L), function(i) c(2L * i, 2L * i))  # 6504 disjoint singleton "ranges"
exprs <- lapply(ranges, function(r) call("==", quote(x), r[1L]))
tree <- Reduce(function(a, b) call("|", a, b), exprs)
deparse(tree)   # <-- overflows / crashes well before finishing, on a plain R node-stack limit

Suggested fix

Cap .build_range_filter()'s applicability by an absolute range count as
well as the existing ratio, e.g. reusing (or a fraction of)
.KEY_FILTER_JOIN_THRESHOLD (currently 256L, used a few lines above for
exactly this "too big, switch to a join" reasoning):

if (length(ranges) <= max(10L, k / 10L) && length(ranges) <= .KEY_FILTER_JOIN_THRESHOLD) {
    range_filter <- .build_range_filter(conn, i, ranges)
}

and/or build the OR-tree without deep R-level recursion (e.g. combine ranges
into a single vectorized BETWEEN-per-row SQL expression via sql()/a
values-table join, rather than an Reduce()-nested call() tree), so range
count is no longer bounded by R's own stack depth at all.

Environment

• DuckDBDataFrame — installed version 0.99.18 (current as of 2026-08-23)
• Reached via DuckDBArray (DuckDBArraySeed's [ -> .subset_DuckDBArraySeed()
  -> table[Nindex, ] -> this package's [.DuckDBTable -> .filter_tblconn())
• R 4.6, macOS