expr: fix persist filter pushdown interpreter soundness bugs (PER-50)#37638
Merged
DAlperin merged 1 commit intoJul 15, 2026
Conversation
antiguru
approved these changes
Jul 14, 2026
Comment on lines
+1159
to
+1161
| if fallible && inputs_multivalued { | ||
| range.fallible = true; | ||
| } |
Member
There was a problem hiding this comment.
Might be good to leave a comment here. The line above seems to indicate that fallibility is already included in the intersect call, but that doesn't seem to be the case, and at least to me it's not clear why we need the if block.
| // excluded: `NaN` is ordered as the maximum but is a fixed point of | ||
| // most monotone functions, so evaluating the endpoints does not | ||
| // bound the interior. Such ranges fall through to the | ||
| // overapproximation below. See database-issues#9656. |
Member
There was a problem hiding this comment.
Not sure how much value we add by referencing the issue. Consider not mentioning it.
071728d to
ab4960d
Compare
The `ColumnSpecs` abstract interpreter decides whether persist filter
pushdown can skip fetching a part, based on the part's column statistics.
For a function marked `is_monotone` the interpreter samples only the
endpoints of an input range and unions the results. That is sound for the
function's output *value*, but the code trusted those two samples for more
than monotonicity guarantees, and a few domain operations had gaps. Each
produced a false negative: a part wrongly proven unable to match, so
pushdown discards it and the runtime audit panics ("persist filter
pushdown correctness violation"). Without the audit it would silently drop
matching or erroring rows.
Bugs fixed in `ColumnSpecs` / `ResultSpec` (src/expr/src/interpret.rs):
- Interior fallibility. A `could_error` function can error on an input in
the interior of a range that the endpoints don't hit, and the code read
fallibility off the endpoint evaluations. `round(numeric, scale)`
overflows depending on the input's decimal exponent (not its magnitude),
and `numeric::mz_timestamp` rejects fractional inputs, so both error
between two non-erroring bounds. A `could_error` function is now treated
as fallible over any multi-valued input range.
- NaN as an input bound. `NaN` sorts as the maximum of the numeric/float
order but is a fixed point of most monotone functions, so a range whose
bound is `NaN` breaks endpoint narrowing. The monotone shortcut is now
skipped when a bound is `NaN`.
- NaN as an output. Multiplication and division with an infinite operand
are not corner-sampleable: `inf * 0` and `inf / inf` evaluate to `NaN`,
and the value they take can be reached only from the interior (e.g.
`[-inf, +inf] * 0` maps both endpoints to `NaN` while the finite interior
maps to `0`). Those operations now fall back to the full value domain
when an operand may be infinite. This is driven by a new
`is_infinity_monotone` property on the `sqlfunc` macro (set `false` for
multiplication and division) rather than a hardcoded function list.
- `If` with a null condition. `MirScalarExpr::eval` takes the `els` branch
on both `False` and `Null`, but the interpreter mapped a null condition
to `fails()`, dropping `els` from the value channel. Null now takes `els`.
- `Values::intersect` of a range and a map spec. `Datum` order places `Map`
between `List` and `Numeric`, so a `Within` range straddling those tags
contains map values. Intersecting such a range with a `Nested` (map) spec
returned `Empty`, dropping values in both. It now returns the `Nested`
side as a sound over-approximation.
- The monotone-narrowing arm now treats the value, null, and error channels
orthogonally, so surfacing an operand's fallibility no longer collapses
the value range of an enclosing monotone operation to "anything".
Tests (src/expr/src/interpret.rs, src/storage-operators/src/persist_source.rs):
- `test_equivalence_ranges`: the equivalence proptest was only ever fed
single-value or unconstrained column specs, so it never exercised the
range narrowing that production stats drive. This variant feeds genuine
`value_between` ranges with an interior row value.
- `gen_expr_for_relation` now generates `If` nodes.
- `test_result_spec_lattice_laws`: union/intersect soundness of the
abstract domain in isolation.
- The generator's type and function coverage is widened (float, interval,
int32, float arithmetic, round, temporal casts) so the fixes above are
exercised beyond `Numeric`.
- `filter_pushdown_audit`: an end-to-end proptest that builds a part from
real rows, computes the real production statistics, and asserts
`filter_result` never discards a part whose MFP produces output on an
actual row. This is the only layer that catches stats-derivation bugs as
well as interpreter bugs.
- Deterministic regressions for the numeric-NaN and interior-error cases.
Behavior change: pushdown is now more conservative for predicates that run
a column through an erroring function (arithmetic, casts, interval math):
such a part is kept rather than pruned. Plain comparison filters are
unaffected. The notable loss is temporal-window pruning built from interval
or division arithmetic. Recovering that precisely is a follow-up: make the
fallibility annotation argument-aware so, for example, `col / 10` with a
constant nonzero divisor stays prunable.
ab4960d to
8896e24
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Fixes PER-50 (database-issues#9656):
persist filter pushdown correctness violationpanics.The
ColumnSpecsabstract interpreter decides whether persist filter pushdown can skip fetching a part, based on the part's column statistics. For a function markedis_monotoneit samples only the endpoints of an input range and unions the results. That is sound for the function's output value, but the code trusted those two samples for more than monotonicity guarantees, and a few domain operations had gaps. Each produced a false negative: a part wrongly proven unable to match, so pushdown discards it and the runtime audit panics. Without the audit it would silently drop matching or erroring rows.I found the first bug with a new range-based soundness proptest, then that plus an end-to-end stats→
filter_resultproptest surfaced the rest.Bugs fixed (
src/expr/src/interpret.rs)could_errorfunction can error on an interior input the endpoints miss (round(numeric, scale)overflows on an exponent-dependent condition,numeric::mz_timestamprejects fractional inputs), and fallibility was read off the endpoint evaluations. Acould_errorfunction is now fallible over any multi-valued input range.NaNsorts as the max of the numeric/float order but is a fixed point of monotone functions, breaking endpoint narrowing. The shortcut is skipped when a bound isNaN.Mul/Divwith an infinite operand are not corner-sampleable (inf * 0,inf / inf=NaN, and the reachable value lives in the interior, e.g.[-inf, +inf] * 0maps both endpoints toNaNbut the finite interior to0). They fall back to the full value domain when an operand may be infinite.Ifwith a null condition.evaltakes theelsbranch onFalseandNull, but the interpreter mapped a null condition tofails(), droppingels. Null now takesels.Values::intersectof a range and a map spec.Datumorder placesMapbetweenListandNumeric, so aWithinrange straddling those tags contains map values; intersecting it with aNestedspec returnedEmpty, dropping common values. It now returns theNestedside.Tests
test_equivalence_ranges: the equivalence proptest was only ever fed single-value or unconstrained specs, so it never exercised the range narrowing production stats drive. This variant feeds genuinevalue_betweenranges.gen_expr_for_relationnow generatesIfnodes; type/function coverage widened (float, interval, int32, float arithmetic, round, temporal casts).test_result_spec_lattice_laws: union/intersect soundness of the abstract domain.filter_pushdown_audit(persist_source.rs): end-to-end proptest that builds a part from real rows, computes the real production stats, and assertsfilter_resultnever discards a part whose MFP produces output on an actual row. This is the only layer that also catches stats-derivation bugs.Behavior change
Pushdown is now more conservative for predicates that run a column through an erroring function (arithmetic, casts, interval math): such a part is kept rather than pruned. Plain comparison filters are unaffected. The notable loss is temporal-window pruning built from interval or division arithmetic. Recovering that precisely is a follow-up: make the fallibility annotation argument-aware so, for example,
col / 10with a constant nonzero divisor stays prunable.