Remove structurally redundant BF pairs

[jianhongshi123]

Motivation

I observed on TPC-H Q10 some Bloom filter pairs are built but provide no filtering benefit, so constructing them will add pure overhead. More generally, predicate transfer can generate redundant Bloom filter pairs, some of which can be proved redundant before execution stage.

Exact Redundancy Condition

For an integer build-side join column, a Bloom filter pair is redundant if the exact NDV equals the size of the build-side value range and the probe-side value range is fully contained in the build-side range.

In this case, every integer value in the build-side range must be present, so the Bloom filter cannot eliminate any probe-side tuples.

Challenges and Alternatives

The exact condition is difficult to apply directly for two reasons. First, predicate transfers are not independent. The min and max given by DuckDB is from base table, so the detector needs to determine whether build-side table is already filtered before applying the exact redundancy condition. Therefore, both local predicates and incoming probe filters need to be considered when applying it.

Second, DuckDB does not provide exact NDV. I initially used DuckDB's approximate NDV, but that was very inaccurate which caused a high false-negative rate and removed few proven redundant pairs.

I considered the following alternatives:

Approach	Exact detection	No extra runtime work	No tuning	No schema constraints	Before execution
Exact NDV*	✓	✓	✓	✓	✓*
Approximate NDV	✗	✓	✓	✓	✓
Approximate NDV + threshold	✗	✓	✗	✓	✓
Declared PK/FK constraints	Limited	✓	✓	✗	✓
Compute exact NDV during execution	✓	✗	✓	✓	✗

* DuckDB does not give exact NDV, so this option is not available in practice.

Approximate NDV is directly available in DuckDB, but in our TPC-H experiments it produced a high false-negative rate. Adding a threshold improves coverage but requires parameter tuning. Declared PK/FK constraints avoid statistical estimation, but they are not always available and cover only a subset of the redundant pairs captured by the Exact Redundancy Condition.

Current Design

The current design reuses statistics already maintained by DuckDB to detect redundant Bloom filter pairs before execution, without additional data scans, runtime profiling, or empirically tuned parameters.

Redundancy Detection

This PR first check whether the probe-side value range is fully contained in the build-side range. It then use two complementary checks that target different common join key structures:

• Row count check. If the build-side table row count equals the size of the build-side table range, then the check pass. This captures contiguous, unique key domains such as 1, 2, 3, ..., N.
• HLL dominance check. If the HyperLogLog value of build side is bigger or equal to that of probe side in every register, then the check pass. The structures that pass this check will be the probe-side domain is covered by the build-side domain and each key of probe side appears many time in build side. This captures FK->PK like structure, when the FK and PK have approximately the same domain, and there are many duplications of the same key in FK. Since DuckDB only samples 30% of the integer values into HLL, an FK value that appears many times is unlikely to be missed by the sample.

A Bloom filter Pair is considered redundant when the range check pass and either of the two structure checks pass.

Filter State Tracking

Redundancy Detection uses statistics from base table, so we need to also prove that the actual domain have not changed and still represented by the base statistics when reaching the Bloom filter. This PR processes Bloom filter pairs in transfer order and tracks the incoming filter state of each table.

Each table has one of these three states:

• NONE: no incoming probe filter that can not be proved non-selective
• ONE: one incoming probe filter that can not be proved non-selective
• MULTIPLE: more than one incoming probe filter that can not be proved non-selective

For every pair, if there is local predicate on build table which can not prove to be non-selective on either table, then the base statistics can be changed when reaching the Bloom filter. In this case, the filter will be directly marked not removable.

The following reasoning is built on the assumption that no filtering local predicate on build table:

The Bloom filter pairs are visited in transfer order, so table is visited as build table after all of its predecessors are visited in this pass. There can be disconnected components in our DAG, but it does not matter due to this. All tables start in the NONE state. When a pair cannot be removed, its probe-side table records the build table as an incoming filter source. A second incoming filter changes the state from ONE to MULTIPLE. If a pair is removed, it does not change the probe-side state.

One special case is the root. If the state of the root table is ONE when visited, and the beginning of the backward pass passes the redundancy check, this pair is redundant because the root has only been filtered by the same table to which the filter is now transferred back. Therefore, transferring the filter back cannot eliminate any additional rows. If the root is marked redundant due to this logic, the same reasoning can propagate along the backward pass.

A pair is eligible for redundancy detection when its build table state is NONE, or when it satisfies the turnaround case above. All pairs identified as redundant are removed together after the iteration.

Evaluation

To isolate the effect of redundant pair removal, I compared this optimization enabled and disabled both under TPC-H SF=100 Threads=8

Query	Removal OFF	Removal ON	Speedup
Q02	0.314s	0.298s	1.056×
Q08	1.063s	0.961s	1.106×
Q10	3.038s	2.280s	1.333×
Q17	1.146s	1.124s	1.020×
Q18	3.753s	3.483s	1.077×

There are 10 queries with Bloom filter and inserted and 5 of them are affected by this optimization. All 5 queries get performance gain in the current ablation setting, with geometric mean speedup 1.113x across affected queries. Most notably, Q10 was the biggest regression without this optimization and was 0.806x baseline DuckDB. With this optimization, the performance improves to 1.072x baseline DuckDB.

Across the complete TPC-H workload, Robust runtime decreases from 38.196s to 36.954s, a 1.034× speedup from redundant pair removal. The DuckDB baseline remains similar between the two runs.

Limitations

The current redundancy checks are approximate and may produce false predictions. However, a false-positive removal only affects performance, not correctness, since only inserted Bloom filter are removed and the original query is still executed.