Parametric Sectorized Bloom filter policy

[sleeepyjack]

Reworks cuco::bloom_filter around a parametric Sectorized Bloom Filter (SBF), following "Optimizing Bloom Filters for Modern GPU Architectures" (arXiv:2512.15595).

Benchmark results

[sleeepyjack]

/ok to test 8b1e995

[sleeepyjack]

/ok to test eae3049

[PointKernel]

The actual code changes are not that big, but the use of work stealing definitely caught my attention.

@sleeepyjack, could you please review all files touched by this PR and make sure the copyright years are updated where necessary?

[PointKernel]

Suggested change

	* @brief A GPU-accelerated Bloom filter.
	* @brief A GPU-accelerated Bloom Filter.

[PointKernel]

It would be helpful to add a brief section describing the underlying algorithm, along with a reference to the original paper. I noticed the paper is referenced in the policy document, but it doesn't appear to be mentioned here.

[sleeepyjack]

Regarding the tuning knobs, I (or better Codex) did an ablation study:

Bloom filter tuning sweep summary

I ran a tuning sweep on sleeepyjack/bloom-filter-release / PR #808 head
c571c33f34631fcfa05b0807188a5c12bbfda617 in the default cuCollections
devcontainer (cuda13.1-gcc14, CTK 13.1) on:

• GPU: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition
• L2 cache: 128 MiB
• Characterization sizes: {1, 16, 32, 64, 128, 256, 512} MiB
• Metrics collected: throughput, time, DRAM throughput, L2 hit rate, FPR
• FPR stayed stable at ~0.00131 for contains variants, so the differences below are performance-only.

Overall recommendation

The only clear default change suggested by this run is:

• For vertical contains, prefer the custom kernel path over the CUB path:
  • use_cub_kernels = false

Everything else should stay at the current default unless we want a very small size-specific horizontal-contains optimization.

Per-knob findings

use_warp_cooperative_add_kernel

Recommendation: keep enabled.

This is strongly beneficial for add.

Screen-stage main effect:

• on vs off: +77.1%

When disabled, add throughput dropped substantially at the screen sizes. This knob should remain enabled.

use_cuda_atomic_ref

Recommendation: keep disabled.

Enabling cuda::atomic_ref hurt add.

Characterization, compared to the matching non-cuda_atomic_ref path:

• Average: -3.44%
• <= 128 MiB: about -3.6%
• 256–512 MiB: about -0.16%

So the current non-cuda::atomic_ref path is better.

use_invoke_one

Recommendation: neutral; current default is fine.

For the useful add/horizontal-contains configurations, toggling invoke_one was essentially noise-level.

Observed deltas:

• Add with warp-coop on and cuda_atomic_ref=0: -0.02%
• Horizontal contains with warp-coop on: about -0.02%

It does not look like a meaningful tuning lever for this workload.

use_cub_kernels

Recommendation: disable for vertical contains.

This was the largest actionable win.

Vertical contains, cub_kernels=false vs baseline cub_kernels=true:

• Average: +14.5%
• <= 128 MiB: +15–16%
• 256–512 MiB: +1.2–1.5%

Average throughput:

• Baseline vertical contains: 61.29 G elem/s
• cub_kernels=false, early_exit=false: 70.19 G elem/s

This suggests the custom vertical contains kernel should be preferred over the CUB transform path on this setup.

use_early_exit

Recommendation: keep disabled.

Early exit did not help in this sweep.

With cub_kernels=false:

• early_exit=true was slightly worse overall: about -0.1%

With cub_kernels=true:

• Difference was noise-level, about +0.02%

No reason to enable it by default from these data.

use_warp_cooperative_contains_kernel

Recommendation: keep enabled.

This is critical for horizontal contains.

Horizontal contains with warp-coop disabled:

• Average: -42.2%
• <= 128 MiB: about -47.2%
• 256–512 MiB: about -4.2%

Disabling this increased L2 hit rate somewhat but throughput collapsed, so it should remain enabled.

use_work_stealing_add_kernel

Recommendation: keep disabled.

Work stealing hurt add throughput.

work_stealing_add=true vs baseline add:

• Average: -1.76%
• <= 128 MiB: -1.82%
• 256–512 MiB: -0.30%

Per-size deltas for {1,16,32,64,128,256,512} MiB:

• -1.97%, -2.74%, -2.84%, -0.19%, -0.92%, -0.35%, -0.24%

No evidence this helps add on the tested workload.

use_work_stealing_contains_kernel

Recommendation: generally keep disabled; possibly consider only for small horizontal contains.

Vertical contains with cub_kernels=false:

• work_stealing=true, early_exit=false: -0.69% average
• work_stealing=true, early_exit=true: -0.44% average

So work stealing does not help vertical contains.

Horizontal contains with warp-coop enabled:

• Average with invoke_one=true: +0.78%
• <= 128 MiB: +0.90%
• 256–512 MiB: -0.13%

Per-size deltas for {1,16,32,64,128,256,512} MiB:

• +2.02%, +1.15%, +0.63%, +0.54%, -0.28%, -0.16%, -0.11%

So work stealing has a small niche benefit for small/cache-resident horizontal contains, but it turns slightly negative for larger filters. I would not enable it globally.

Suggested default policy from this run

For the tested Blackwell + CTK 13.1 setup:

use_invoke_one                       = true;  // neutral / current default OK
use_early_exit                       = false;
use_cub_kernels                      = false; // for vertical contains; biggest win
use_warp_cooperative_add_kernel      = true;
use_warp_cooperative_contains_kernel = true;
use_work_stealing_add_kernel         = false;
use_work_stealing_contains_kernel    = false; // unless specializing small horizontal contains
use_cuda_atomic_ref                  = false;

Code paths that could be removed if we want to simplify

If the goal is to keep the implementation lean rather than preserve all experimental tuning paths, the sweep suggests the following removal candidates.

Good removal candidates

• use_cuda_atomic_ref alternate path:

  • The cuda::atomic_ref path was consistently slower for add.
  • Suggested simplification: remove the use_cuda_atomic_ref knob and keep the current faster non-cuda::atomic_ref atomic OR path only.

• use_early_exit path:

  • Early exit did not improve vertical contains and was slightly negative with the recommended cub_kernels=false path.
  • Suggested simplification: remove the use_early_exit knob and simplify the compare recursion to always evaluate the full pattern.

• use_work_stealing_add_kernel path:

  • Work stealing was slower for add at every characterized size.
  • Suggested simplification: remove add_work_stealing_n_impl, add_work_stealing_n, and the host-side launch branch guarded by use_work_stealing_add_kernel.

• Vertical-contains use of use_work_stealing_contains_kernel:

  • Work stealing was slower with the recommended vertical contains path (cub_kernels=false).
  • Suggested simplification: do not route vertical contains through the work-stealing kernel.

• use_cub_kernels contains path:

  • The CUB DeviceTransform path for vertical contains was substantially slower than the custom contains kernel.
  • Suggested simplification: remove the use_cub_kernels knob for contains and always use the custom contains kernel.
  • Note: this does not imply removing CUB from the bloom filter implementation entirely, because other operations still use CUB utilities.

Probably keep, or remove only with care

• use_warp_cooperative_add_kernel:

  • Keep this path. It is strongly beneficial.
  • If simplifying, remove the non-warp-cooperative add path instead, not the warp-cooperative path.

• use_warp_cooperative_contains_kernel:

  • Keep this path. It is critical for horizontal contains.
  • If simplifying, remove the non-warp-cooperative horizontal contains path instead, not the warp-cooperative path.

• use_invoke_one:

  • Performance impact was effectively neutral.
  • I would keep this as a compatibility/implementation knob unless we want to simplify aggressively.
  • If removing it, keep the current default behavior (invoke_one enabled when available) and remove the fallback branch only where toolkit support guarantees it.

• Horizontal use_work_stealing_contains_kernel:

  • This has a small benefit for small/cache-resident horizontal contains (+0.5–2.0% up to 64 MiB, +0.9% through 128 MiB), but turns slightly negative for larger filters.
  • I would not enable it by default. Removal is reasonable if we do not want a size-specialized path; otherwise keep it only behind a size/architecture heuristic.

Caveat

These results are from a uniform benchmark workload on one Blackwell GPU. The clearest and most robust conclusions are:

1. keep warp-cooperative add/contains enabled,
2. keep cuda_atomic_ref disabled,
3. disable CUB for vertical contains,
4. keep work stealing disabled by default.

[sleeepyjack]

tl;dr here is a summary and my suggestion on what we should do with each tuning knob/ code path:

• use_invoke_one: No performance benefit in using it so we can safely remove any use of invoke_one as well as the tuning knob.
• use_early_exit: Although it didn't show any perf impact in the tested scenarios, it might become relevant for when the match rate of the filter is very low. Kevin had a usecase that was showing significant improvements from this knob. We should even expose this as a tparam of the policy instead of hiding it inside the implementation.
• use_cub_kernels: I'd suggest we keep this knob for now, set it to false for contains and true for add.
• use_warp_cooperative*: Keep this code path. Remove the tuning knobs and make warp cooperative kernels the default.
• use_work_stealing*: Not much benefit with the tested configuration but in the paper there were some scenarios on B200 where it was slightly better. I'd say it's not worth the complexity keeping it in our codebase right now. Instead, I would rather wait for a neat CCCL abstraction and then reintroduce it.
• use_atomic_ref: atomicOr is consistently faster than cuda::atomic_ref::fetch_or so I suggest to remove this knob and use atomicOr as default. This is fine since we control the type of the underlying atomic aka word_type.

What do you think?

[PointKernel]

tl;dr here is a summary and my suggestion on what we should do with each tuning knob/ code path:

Too late. I've already gone through the whole lengthy AI-generated report. 😉

What do you think?

All looks valid to me. Several points:

• use_cub_kernels: this is surprising. Worth bringing this up to CCCL?
• use_atomic_ref: this aligns with my observations duing the new hash table design as well, should report it to CCCL
• use_work_stealing: Bloom filter operations tend to have fairly uniform costs compared to hash table operations, so I don't expect work stealing to provide much benefit here. Still, it was great to explore.

[sleeepyjack]

Benchmark default policy upstream/dev vs PR

Setup

• GPU: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, L2 = 128 MiB
• Devcontainer: cuda13.1-gcc14, CTK 13.1
• Axes: FilterSizeMB={1,16,32,64,128,256,512}, NumInputs=1e9
• NVBench args for both runs: --min-samples 10 --min-time 0.05 --max-noise 5 --timeout 120 --throttle-threshold 0

Add throughput

Geomean speedup: +12.85%. Arithmetic mean: +13.57%.

Filter size	Dev throughput	PR throughput	Speedup	Dev time	PR time	Dev noise	PR noise
1 MiB	48.78 G elem/s	63.68 G elem/s	+30.54%	20.499 ms	15.703 ms	3.99%	2.65%
16 MiB	47.73 G elem/s	60.53 G elem/s	+26.81%	20.951 ms	16.522 ms	4.01%	4.40%
32 MiB	47.73 G elem/s	59.99 G elem/s	+25.69%	20.952 ms	16.669 ms	3.81%	4.30%
64 MiB	36.43 G elem/s	40.64 G elem/s	+11.55%	27.453 ms	24.609 ms	1.03%	0.17%
128 MiB	8.82 G elem/s	8.81 G elem/s	-0.05%	113.404 ms	113.460 ms	0.06%	0.03%
256 MiB	5.36 G elem/s	5.38 G elem/s	+0.38%	186.582 ms	185.872 ms	0.02%	0.03%
512 MiB	4.77 G elem/s	4.77 G elem/s	+0.09%	209.773 ms	209.591 ms	0.01%	0.01%

Contains throughput and FPR

Geomean throughput speedup: +17.77%. Arithmetic mean: +18.94%.

Filter size	Dev throughput	PR throughput	Speedup	Dev FPR	PR FPR	FPR reduction	Dev time	PR time	Dev noise	PR noise
1 MiB	78.39 G elem/s	105.98 G elem/s	+35.20%	0.013022	0.001308	10.0x	12.757 ms	9.436 ms	4.13%	2.19%
16 MiB	73.37 G elem/s	97.73 G elem/s	+33.19%	0.029271	0.001313	22.3x	13.629 ms	10.232 ms	4.93%	3.53%
32 MiB	73.23 G elem/s	97.05 G elem/s	+32.54%	0.029418	0.001315	22.4x	13.656 ms	10.304 ms	4.98%	3.51%
64 MiB	73.01 G elem/s	96.07 G elem/s	+31.59%	0.046045	0.001315	35.0x	13.697 ms	10.409 ms	5.00%	3.42%
128 MiB	38.73 G elem/s	38.74 G elem/s	+0.03%	0.060881	0.001316	46.3x	25.819 ms	25.812 ms	0.05%	0.06%
256 MiB	18.81 G elem/s	18.82 G elem/s	+0.04%	0.067084	0.001316	51.0x	53.167 ms	53.149 ms	0.01%	0.01%
512 MiB	16.41 G elem/s	16.41 G elem/s	+0.02%	0.067727	0.001315	51.5x	60.940 ms	60.930 ms	0.01%	0.01%

Note: The huge FPR gap comes from a problem with the old algorithm. Block indexing and fingerprint generation reuse the same hash bits. As the filter gets larger, more bits are consumed by the block index, leaving fewer effective bits for the fingerprint, which inflates FPR. The new algorithm splits those bits, enabling orders-of-magnitude smaller filters at comparable FPR — likely small enough to fit in L2 and run faster.

[sleeepyjack]

/ok to test cc6675b

[PointKernel]

One small but not blocking. Let's ship it.

[PointKernel]

Is unsigned long long required here because it's the type supported by atomicOr?