ivfpq: Push down row filters for 8-bit batch scans

[shyjsarah]

Summary

Optimize selective row-filtered IVF-PQ batch scanning by evaluating filters once per relevant inverted list and sharing the resulting matches across all queries in a batch.

This change focuses distance-computation pushdown on the standard 8-bit IVF-PQ scan paths while preserving existing 4-bit and FastScan scoring behavior.

Changes

• Materialize matching rows once per probed or loaded inverted list instead of evaluating the filter for every query.
• Evaluate in-memory filters only for lists actually probed by the current query batch.
• Represent sparse matches as positions and switch dense matches to a compact bitmap.
• Scan only matching rows in selective 8-bit row-major and transposed layouts.
• Stream dense row-major scans directly into the top-k heap without allocating a list-sized distance buffer.
• Preserve existing 4-bit and FastScan distance semantics.
• Keep public APIs unchanged.

Testing

• cargo fmt --all -- --check
• cargo clippy -p paimon-vindex-core --all-targets -- -D warnings
• cargo test -p paimon-vindex-core --locked
• Added coverage for:
  • filters evaluated once per loaded/probed row,
  • unprobed lists skipped,
  • sparse 8-bit row-major and transposed scans,
  • dense bitmap scans.

Notes

• Public API: unchanged.
• Batch-local distance-table reuse remains intentionally excluded and is handled separately in ivfpq: Reuse list tables in batch search #65.

[JingsongLi]

+1