feat(eml-hnsw): v2 integrated pipeline — retention selector + SIMD rerank + PQ + progressive cascade (supersedes #353)

[ruvnet]

Credit

This work builds directly on two outstanding upstream contributions:

• @aepod (Mathew Beane) — original PR feat: EML-enhanced HNSW — 6 learned optimizations (10-30x distance, 2-5x search) #353 author. Designed and implemented all six learned models (EmlDistanceModel, ProgressiveDistance, AdaptiveEfModel, SearchPathPredictor, RebuildPredictor, PqDistanceCorrector), the gradient-free eml-core training library, and the 4-stage proof chain methodology. Without @aepod's Stage 4 hypothesis ("EML is the teacher, not the runtime — use plain cosine on selected dims") this v2 would not exist. The architectural pivot described in his own PR #353 comment thread is exactly what this branch ships as callable code.
• @shaal (Ofer Shaal) — author of issue EML Operator-Inspired Optimizations: Log Quantization, Unified Distance, EML Trees #351 and PR feat: EML operator-inspired optimizations for quantization, distance, and learned indexes #352. The SimSIMD-backed UnifiedDistanceParams kernel, the four-stage proof methodology (adopted verbatim here), and the honest SIFT1M+GloVe measurement discipline all originated in his work. Tier 1B of this branch is a direct port of his SIMD cosine approach into the reduced-dim rerank stage.

Both authors are credited as Co-Authored-By: on the merged commit, and every piece of measured evidence below is traceable to one or both of their PRs.

Supersedes #353

Rewrites the EML-HNSW contribution into a working integrated pipeline with measured SIFT1M numbers. The original PR shipped six standalone learned models but had no downstream consumer — the ruvector-eml-hnsw crate compiled but its code never reached any RuVector HNSW path. This branch closes that gap and folds in the winning results from a six-experiment swarm run on ruvultra (AMD Ryzen 9 9950X / 32T / 123 GB) against real SIFT1M.

What's in v2

Component	Source tier	Measured result on SIFT1M
EmlHnsw wrapper around hnsw_rs::Hnsw + search_with_rerank	fix/eml-hnsw-integration	baseline — unlocks every result below
SimSIMD rerank kernel (cosine_distance_simd), after @shaal's PR #352 kernel	Tier 1B	5.65× @ d=128, 6.22× @ d=384; recall unchanged
EmlDistanceModel::train_for_retention — greedy forward selection	Tier 1C	+10.5 pp recall@10 vs @aepod's Pearson (0.712 → 0.817), > 3σ
ProgressiveEmlHnsw [8, 32, 128] multi-level cascade, using @aepod's ProgressiveDistance	Tier 3A	0.984 recall@10 at 961 µs p50 (2× latency at matched recall; 5.9× build cost)
PqEmlHnsw 8×256 Product Quantizer paired with @aepod's PqDistanceCorrector	Tier 3B	64× memory reduction (512 B → 8 B/vec); rerank recall 0.9515 ≥ 0.80 floor

What's NOT in v2 (and why)

• EmlDistanceModel::fast_distance (EML tree per call): measured 2.35× slower than scalar baseline. Kept as reference impl; not on any query-time path. This matches @aepod's own Stage-1 finding on his test hardware.
• AdaptiveEfModel: 290 ns/query actual overhead vs 3 ns claimed — too expensive to amortize against the ef-search work it would save.
• Sliced Wasserstein rerank (Tier 2 experiment): 50.9× slower and 38.1 pp worse recall than cosine rerank on SIFT. Cleanly falsified for gradient-histogram datasets — documented as closed in ADR-151.
• PqDistanceCorrector is kept but held advisory-only: under training on SIFT1M it increased MSE (1.4e9 → 6.4e10) because feature normalization against a global max_pq_dist saturates on SIFT's O(10⁵) distance scale. Final rank is exact cosine so this does not hurt recall. Noted in ADR-151 as a design flaw with a proposed fix direction (per-vector exact normalization).

Test surface

92 tests pass on the merged branch:

• 85 unit tests across 10 modules (new: selected_distance, pq, pq_hnsw, progressive_hnsw, hnsw_integration; retained: all original ruvector-eml-hnsw tests from @aepod's PR feat: EML-enhanced HNSW — 6 learned optimizations (10-30x distance, 2-5x search) #353)
• 3 integration tests (recall_integration)
• 4 SIFT1M real-data tests (env-gated; skipped in CI without the dataset): sift1m_real, retention_vs_pearson, progressive_sift1m, sift1m_pq
• 1 micro-benchmark (benches/rerank_kernel.rs)

Reproducibility recipe (on any Linux box with rustc ≥ 1.80):

# One-time: fetch SIFT1M (Texmex, ~400MB)
mkdir -p bench_data && cd bench_data
curl -fLO ftp://ftp.irisa.fr/local/texmex/corpus/sift.tar.gz && tar xzf sift.tar.gz
cd ..

# Full SIFT1M test suite
B=$(pwd)/bench_data/sift
export RUVECTOR_EML_SIFT1M_BASE=$B/sift_base.fvecs \
       RUVECTOR_EML_SIFT1M_QUERY=$B/sift_query.fvecs \
       RUVECTOR_EML_SIFT1M_LEARN=$B/sift_learn.fvecs \
       RUVECTOR_EML_SIFT1M_GT=$B/sift_groundtruth.ivecs
cargo test --release -p ruvector-eml-hnsw -- --nocapture

Coupling with #352

@shaal's PR #352 (unified SIMD kernel + QuantizationConfig::Log) is strictly additive over this branch. Landing both captures the full effect: #352 accelerates the inner distance kernel, this branch adds the pre-filter stage that makes wide fetch_k viable. See issue #351 for the cross-PR measurements.

Surface area and compatibility

• DbOptions::default() behavior unchanged.
• HnswIndex::new(...) and all existing RuVector retrieval paths unchanged.
• EmlHnsw / ProgressiveEmlHnsw / PqEmlHnsw are explicitly constructed by callers opting into the approximate-then-exact pipeline.

References

• ADR-151 (docs/adr/ADR-151-eml-hnsw-selected-dims.md) — acceptance matrix, per-tier measured numbers, closed/open questions.
• PR feat: EML-enhanced HNSW — 6 learned optimizations (10-30x distance, 2-5x search) #353 (@aepod, original contribution this builds on) — feat: EML-enhanced HNSW — 6 learned optimizations (10-30x distance, 2-5x search) #353
• Issue EML Operator-Inspired Optimizations: Log Quantization, Unified Distance, EML Trees #351 (@shaal, proof methodology + proposal) — EML Operator-Inspired Optimizations: Log Quantization, Unified Distance, EML Trees #351
• PR feat: EML operator-inspired optimizations for quantization, distance, and learned indexes #352 (@shaal, SIMD unified kernel) — feat: EML operator-inspired optimizations for quantization, distance, and learned indexes #352

Closes #353 on merge. Cc @aepod @shaal for review — your work drove every measured result in this PR.

[ruvnet]

v3 update (branch feat/eml-hnsw-optimizations-v3)

Merge of four SOTA tiers on top of v2 (dac6f60e). v3 tip: 1fa28216.

Tier landing

tier	landed	measured on merged v3
SOTA-A PQ-native HNSW (+ OPQ)	yes	rerank@10 = 0.9510 @ 8 B/vec (64× payload reduction vs 512 B legacy). p50 rerank = 371.6 µs (2.56× faster than legacy 952 µs). OPQ gives no measurable gain on SIFT-native basis — kept as documented null result.
SOTA-B parallel rerank + 1M benchmarks + hnswlib baseline	yes (reframed)	parallel rerank = 1.10× serial (overhead-bound on SIFT128 × fetch_k=500). Plain hnsw_rs DistCosine @ 1M hits recall=0.9525 @ QPS=893 (ef=100); EmlHnsw selected_k=48 + fetch_k=500 plateaus at 0.8159 across all ef_search.
SOTA-C corrector local-scale fix (promoted) + beam selector (falsified)	yes (partial)	corrector held-out MSE −60.5% (1.397e9 → 5.52e8), non-advisory path wired into search_with_rerank (15×k pre-truncation). Beam selector gives +0.0065 recall over greedy at 4.39× training cost — inside SE ≈ 0.027, not promoted.
SOTA-D HnswIndex::new_with_selected_dims() in ruvector-core	yes	4 new integration tests passing (crates/ruvector-core/tests/hnsw_selected_dims.rs). Selected-dim prefilter now first-class in core without ruvector-eml-hnsw dependency.

Retention selector A/B (SIFT1M, selected_k=32)

selector	recall@10	train cost
pearson	0.7125	1.02 s
retention_greedy (v3 default)	0.8165	39.8 s
retention_beam (beam=4)	0.8230	174.7 s

Greedy wins: +10.4 pp over pearson. Beam gain is inside noise.

Honest reframe

Plain hnsw_rs beats EmlHnsw''s reduced-dim prefilter at 1M scale on both recall and QPS at matched HNSW config (m=16, ef_construction=200). The v3 speed story does not survive full-corpus scaling.

What v3 does deliver:

1. Memory win (PQ-native): 64× graph payload reduction (512 → 8 B/vec) with a 0.65 pp gain in rerank-recall over the legacy PQ path. The PQ-native HNSW now stores only u8 codes and computes asymmetric distances at query time via PqAsymmetricDistance.
2. Integration win (ruvector-core): HnswIndex::new_with_selected_dims() is first-class in core, with 4 new integration tests. Closes ADR-151 Q7.
3. Selector quality win: retention-greedy is a measurable +10.4 pp recall improvement over pearson and is the v3 default.
4. Corrector fix: the SOTA-C local-scale corrector fixes the global-max bug and is now non-advisory (pre-rerank truncation to rerank_k = 15·k). Closes ADR-151 Q6.

Clean falsifications (kept in repo)

• OPQ on SIFT-native corpus — test stays (sift1m_opq.rs) as null result
• Rayon parallel rerank — 1.10× only, kept for wider-embedding future; commit text marks it modest
• Beam selector — 0.65 pp over greedy inside SE, not promoted as default

Files

• ADR evidence: docs/adr/ADR-151-eml-hnsw-selected-dims.md §v3 SOTA Evidence
• PQ-native: crates/ruvector-eml-hnsw/src/pq_hnsw.rs, src/opq.rs
• Corrector fix: crates/ruvector-eml-hnsw/src/pq_corrector.rs
• Retention-greedy selector: crates/ruvector-eml-hnsw/src/cosine_decomp.rs::train_for_retention_beam
• ruvector-core API: crates/ruvector-core/src/index/hnsw_selected.rs, crates/ruvector-core/tests/hnsw_selected_dims.rs
• Merge commits: a842f0d5 (D) → 6a797c08 (A) → e13de438 (C) → 54483c45 (B) → 1fa28216 (ADR)

Readiness

v3 is ready for review. All 93 lib + 4 new core integration tests green on merged branch. Recommend reading ADR-151 §v3 SOTA Evidence first — it carries the honest-reframe framing this comment summarizes.