Leiden 1.0.0

.github/workflows/ci.yml

@@ -36,6 +36,31 @@ jobs:
       - name: Test (release)
         run: cargo test --no-default-features --release
 
+      # `parallel` is on by default, so both halves need their own run
+      - name: Clippy (parallel)
+        run: cargo clippy --no-default-features --features parallel --all-targets -- -D warnings
+      - name: Test (parallel)
+        run: cargo test --no-default-features --features parallel --release
+
+      # and the two must agree exactly, or the feature is changing more than speed
+      - name: Feature parity
+        run: |
+          cargo run -q --release --no-default-features --example feature_parity > seq.txt
+          cargo run -q --release --no-default-features --features parallel --example feature_parity > par.txt
+          diff seq.txt par.txt
+
+  # the #[ignore]d tests: scale runs and the exhaustive optimality sweep, ~11s. Without this
+  # the HD-scale path and the Bell(12) check never run anywhere.
+  scale:
+    name: Scale and exhaustive
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: Swatinem/rust-cache@v2
+      - name: Ignored tests
+        run: cargo test --no-default-features --release -- --ignored --nocapture
+
   # separate so an hnsw breakage doesn't take the core red
   knn:
     name: With kNN feature

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "single-clustering"
-version = "0.7.0"
+version = "1.0.0"
 edition = "2024"
 authors = ["Ian F. Diks"]
 homepage = "https://singlerust.com"
@@ -10,29 +10,43 @@ readme = "README.md"
 description = "A high-performance network clustering library implementing community detection algorithms like Louvain and Leiden. Features efficient graph representation, abstract grouping systems, and K-NN graph creation from high-dimensional data. Provides parallel computation support via Rayon for handling large networks."
 
 [features]
-default = ["knn"]
-# k-nearest-neighbour graph construction from high-dimensional data. Split out so the
-# clustering core can be built and tested without the HNSW/kd-tree stack, which does not
-# compile on every target.
+default = ["knn", "parallel"]
+# kNN graph construction. Split out so the clustering core builds without the HNSW/kd-tree
+# stack. Spatial graphs need no feature of their own - the grid index has no dependencies.
 knn = ["dep:hnsw_rs", "dep:kiddo", "dep:ndarray"]
+# Every rayon call in the crate, behind one switch. Off gives a genuinely single-threaded
+# build - except that `knn` drags in hnsw_rs, which threads internally either way.
+parallel = ["dep:rayon"]
+# Shared test fixtures, so unit and integration tests use one copy. Not public API.
+testdata = []
 
 [dependencies]
 anyhow = "1.0.100"
 nalgebra-sparse = "0.10.0"
-num-traits = "0.2.19"
-rayon = "1.10.0"
+rayon = { version = "1.10.0", optional = true }
 single-utilities = "0.8.6"
 rand = "0.9.0"
 rand_chacha = { version = "0.9.0" }
 
 kiddo = { version = "5.2.2", optional = true }
-ndarray = { version = "0.16.1", features = ["rayon"], optional = true }
-hnsw_rs = { version = "0.3.2", features = ["simdeez_f"], optional = true }
+ndarray = { version = "0.16.1", optional = true }
+delaunator = "1.1.0"
 
 [dev-dependencies]
+single-clustering = { path = ".", default-features = false, features = ["testdata"] }
+# thread_invariance builds its own pools, so it needs rayon even when `parallel` is off
+rayon = "1.10.0"
 serde_json = "1.0"
 criterion = { version = "0.5", features = ["html_reports"] }
 
 [[bench]]
 name = "leiden"
 harness = false
+
+# simdeez_f is x86-only: anndists imports simdeez::{avx2,sse2} unguarded, so it cannot
+# compile on aarch64. Keep the SIMD where it works, drop it where it doesn't.
+[target.'cfg(target_arch = "x86_64")'.dependencies]
+hnsw_rs = { version = "0.3.2", features = ["simdeez_f"], optional = true }
+
+[target.'cfg(not(target_arch = "x86_64"))'.dependencies]
+hnsw_rs = { version = "0.3.2", optional = true }

README.md

@@ -1,7 +1,8 @@
 # single-clustering
 
-⚠️ **Development Status**: This library is under active development. APIs may change between
-versions; see the changelog for 0.7.0, which is a breaking rewrite of the Leiden core.
+⚠️ **Development Status**: Under active development, APIs may change between versions. 1.0
+drops the `NetworkGrouping` trait and trims `VectorGrouping` to what `aggregate` actually
+uses.
 
 A Rust library for community detection and graph clustering, focused on being correct,
 reproducible, and fast enough for single-cell-scale graphs.
@@ -20,6 +21,105 @@ reproducible, and fast enough for single-cell-scale graphs.
   you already hold
 - **k-NN graph construction** from high-dimensional data (optional `knn` feature)
 
+## Accuracy
+
+Correctness is checked against the implementations people actually use, not against itself.
+Fixtures generated from `leidenalg` and `banksy-py` are committed, so the test suite needs no
+Python.
+
+**Against `leidenalg`** — 160 committed fixtures spanning karate, LFR at several mixing
+parameters, SBM and a real k-NN graph, across resolutions and seeds, for both RB and CPM:
+
+| | modularity vs `leidenalg` |
+|---|---|
+| single seed | −0.09% mean (70 better, 70 equal, 20 worse) |
+| best of 2 seeds | +0.42% mean |
+
+Both libraries are stochastic heuristics over the same landscape — `leidenalg`'s own two
+fixture seeds differ by up to 3.5% on the harder instances — so a single-run comparison is
+mostly noise, and the sign of the mean is not meaningful. What the fixtures do pin exactly is
+the *definition*: on the same membership our quality function reproduces `leidenalg`'s value
+to floating point, which is what caught the 0.6.x factor-of-two resolution bug.
+
+**Against `igraph`**: +0.03% mean modularity.
+
+**Against brute force** — every set partition enumerated on graphs small enough to allow it,
+so this compares to the true optimum rather than to another heuristic:
+
+| | reaches the exact optimum |
+|---|---|
+| this crate | 93.5% |
+| `igraph` | 94.4–96.3% |
+
+**Against `banksy-py`**: the feature augmentation reproduces the reference elementwise to
+**1e-6** across five decay kernels, harmonics 0–2, and λ from 0 to 1. The residual is the
+reference's own precision — it stores azimuths as `float32`.
+
+### On real data
+
+PBMC3k (2,638 cells, 15-NN), clustering the *same* connectivity matrix `scanpy` did, so
+every difference is the algorithm rather than graph construction:
+
+| resolution | clusters (ours / scanpy) | modularity (ours / scanpy) |
+|---|---|---|
+| 0.1 | 3 / 3 | 0.9511 / 0.9511 |
+| 0.5 | 6 / 5 | 0.8063 / 0.8048 |
+| 1.0 | 8 / 8 | 0.6715 / 0.6696 |
+| 2.0 | 20 / 19 | 0.5304 / 0.5315 |
+| 4.0 | 50 / 49 | 0.4217 / 0.4240 |
+
+Cluster counts track `scanpy` within one at every resolution, so resolution means the same
+thing in both. Against the authors' cell-type annotations — the only measure here about
+biology rather than about matching another implementation — **ours scores ARI 0.8599 and
+`scanpy` 0.8609**, both peaking at resolution 0.75.
+
+## Performance
+
+The comparison is against `igraph`, which is C called from Python — so these are not
+interpreter-overhead numbers. Both sides cluster the identical graph and **only the
+clustering call is timed**; graph construction is excluded, because including it measures the
+pipeline rather than the algorithm.
+
+| nodes | edges | this crate | + `parallel` | `igraph` | speedup |
+|---|---|---|---|---|---|
+| 100k | 798k | 0.150 s | 0.117 s | 0.199 s | 1.33× / **1.70×** |
+| 500k | 4.0M | 1.113 s | 0.912 s | 1.469 s | 1.32× / **1.61×** |
+| 1M | 8.0M | 2.568 s | 2.202 s | 3.410 s | 1.33× / **1.55×** |
+
+Single-threaded it is a consistent **1.33×**; with `parallel` — on by default, and what
+multi-threads refinement — **1.55–1.70×**. Modularity is identical to four decimals throughout — the same
+answer computed faster, not a different tradeoff. Reproduce with `examples/vs_igraph.rs` and
+`tools/igraph_bench.py`.
+
+Both implementations are memory-latency bound on the same scatter-gather access pattern, so
+single-threaded tuning is close to exhausted — node reordering for cache locality was measured
+at only 1.13× (`examples/reorder.rs`). The remaining headroom is parallelism: `examples/scaling_ceiling.rs`
+measures the memory system sustaining 5.1× on 8 threads and 8× on 16, and 89% of a run is
+parallelisable, so the ceiling is real. Refinement is done; local moving is the larger half and
+is not yet parallel.
+
+End to end through `scanpy`, `sc.tl.leiden` takes 22–24 ms per resolution on PBMC3k against
+7–11 ms here, but that gap is **not** the algorithm — most of it is `scanpy` converting its
+sparse matrix into an igraph object inside the timed call. The table above is the honest
+algorithmic comparison.
+
+Single-threaded scaling on synthetic k-NN graphs:
+
+| nodes | edges | time |
+|---|---|---|
+| 20k | 170k | 18 ms |
+| 100k | 840k | 111 ms |
+| 3M | 25M | 6.8 s |
+| 8M | 57M | 20.1 s |
+
+Spatial graph construction, which `parallel` also covers:
+
+| | default | single-threaded |
+|---|---|---|
+| Visium HD lattice, 10.5M bins / 21.1M edges | 954 ms, 591 MB | — |
+| 500k cells, k-NN (k=6) | 283 ms | 731 ms |
+| 500k cells, radius | 73 ms | 233 ms |
+
 ## Usage
 
 ```rust
@@ -96,29 +196,58 @@ graphs.
 
 ```toml
 [dependencies]
-single-clustering = "0.7"
+single-clustering = "1.0"
 ```
 
 The k-NN graph construction is behind the default-on `knn` feature. To build just the
-clustering core — useful in CI, or on targets where the HNSW stack does not compile:
+clustering core:
 
 ```toml
-single-clustering = { version = "0.7", default-features = false }
+single-clustering = { version = "1.0", default-features = false }
 ```
 
+The default-on `parallel` feature is the single switch for every rayon call in the crate —
+refinement, and the spatial graph builders. Turning it off gives a single-threaded build:
+
+```toml
+single-clustering = { version = "1.0", default-features = false, features = ["knn"] }
+```
+
+The one caveat is that `knn` pulls in `hnsw_rs`, which threads internally either way.
+
+Results do not depend on it. `tests/thread_invariance.rs` asserts byte-identical labels across
+thread counts, objectives and seeds, and `examples/feature_parity.rs` fingerprints clustering
+and both spatial builders so CI can diff a sequential build against a parallel one. The feature
+changes speed and nothing else.
+
 ## Current status
 
 - ✅ **Leiden algorithm**: local moving, refinement, and aggregation
 - ✅ **CSR network representation**
 - ✅ **Quality functions**: RB configuration model and CPM
 - ✅ **Reproducibility**: deterministic under a fixed seed
+- ✅ **Spatial neighbour graphs**: Visium/HD lattice, radius, k-NN, Delaunay with adaptive
+  pruning, graph fusion, and per-sample construction for multi-slice experiments
+- ✅ **BANKSY feature augmentation**: neighbourhood mean and azimuthal gradient, so ordinary
+  Leiden finds spatial domains
 - 🚧 **Louvain**: available as `LeidenConfig { refine: false, .. }`; no separate entry point
 - 🚧 **Benchmarks**: `cargo run --release --example scaling`
-- ❌ **Parallel local moving**: planned, deliberately deferred until the sequential path is
-  measured
-- ❌ **DBSCAN / HDBSCAN / spatial-aware clustering**: planned
+- ✅ **Parallel refinement** (`parallel` feature, on by default): communities refine
+  independently, so there is no shared state to race on. 1.2-1.3x, byte-identical
+- ❌ **Parallel local moving**: the other 65% of a pass, and the harder half — it mutates
+  shared community aggregates, which is what the pre-0.7 parallel path got wrong
+- ❌ **DBSCAN / HDBSCAN**: planned
 - ❌ **Python bindings**: PyO3 integration (planned)
 
+### Not yet measured
+
+Spatial *domain detection* has no comparative result. The pieces exist and each is verified
+against its reference, but the end-to-end number — ARI against manual annotations on the
+DLPFC benchmark — has not been produced. Until it has, nothing here should be read as a claim
+about spatial domain accuracy. The target is pre-registered in
+[`docs/spatial-benchmark.md`](docs/spatial-benchmark.md), along with what the established
+methods score and which of their published numbers reproduce independently.
+
 ## Contributing
 
 This project is in active development. Contributions, bug reports, and feature requests are