Multi-node linclust

[bbuschkaemper]

Multi-node Linclust via MPI

Summary

Adds multi-node/multi-rank support to align2clust, letting the most expensive alignment/candidate-generation stage scale across multiple (slurm) nodes while producing identical clustering results to a single-node run. Reuses the
existing --mpi-runner/RUNNER mechanism already used by search/cluster.

Motivation

To make MMseqs2 linclust competitive vs. DIAMOND for large-scale clustering runs (>10B), it needs to support multi-node environments. Our test-run to cluster 8B proteins took 7 days on a single 128-core, 4TB ram machine, which makes MMseqs2 currently unfeasible for our planned 110B clustering.

Changelog

• Refactor Align2clust.cpp candidate generation and set-cover/greedy reduction into a reusable Align2ClustReducer class (order-independent, deterministic). No behavior change on its own.
• Added distributed candidate generation: MPI ranks own a round-robin, size-based static chunking of the work (Align2ClustChunking), each generating candidates independently into a local buffer with no shared state. Rank 0 collects all chunks and performs the single deterministic reduction + final DB write.
• Added chunk-level checkpointing (Align2ClustCheckpoint): completed chunks are persisted to disk (atomic write + .done marker) under <output>_chunks/<fingerprint>/. An interrupted run resumes by only regenerating not-yet-completed chunks. The fingerprint depends only on the clustering-relevant parameters (thresholds, mode, DB identity/size, chunk size) - not on rank or thread count - so a run can be interrupted on N nodes and resumed on M nodes.
• Hardened linclust.sh so only MPI-aware sub-stages (kmermatcher, align2clust, rescorediagonal/align for the legacy path) are wrapped in $RUNNER; the non-MPI-aware clusthash/clust stages always run as a single process.
• Added a fail-fast safety check in MMseqsMPI.cpp: a non-MPI build launched under a multi-task runner (srun -n>1, mpirun -np>1, etc.) now exits immediately with a clear error instead of silently running every task as an unaware duplicate that corrupts shared output.
• Docs: new README section with build requirements, an example sbatch script, and checkpoint/resume semantics.
• Tests: new TestAlign2ClustReducer unit tests covering tie-breaking, ordering, edge cases, and chunking/checkpoint determinism.

Constraints

• Requires a shared POSIX filesystem across nodes
• --include-align-files is not currently supported together with multi-rank/resumed distributed generation (rejected at startup with a clear error); works normally single-rank.
• Deploy linclust as a single top-level process with --mpi-runner/RUNNER; do not wrap the top-level mmseqs linclust invocation itself in mpirun/srun (nested MPI launches are rejected by OpenMPI).

Testing performed

• Refactor-only correctness: identical output vs. pre-refactor code (md5 comparison of clu/clu_rep/clu_aln/clu_rep_aln) for SET_COVER, GREEDY, and --include-align-files runs.
• MPI correctness: align2clust run directly under mpirun -np 1/2/4 against a single-node reference - identical results in all cases.
• Checkpoint/resume across topology change: started a 4-rank run with a small chunk size, killed it mid-run, resumed with 1 rank - confirmed only incomplete chunks were regenerated and the final output matched the reference exactly.
• Full end-to-end linclust workflow via --mpi-runner "mpirun -np 4" as a single top-level process - correct rank init, chunked processing, and byte-identical intermediate outputs.
• All new unit tests pass in both a non-MPI (-DHAVE_MPI unset) and MPI (-DHAVE_MPI=1) build.