diff --git a/docs/algorithms/README.md b/docs/algorithms/README.md index 5542af4..105a4be 100644 --- a/docs/algorithms/README.md +++ b/docs/algorithms/README.md @@ -17,6 +17,39 @@ Each page below focuses on a single module: the concept it implements, the data | Reference Matching | [reference-matching.md](reference-matching.md) | K-mer containment scoring with streaming batches | | Assembly Classification | [assembly-classification.md](assembly-classification.md) | Marker calling on FASTA assemblies with GFF annotation | +## Where the code lives + +Each document above describes one part of this tree, so the two read together. + +```text +pathotypr-core/src/ +├── main.rs # CLI entry point +├── lib.rs # Library root +├── defaults.rs # Default resource URLs and filenames +├── train.rs # Random Forest training + OOB + CV +├── predict.rs # Streaming batch prediction +├── classify/ # Assembly-based marker classification +│ ├── mod.rs # Orchestration + genome analysis +│ ├── markers.rs # Marker parsing + k-mer generation +│ ├── annotation.rs # GFF parsing + AA translation +│ └── masking.rs # FASTA masking at marker sites +├── classify_split_fastq.rs # FASTQ genotyping orchestration +├── split_kmer.rs # Diagnostic k-mer engine + Bloom filter +├── match/mod.rs # Reference matching: scoring + coarse-to-fine +├── sparse_tree.rs # Custom CART on sparse vectors +├── vectorizer.rs # Feature hashing (hashing trick) +├── model.rs # Model bundle + label encoder +├── lineage.rs # Hierarchical lineage classification +├── fasta_io.rs # FASTA reading (needletail) +├── paired_end.rs # Paired-end FASTQ detection +├── excel.rs # Streaming Excel export +├── errors.rs # Error types + cancellation +└── common.rs # Thread pool + shared utilities +``` + +The desktop app lives alongside it in `src-tauri/` (Rust backend) and +`frontend/` (HTML/CSS/JS); see [Desktop GUI](../gui.md) for that side. + ## See also - [train](../train.md) diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 1e92991..c690d79 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -115,6 +115,27 @@ Match is the most memory-intensive module: each reference batch loads ~4.4 Mb ge --- +## Feature comparison with other MTBC tools + +Where pathotypr sits next to the tools it is most often compared with. + +| | pathotypr | fastlin | TB-Profiler | Mykrobe | SNP-IT | KvarQ | +|---|---|---|---|---|---|---| +| Alignment-free (FASTQ) | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ | +| Assemblies (FASTA) | ✅ | ✅ | ❌ | ❌ | VCF only | ❌ | +| Custom markers | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | +| ML training | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ | +| DR prediction | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | +| Desktop GUI | ✅ | ❌ | Web | ✅ | ❌ | ✅ | +| Standalone binary | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | +| Organism-agnostic | ✅ | TB only | TB only | Limited | TB only | TB only | +| Speed (per sample) | ~1 s | <5 s | 3–10 min | ~3 min | 1–2 min | ~2 min | + +Only the pathotypr and fastlin timings below were measured here; the rest of the +table reflects each tool's documented capabilities and reported runtimes. + +--- + ## pathotypr vs fastlin — Real TB Data Head-to-head comparison using real *Mycobacterium tuberculosis* FASTQ samples from the European Nucleotide Archive. diff --git a/readme.md b/readme.md index dcedaa5..15eb007 100644 --- a/readme.md +++ b/readme.md @@ -13,7 +13,9 @@ **Lineage classification and marker-driven genotyping — from assemblies or raw reads.** -[Quick Start](#quick-start) · [Commands](#commands) · [GUI](#gui) · [Documentation](https://pathogenomics-lab.github.io/pathotypr/) · [Citation](#citation) +### 📖 [**Read the documentation**](https://pathogenomics-lab.github.io/pathotypr/) + +Tutorials, every command option, input formats, output columns and benchmarks. @@ -36,46 +38,20 @@ pathotypr is a Rust toolkit that classifies microbial genomes into lineages and

-**Five commands, one binary:** - -| Command | What it does | Input | -|---|---|---| -| **`train`** | Build a Random Forest classifier from labeled genomes | FASTA | -| **`predict`** | Assign lineages using a trained model | FASTA + model | -| **`classify`** | Call known SNP markers in assemblies | FASTA + markers | -| **`split-fastq`** | Alignment-free genotyping from reads | FASTQ + markers | -| **`match`** | Find the closest reference genome | FASTQ + references | - -**Key features:** -- 🦠 **Organism-agnostic** — bring your own markers for any pathogen -- ⚡ **Fast** — Rust + SIMD gzip + parallel k-mers (~1–2 s per sample) -- 🖥️ **Desktop GUI** — native app via Tauri, no server required -- 📊 **Excel + TSV** output with interactive visualizations in the GUI - -## Installation - -### Desktop GUI (pre-built) - -**[⬇ Download the latest release](https://github.com/PathoGenOmics-Lab/pathotypr/releases/latest)**, then pick the file for your platform: - -| Platform | File | Notes | -|---|---|---| -| 🍎 macOS (Apple Silicon) | `Pathotypr__aarch64.dmg` | M1 / M2 / M3 / M4 Macs | -| 🍎 macOS (Intel) | `Pathotypr__x64.dmg` | Pre-2020 Macs | -| 🐧 Linux (.deb) | `Pathotypr__amd64.deb` | Debian / Ubuntu | -| 🐧 Linux (.rpm) | `Pathotypr--1.x86_64.rpm` | Fedora / RHEL | -| 🐧 Linux (AppImage) | `Pathotypr__amd64.AppImage` | Any distro, no install needed | -| 🪟 Windows (installer) | `Pathotypr__x64-setup.exe` | Windows 10+ | -| 🪟 Windows (.msi) | `Pathotypr__x64_en-US.msi` | Windows 10+ (MSI) | +| Command | What it does | Input | Guide | +|---|---|---|---| +| **`train`** | Build a Random Forest classifier from labeled genomes | FASTA | [docs](https://pathogenomics-lab.github.io/pathotypr/train/) | +| **`predict`** | Assign lineages using a trained model | FASTA + model | [docs](https://pathogenomics-lab.github.io/pathotypr/predict/) | +| **`classify`** | Call known SNP markers in assemblies | FASTA + markers | [docs](https://pathogenomics-lab.github.io/pathotypr/classify/) | +| **`split-fastq`** | Alignment-free genotyping from reads | FASTQ + markers | [docs](https://pathogenomics-lab.github.io/pathotypr/split-fastq/) | +| **`match`** | Find the closest reference genome | FASTQ + references | [docs](https://pathogenomics-lab.github.io/pathotypr/match/) | > [!NOTE] -> **macOS users**: The app is not signed with an Apple Developer certificate. On first launch, right-click the app → **Open** → click **Open** in the dialog. See [Apple support](https://support.apple.com/en-us/HT202491) for details. -> -> **Windows users**: Windows SmartScreen may show a warning for unrecognized apps. Click **More info** → **Run anyway** to proceed. +> Nothing is hard-coded to one organism: the marker panel you supply defines what is typed. In practice pathotypr has only been validated on the *M. tuberculosis* complex, so treat other organisms as exploratory and check them against a truth set you trust. -Older versions are on the [releases page](https://github.com/PathoGenOmics-Lab/pathotypr/releases). +## Install -### CLI (Bioconda) +**Command line** ```bash conda create -n pathotypr -c bioconda pathotypr @@ -83,197 +59,55 @@ conda activate pathotypr pathotypr --help ``` -### CLI (from source) - -```bash -git clone https://github.com/PathoGenOmics-Lab/pathotypr.git -cd pathotypr -cargo build --release -p pathotypr-core --bin pathotypr -./target/release/pathotypr --help -``` - -### GUI (from source) - -See [docs/gui.md](https://pathogenomics-lab.github.io/pathotypr/gui/) for building the Tauri desktop app from source. - -## MTBC Marker Files & Pre-trained Model - -Ready-to-use marker panels and a pre-trained Random Forest model for *Mycobacterium tuberculosis* complex (MTBC) are available on Zenodo: +**Desktop app**: [download an installer](https://github.com/PathoGenOmics-Lab/pathotypr/releases/latest) for macOS, Linux or Windows. No compiler needed. -| File | Description | Download | -|---|---|---| -| `pathotypr_lineage_markers_v1.0.0.tsv` | 3,707 lineage SNPs (L1–L10, A1–A4) | [⬇ Download](https://zenodo.org/records/19210044/files/pathotypr_lineage_markers_v1.0.0.tsv?download=1) | -| `pathotypr_dr_markers_v1.0.0.tsv` | 102,213 DR mutations (WHO catalogue v2, 2023) | [⬇ Download](https://zenodo.org/records/19210044/files/pathotypr_dr_markers_v1.0.0.tsv?download=1) | -| `pathotypr_rf_model_v1.0.0.pathotypr` | Pre-trained RF model (k=31, 100 trees) | [⬇ Download](https://zenodo.org/records/19210044/files/pathotypr_rf_model_v1.0.0.pathotypr?download=1) | +Building from source, system dependencies and the first-launch notes for unsigned apps are in the [installation guide](https://pathogenomics-lab.github.io/pathotypr/installation/). -> **DOI:** [10.5281/zenodo.19210044](https://zenodo.org/records/19210044) - -## Quick Start +## Quick start ```bash -# Train a lineage model -pathotypr train -i labeled_genomes.fasta -o model.pathotypr.zst - -# Predict lineages -pathotypr predict -i query.fasta -m model.pathotypr.zst -o predictions.tsv - -# Classify markers in assemblies +# Genotype an assembly against a marker panel pathotypr classify -m markers.tsv -r reference.fasta -i sample.fasta -o results -# Genotype from FASTQ reads +# Genotype straight from reads pathotypr split-fastq -m markers.tsv -r reference.fasta \ - -i reads_R1.fastq.gz -i reads_R2.fastq.gz --paired -o genotype + -i reads_R1.fastq.gz -i reads_R2.fastq.gz -o genotype -# Find best reference match -pathotypr match -i reads_R1.fastq.gz reads_R2.fastq.gz \ - -r references.fasta -o match.tsv +# Train a model, then apply it +pathotypr train -i labeled_genomes.fasta -o model.pathotypr.zst +pathotypr predict -i query.fasta -m model.pathotypr.zst -o predictions.tsv ``` -Add `--excel` to any command to also generate `.xlsx` files. - -## Commands - -Each command has its own detailed documentation: +Add `--excel` to any command to also write `.xlsx`. The +[getting started tutorial](https://pathogenomics-lab.github.io/pathotypr/getting-started/) +walks through a full MTBC run, from install to reading the output. -| Command | Docs | Summary | -|---|---|---| -| `train` | [docs/train.md](https://pathogenomics-lab.github.io/pathotypr/train/) | Random Forest on k-mer feature-hashed vectors | -| `predict` | [docs/predict.md](https://pathogenomics-lab.github.io/pathotypr/predict/) | Streaming batch prediction with confidence scores | -| `classify` | [docs/classify.md](https://pathogenomics-lab.github.io/pathotypr/classify/) | Marker k-mer matching + GFF annotation + masked FASTA | -| `split-fastq` | [docs/split-fastq.md](https://pathogenomics-lab.github.io/pathotypr/split-fastq/) | Alignment-free genotyping with Bloom filter acceleration | -| `match` | [docs/match.md](https://pathogenomics-lab.github.io/pathotypr/match/) | K-mer containment scoring against reference databases | +## Ready-to-use MTBC data -Run `pathotypr --help` for all options. +Marker panels and a pre-trained model for the *M. tuberculosis* complex are published on Zenodo, DOI [10.5281/zenodo.19210044](https://doi.org/10.5281/zenodo.19210044): -### Algorithm Details - -For in-depth descriptions of the algorithms, data structures, and design decisions behind each module, see [docs/algorithms/](https://pathogenomics-lab.github.io/pathotypr/algorithms/): - -| Document | Topic | +| File | Contents | |---|---| -| [Feature Hashing](https://pathogenomics-lab.github.io/pathotypr/algorithms/feature-hashing/) | The hashing trick: k-mers → fixed-size sparse vectors | -| [Random Forest](https://pathogenomics-lab.github.io/pathotypr/algorithms/random-forest/) | Sparse CART trees with bootstrap aggregation | -| [Training Pipeline](https://pathogenomics-lab.github.io/pathotypr/algorithms/training/) | Vectorize → evaluate → train → OOB → export | -| [Prediction](https://pathogenomics-lab.github.io/pathotypr/algorithms/prediction/) | Streaming batch prediction with majority voting | -| [Marker Genotyping](https://pathogenomics-lab.github.io/pathotypr/algorithms/marker-genotyping/) | Diagnostic k-mers + Bloom filter for FASTQ scanning | -| [Reference Matching](https://pathogenomics-lab.github.io/pathotypr/algorithms/reference-matching/) | K-mer containment scoring with streaming batches | -| [Assembly Classification](https://pathogenomics-lab.github.io/pathotypr/algorithms/assembly-classification/) | Marker calling on FASTA with GFF annotation | - -## Input Formats - -### Training FASTA +| `pathotypr_lineage_markers_v1.0.0.tsv` | 3,707 lineage SNPs (L1–L10, A1–A4) | +| `pathotypr_dr_markers_v1.0.0.tsv` | 102,213 DR mutations (WHO catalogue v2, 2023) | +| `pathotypr_rf_model_v1.0.0.pathotypr` | Pre-trained Random Forest (k=31, 100 trees) | -The first token in each header is the class label: - -``` ->L4 sample_0001 -ACTG... ->L2 sample_0002 -ACTG... -``` +Download links and usage are in the [installation guide](https://pathogenomics-lab.github.io/pathotypr/installation/#mtbc-marker-files-pre-trained-model). -### Marker TSV +## Documentation -Tab-separated: `position REF ALT level1 [level2 ...]` +Everything lives at **[pathogenomics-lab.github.io/pathotypr](https://pathogenomics-lab.github.io/pathotypr/)**: -``` -#pos ref alt level1 level2 -761155 C T L4 L4.9 -2155168 G A L2 L2.2 -``` - -Lineage columns are read until the first empty cell. Columns after the empty cell are treated as annotations. - -> See [docs/input-formats.md](https://pathogenomics-lab.github.io/pathotypr/input-formats/) for full format specifications. - -## GUI - -The desktop app includes all five workflows with drag-and-drop file selection, interactive result tables, and real-time progress indicators. - -```bash -# Development -cargo tauri dev - -# Production build -cargo tauri build -``` - -> See [docs/gui.md](https://pathogenomics-lab.github.io/pathotypr/gui/) for system dependencies and build instructions. - -## Performance - -

- - - - pathotypr performance benchmarks - -

- -Benchmarked on real *M. tuberculosis* genomes (~4.4 Mb, k=21), Mac mini M4, 4 threads: - -| Module | Time | Peak RAM | Key property | -|---|---|---|---| -| **train** (10 genomes) | 0.6 s | 302 MB | Scales with dataset size | -| **train** (50 genomes) | 55 s | 1.4 GB | | -| **predict** (5 genomes) | 0.25 s | 198 MB | ~50 ms/genome, constant | -| **classify** (5 genomes) | 0.10 s | 92 MB | ~20 ms/genome | -| **split-fastq** (65× PE) | 10.5 s | 26 MB | **Constant memory** | -| **match** (20 refs) | 78 s | 4.6 GB | Streaming batches | - -- **SIMD-accelerated** gzip decompression (zlib-ng) -- **Streaming** I/O — split-fastq holds 26 MB regardless of input size -- **85,000+ genomes/second** prediction throughput (synthetic benchmarks) - -> See [docs/benchmarks.md](https://pathogenomics-lab.github.io/pathotypr/benchmarks/) for detailed charts, scaling plots, and pathotypr vs fastlin comparison. - -## Comparison - -| | pathotypr | fastlin | TB-Profiler | Mykrobe | SNP-IT | KvarQ | -|---|---|---|---|---|---|---| -| Alignment-free (FASTQ) | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ | -| Assemblies (FASTA) | ✅ | ✅ | ❌ | ❌ | VCF only | ❌ | -| Custom markers | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | -| ML training | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ | -| DR prediction | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | -| Desktop GUI | ✅ | ❌ | Web | ✅ | ❌ | ✅ | -| Standalone binary | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | -| Organism-agnostic | ✅ | TB only | TB only | Limited | TB only | TB only | -| Speed (per sample) | ~1 s | <5 s | 3–10 min | ~3 min | 1–2 min | ~2 min | - -## Project Structure - -``` -pathotypr/ -├── pathotypr-core/ # Core library + CLI -│ └── src/ -│ ├── main.rs # CLI entry point -│ ├── train.rs # Random Forest training + OOB + CV -│ ├── predict.rs # Streaming batch prediction -│ ├── classify/ # Assembly-based marker classification -│ │ ├── mod.rs # Orchestration + genome analysis -│ │ ├── markers.rs # Marker parsing + k-mer generation -│ │ ├── annotation.rs # GFF parsing + AA translation -│ │ └── masking.rs # FASTA masking at marker sites -│ ├── classify_split_fastq.rs # FASTQ genotyping orchestration -│ ├── split_kmer.rs # Diagnostic k-mer engine + Bloom filter -│ ├── match/ # Reference matching -│ │ ├── mod.rs # Scoring + coarse-to-fine matching -│ │ └── index.rs # Compact inverted index + cache -│ ├── sparse_tree.rs # Custom CART on sparse vectors -│ ├── vectorizer.rs # Feature hashing (hashing trick) -│ ├── model.rs # Model bundle + label encoder -│ ├── lineage.rs # Hierarchical lineage classification -│ ├── fasta_io.rs # FASTA reading (needletail) -│ ├── paired_end.rs # Paired-end FASTQ detection -│ ├── excel.rs # Streaming Excel export -│ ├── errors.rs # Error types + cancellation -│ └── common.rs # Thread pool + shared utilities -├── src-tauri/ # Desktop app backend (Tauri) -├── frontend/ # GUI (HTML/CSS/JS) -├── docs/ # Detailed documentation -└── logo/ # Branding assets -``` +| | | +|---|---| +| [Getting started](https://pathogenomics-lab.github.io/pathotypr/getting-started/) | End-to-end MTBC tutorial | +| [Input formats](https://pathogenomics-lab.github.io/pathotypr/input-formats/) | What every file must look like, per command | +| [Marker format](https://pathogenomics-lab.github.io/pathotypr/marker_format/) | Curating your own panel | +| [Output files](https://pathogenomics-lab.github.io/pathotypr/output-files/) | Every column of every file | +| [Desktop GUI](https://pathogenomics-lab.github.io/pathotypr/gui/) | The app, and building it | +| [Algorithms](https://pathogenomics-lab.github.io/pathotypr/algorithms/) | How each module works | +| [Benchmarks](https://pathogenomics-lab.github.io/pathotypr/benchmarks/) | Speed, memory and tool comparison | +| [FAQ](https://pathogenomics-lab.github.io/pathotypr/faq/) | Common problems | ## Citation @@ -281,25 +115,14 @@ If you use pathotypr, please cite: > Ruiz-Rodriguez P, Coscollá M. **Pathotypr: harmonised MTBC lineage assignment and resistance-associated variant detection for genomic surveillance.** *bioRxiv* (2026). doi: [10.64898/2026.03.24.714002](https://doi.org/10.64898/2026.03.24.714002) -```bibtex -@article{ruiz-rodriguez_pathotypr_2026, - title = {Pathotypr: harmonised {MTBC} lineage assignment and resistance-associated variant detection for genomic surveillance}, - author = {Ruiz-Rodriguez, Paula and Coscoll{\'a}, Mireia}, - journal = {bioRxiv}, - year = {2026}, - doi = {10.64898/2026.03.24.714002}, - url = {https://www.biorxiv.org/content/10.64898/2026.03.24.714002v1} -} -``` - -> Software & markers DOI: [10.5281/zenodo.19210044](https://doi.org/10.5281/zenodo.19210044) +BibTeX, RIS and APA entries, plus the software DOI, are on the +[citation page](https://pathogenomics-lab.github.io/pathotypr/citation/). ## License [GNU Affero General Public License v3.0](LICENSE) --- -

✨ [Contributors](https://github.com/PathoGenOmics-Lab/pathotypr/graphs/contributors) @@ -348,4 +171,3 @@ This project follows the [all-contributors](https://github.com/all-contributors/ ----