diff --git a/.gitignore b/.gitignore index b83a8aa..fb34530 100644 --- a/.gitignore +++ b/.gitignore @@ -33,8 +33,10 @@ speedup.md scripts/run_*.sh scripts/_probe*.py -# internal docs/specs/analysis: kept local, NOT published (public repo ships only READMEs) +# Root-level research/spec/analysis notes stay local. The public README, +# CLAUDE agent guide, and the tracked docs/ developer guide are source. /*.md !/README.md +!/CLAUDE.md # local archive of superseded working docs (not published) /_archive/ diff --git a/Dockerfile b/Dockerfile index c4f22d4..d6f2a37 100644 --- a/Dockerfile +++ b/Dockerfile @@ -8,8 +8,9 @@ # --out-dir /data/out \ # --config /opt/mumdia/config.dia.json # -# The baked /opt/mumdia/config.dia.json wires the sidecars to the in-image conda -# envs and selects the Extended feature set + DIA apex settings. +# The baked /opt/mumdia/config.dia.json wires the FASTA workflow to the in-image +# conda envs. /opt/mumdia/config.diann-lib.json selects imported-library +# fine-tuning plus the torch rescorer. # ---------- Stage 1: build the Rust binary ---------- FROM rust:1.96-bookworm AS build @@ -36,10 +37,11 @@ RUN micromamba create -y -n rescore -f /tmp/env/docker-rescore.yml \ && micromamba clean -a -y \ && rm -rf /tmp/env -# Engine binary, sidecar workers, and the baked DIA config. +# Engine binary, sidecar workers, and the baked FASTA/library DIA configs. COPY --from=build /build/release/mumdia /usr/local/bin/mumdia COPY scripts /opt/mumdia/scripts COPY docker/config.dia.json /opt/mumdia/config.dia.json +COPY docker/config.diann-lib.json /opt/mumdia/config.diann-lib.json # mokapot logistic-regression is the recommended default rescorer. ENV MUMDIA_RESCORE_MODEL=logreg diff --git a/README.md b/README.md index 41e57f8..47cdaba 100644 --- a/README.md +++ b/README.md @@ -22,15 +22,35 @@ and proteins with target-decoy FDR control. models (DeepLC retention time, MS2PIP fragment intensities, mokapot rescoring, and an entrapment-based rescorer). -Each stage is an independent subcommand that reads path-addressable inputs and -writes Parquet plus a per-artifact `report.json`, so the pipeline is inspectable -and resumable at any step. +Each stage is an independent subcommand with path-addressable inputs and Parquet +outputs. Primary artifacts carry adjacent `report.json` provenance. Standalone +stage outputs can be reused manually; the `mumdia run` orchestrator itself does +not cache or resume and always recomputes its named outputs. + +## Documentation + +`docs/` is the developer guide: a per-subsystem reference grounded in the current +code (crates, each pipeline stage and its artifacts, the config and data model, +the sidecars, and the build and deploy machinery); start at +[`docs/README.md`](docs/README.md). For a practical local on-ramp with +copy-pasteable end-to-end runs, see +[`docs/19_getting_started.md`](docs/19_getting_started.md). +For the distinction between identification sensitivity, FDR validity, and +quantification accuracy, see +[`docs/20_sensitivity_and_quantification_playbook.md`](docs/20_sensitivity_and_quantification_playbook.md). ## Pipeline -``` -convert -> digest -> peptidoforms -> predict-frag -> search-seed -> -rt-im-train -> extract -> features -> compete -> rescore +```text +FASTA -> digest -> peptidoforms -> predict-frag --+ + +-> search-seed +imported spectral library ------------------------+ | +mzML -> convert ----------------------------------+ v + optional RT fine-tune + | + v + rt-im-train -> extract -> features + -> compete -> rescore -> quant -> report ``` Conversion retains all MS2 peaks by default. The seed search independently probes @@ -41,13 +61,15 @@ therefore also affects extraction, features, and quantification. `mumdia run` orchestrates the whole chain on one file and writes a `manifest.json`; `mumdia inspect ` prints schema, head, and row count -for any Parquet output. +for any Parquet output. Use a fresh output directory for every `run`: rerunning +in place overwrites named outputs and can leave stale optional sidecars after a +failed or differently configured run. ## Run with Docker (bundles all sidecars) The published image contains the engine plus the Python sidecars (mokapot, -MS2PIP, DeepLC), so the full high-sensitivity recipe runs with nothing to install -but Docker: +PyTorch, MS2PIP, DeepLC), so the portable FASTA workflow runs with nothing to +install but Docker: ``` docker pull ghcr.io/compomics/mumdia:latest @@ -64,13 +86,18 @@ and `proteins.tsv`) appear under `results/`. On Windows PowerShell, use `-v "${PWD}:/data"` (not `$PWD`, which PowerShell parses as a drive reference). The baked `/opt/mumdia/config.dia.json` selects the Extended feature set and the DIA apex settings, and wires DeepLC, MS2PIP, and mokapot (logistic regression) to the -in-image conda environments. To run the native, dependency-free models instead, -drop `--config` and add `--profile dia`. +in-image conda environments. The image also includes +`/opt/mumdia/config.diann-lib.json` for an imported library, per-run DeepLC +fine-tuning, and the higher-sensitivity `nn_torch` rescorer. Both configs use +strict rescoring, so a requested sidecar cannot silently become a native run. To +run the native, dependency-free models instead, drop `--config` and add +`--profile dia`. ## Build -Requires Rust >= 1.85 (the dependencies use edition 2024; `rustup update` if -older). All dependencies are pure Rust, so no C toolchain is needed. +Requires Rust >= 1.85 (the workspace `Cargo.toml` pins `rust-version = 1.85`; +`rustup update` if older). All dependencies are pure Rust, so no C toolchain is +needed. ``` cd rust/mumdia @@ -89,9 +116,9 @@ One command from a FASTA and a DIA mzML, using the validated DIA preset: ``` mumdia run \ - --fasta proteome.fasta \ - --mzml sample.mzML \ - --out results \ + --fasta proteome.fasta \ + --mzml sample.mzML \ + --out-dir results \ --profile dia ``` @@ -103,6 +130,11 @@ and quantities), alongside the Parquet artifacts and a `manifest.json`; use (`mumdia report --scored … --out-dir …`), can also be run standalone on prior outputs. +This quickstart uses uncapped converted MS2 spectra. On the validated Orbitrap +AIF benchmark, an explicit `--top-peaks-ms2 300` was slightly better; that cap +is acquisition-specific, not a universal DIA default. It is separate from +`search_seed.top_n_peaks = 300`, which limits only the calibration search. + ## Optional Python sidecars The native predictors and rescorer run with zero external dependencies. For @@ -110,19 +142,31 @@ higher sensitivity, MuMDIA can call Python sidecars over a simple file contract (input Parquet in, output Parquet out). The mokapot rescorer, for example, needs only a small environment (`mokapot`, `scikit-learn`, `numpy`, `pyarrow`, `pandas`); DeepLC and MS2PIP need their own environments. Sidecar selection and -the Python interpreter path are set in the configuration. The Docker image above -bundles all three so no manual environment setup is needed; the environment +the Python interpreter path are set in the configuration. Production and +benchmark configs should set `rescore.strict = true` and verify the actual +classifier in `psms_scored.parquet.report.json`. The Docker image above +bundles the required environments so no manual setup is needed; the environment specifications are under `env/` (`mumdia-rescore.yml`, `docker-rescore.yml`, `docker-deeplc.yml`). +The `scripts/` directory holds ten Python programs. Seven are engine-invoked +sidecar workers, called by the relevant stage over that file contract: MS2PIP +(`ms2pip_worker.py`), DeepLC (`deeplc_worker.py`), the DeepLC fine-tune +(`deeplc_finetune.py`), mokapot (`mokapot_worker.py`), the native-torch rescorer +(`nn_rescore_worker.py`), the entrapment rescorer (`entrapment_worker.py`), and +match-between-runs (`mbr_worker.py`). The other three are helpers for the DIA-NN +library recipe below and are run by hand: `import_diann_lib.py`, +`make_reverse_decoys.py`, and `make_shift_decoys.py`. + ## Using a DIA-NN spectral library (highest sensitivity) By default MuMDIA builds its library from a FASTA digest and predicts fragment intensities with the native model or MS2PIP. For the highest sensitivity, and to reproduce the benchmark numbers, you can supply a spectral library predicted by DIA-NN and have MuMDIA consume it directly. In this **library-input mode**, `run` -skips the digest, MS2PIP, and DeepLC steps and uses DIA-NN's fragment intensities -and retention times. +skips digest, peptidoform expansion, and initial fragment/RT prediction and uses +DIA-NN's fragment intensities and retention times. An optional per-run DeepLC +fine-tune can still rewrite the imported iRT values after seed search. MuMDIA does not include or download DIA-NN. You run DIA-NN yourself, under your own license: the DIA-NN "Academia" build is free for non-profit academic research @@ -171,10 +215,13 @@ both; in Docker use `/opt/conda/envs/rescore/bin/python`). --lib-fragments lib_fragments.parquet \ --mzml sample.mzML \ --out-dir results \ - --profile dia + --profile dia \ + --top-peaks-ms2 300 ``` - Everything downstream (search-seed, RT calibration, extraction, features, + The explicit 300 cap reproduces the validated AIF setting; omit or retune it + for another acquisition scheme. Everything downstream (search-seed, RT + calibration, extraction, features, competition, rescoring, quant, report) is unchanged. Because the DIA-NN library supplies both fragment intensities and retention times, no fragment or RT prediction sidecar is required in this mode. @@ -191,11 +238,15 @@ in the config and point at a DeepLC 4.0 multitask environment: } ``` -Pass it with `--config`. `run` then fine-tunes on the confident seed PSMs and -re-predicts iRT for the whole library between search-seed and RT calibration. A -ready-made config for the Docker image is `docker/config.diann-lib.json` (it -targets the image's bundled `deeplc` environment). The fine-tune uses no fixed -random seed, so identification counts vary slightly between runs. +Pass it with `--config` and supply the original imported precursor table, not a +previous run's already-fine-tuned table. `run` then fine-tunes on the confident +seed PSMs and re-predicts iRT for the whole library between search-seed and RT +calibration. A ready-made config for the Docker image is +`docker/config.diann-lib.json` (it targets the bundled `deeplc` environment and +`nn_torch` rescorer). The DeepLC fine-tune is not guaranteed deterministic, so +identification counts can vary slightly between runs. The NN rescorer seeds +NumPy and PyTorch, but numerical kernels are likewise not guaranteed +bit-for-bit reproducible. In Docker, mount the library files and point `--lib-precursors` / `--lib-fragments` at the mounted paths; steps 2-3 (and the fine-tune) run in the @@ -203,19 +254,31 @@ image's bundled environments. ## FDR -MuMDIA controls the false discovery rate with target-decoy competition (reverse -or fragment-shift decoys), the standard, community-accepted approach, and reports -target-decoy q-values at PSM, peptide, and protein-group level. An optional -entrapment (foreign-proteome spike-in) rescorer is available as a -decoy-independent cross-check for experiments that want one, but it is not -required. +MuMDIA estimates false discovery rates with paired target-decoy competition +(reverse or scramble decoys for a native digest, or the paired decoys already +present in an imported library) and reports q-values at PSM, precursor, peptide, +and protein-group levels. Those estimates depend on a valid, +exchangeable decoy population; the engine rejects malformed or decoy-free +libraries. Entrapment (a foreign-proteome spike-in) is available as an empirical +cross-check and should be part of validation before a new sensitivity setting is +promoted across datasets. ## Benchmark -On the ProteomeXchange E. coli AIF file `LFQ_Orbitrap_AIF_Ecoli_01`, with a -DIA-NN-predicted library and per-run fine-tuned retention time, MuMDIA reports on -the order of 9,000 to 10,000 peptides at 1% FDR (mokapot), at roughly 97 to 98% -sequence concordance with DIA-NN. +All counts below are historical single-run validation targets on the +ProteomeXchange E. coli AIF file `LFQ_Orbitrap_AIF_Ecoli_01`, with converted MS2 +spectra capped at 300 peaks. The report count is a precursor-shaped +`(peptidoform, charge)` row selected by stripped-peptide q-value, not a +precursor-q-controlled count. Each number states the rescorer actually used. + +- **Native FASTA digest (zero dependencies):** about 1,213 confident report rows, using + the built-in models and the native rescorer (`native_tda`). Conservative and + high-precision. +- **Imported DIA-NN library with per-run DeepLC fine-tune:** about 9,300 to 9,500 + confident report rows with the mokapot rescorer (`mokapot`), and about 10,300 with the + native PyTorch rescorer (`nn_torch`), which is nonlinear and outperforms the + linear mokapot model on the same feature set. Roughly 97 to 98% sequence + concordance with DIA-NN. ## License diff --git a/config.local-diann-lib.json b/config.local-diann-lib.json new file mode 100644 index 0000000..72a6f94 --- /dev/null +++ b/config.local-diann-lib.json @@ -0,0 +1,24 @@ +{ + "features": { "set": "extended" }, + "extract": { + "min_frag_corr": 0.2, + "apex_count_window": 5, + "apex_rt_prior_s": 120.0 + }, + "rt_im_train": { + "finetune_deeplc": true, + "rt_window_multiplier": 1.5 + }, + "predict_frag": { + "deeplc_python": "C:/Users/robbi/anaconda3/envs/deeplc_mt/python.exe", + "sidecar_script_dir": "c:/Users/robbi/OneDrive - UGent/MuMDIA_NG/scripts" + }, + "rescore": { + "classifier": "nn_torch", + "python": "C:/Users/robbi/anaconda3/envs/py312_mumdia/python.exe", + "strict": true + }, + "quant": { + "q_filter": "run_psm_q" + } +} diff --git a/docker/config.dia.json b/docker/config.dia.json index 654ad1a..799819e 100644 --- a/docker/config.dia.json +++ b/docker/config.dia.json @@ -10,6 +10,7 @@ }, "rescore": { "classifier": "mokapot", - "python": "/opt/conda/envs/rescore/bin/python" + "python": "/opt/conda/envs/rescore/bin/python", + "strict": true } } diff --git a/docker/config.diann-lib.json b/docker/config.diann-lib.json index 3ad3195..aa0a64a 100644 --- a/docker/config.diann-lib.json +++ b/docker/config.diann-lib.json @@ -7,7 +7,8 @@ "sidecar_script_dir": "/opt/mumdia/scripts" }, "rescore": { - "classifier": "mokapot", - "python": "/opt/conda/envs/rescore/bin/python" + "classifier": "nn_torch", + "python": "/opt/conda/envs/deeplc/bin/python", + "strict": true } } diff --git a/docs/01_overview_and_dataflow.md b/docs/01_overview_and_dataflow.md new file mode 100644 index 0000000..e55be9e --- /dev/null +++ b/docs/01_overview_and_dataflow.md @@ -0,0 +1,622 @@ +# Overview and end-to-end dataflow + +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +MuMDIA is a clean-room Rust reimplementation of a data-independent-acquisition +(DIA) proteomics search engine. It takes an mzML run plus a spectral library +source and produces peptide and protein-group identifications at target-decoy +FDR, with optional quantification. This document describes the pipeline as a +graph: every stage, the artifacts each stage reads and writes, the two library +sources (FASTA digest versus imported DIA-NN library) and how the orchestrator +branches between them, the single-run `run` orchestration and its +`manifest.json`, and the current best-performing workflow. + +The design principle is that every computational stage is an independent command +over path-addressable inputs and outputs on disk (the interstage contract, +docs/18 B1). No stage shares in-memory state with another. `run` threads those +file paths, invokes the optional DeepLC fine-tune between stages, writes the +human-readable report, and records provenance. Most stage commands can therefore +be run standalone on prior outputs, on a different configuration, or on +hand-crafted minimal files. + +`run` recomputes every stage on every invocation. There is no artifact caching, +no skip-if-exists, and no resume: `run` calls each stage unconditionally and +overwrites its outputs even when identical artifacts already exist in `--out-dir` +(`run.rs:85-509`; no existence check precedes any stage call, and +`std::fs::create_dir_all` at `run.rs:90` does not clear or skip). To reuse a +prior artifact, invoke the downstream stage command standalone on it instead of +rerunning `run`. Use a fresh output directory for each orchestrated run. Reusing +one can leave a stale optional sidecar from an earlier configuration, and a +failed rerun can leave an old manifest beside partially replaced outputs. + +The validated findings and the interstage, determinism, and sidecar contracts +cited throughout this document are consolidated, self-contained, in +`docs/18_findings_and_decisions.md` (sections A1-A7 for findings, B1-B3 for +contracts); the citations below point there. plan.md holds the deeper +algorithmic spec but is local-only and gitignored. + +## Files + +| Path | Role | +|---|---| +| `rust/mumdia/crates/mumdia/src/main.rs` | CLI: one `clap` subcommand per stage (`Cmd` enum, `main.rs:20`); each arm loads config, hashes it, and calls the stage `run`. | +| `rust/mumdia/crates/mumdia/src/stages/run.rs` | `run` orchestrator: preflight, library-source branch, per-run chain, manifest assembly (`run.rs:85`). | +| `rust/mumdia/crates/mumdia/src/stages/mod.rs` | Re-exports every stage module. | +| `rust/mumdia/crates/mumdia/src/stages/*.rs` | The stage implementations (`convert`, `digest`, `peptidoforms`, `predict_frag`, `search_seed`, `rt_im_train`, `extract`, `features`, `compete`, `rescore`, `quant`, `report`, `align`, `audit`). | +| `rust/mumdia/crates/mumdia/src/lib.rs` | Crate lib root (bin+lib split, `lib.rs:5-16`): re-exports the pipeline modules so integration tests can drive stages directly. Public modules: `stages`, `matchers` (fragment matchers, doc 06), `index` (`Library` + inverted index, doc 06), `predict` (predictor traits + native fallbacks, doc 06), `rescoring` (`percolator_lite` + native NN, doc 11), `calibrate` (LOESS/linear/percentile, doc 08), `peaks` (peak enumeration, doc 09), `spectra` (in-memory spectrum model + loaders), `sidecar` (Python worker dispatch, doc 13), `stats` (pearson/cosine/spectral_angle kernel), `fdr` (target-decoy + entrapment q, doc 11). | +| `rust/mumdia/crates/mumdia-core/src/schema.rs` | Frozen `(logical name, schema version)` for every artifact (`artifact` module, `schema.rs:6`). | +| `rust/mumdia/crates/mumdia-core/src/manifest.rs` | `Manifest` and `ArtifactRecord` (`manifest.rs:10`, `manifest.rs:22`). | +| `rust/mumdia/crates/mumdia-core/src/config.rs` | The single typed serde `Config` (`config.rs:973`) with per-stage sections and `apply_profile` (`config.rs:1107`). | +| `rust/mumdia/crates/mumdia-io/src/table.rs` | The `Col`/`Table` typed Parquet layer every stage writes through (`table.rs:23`). | +| `rust/mumdia/crates/mumdia-io/src/lib.rs` | `record_artifact`, `inspect`, `init_logging`, blake3 hashing. | + +### Crate responsibilities and predictor plumbing + +Three crates, one direction of dependency (`mumdia` -> `mumdia-io` -> +`mumdia-core`): + +``` +mumdia-core types, mass model, Config (per-stage sections + strategy enums), + schema (frozen artifact names/versions), manifest, errors +mumdia-io Col/Table over arrow+parquet (SNAPPY), blake3 hashing, JSON, + inspect, per-artifact report.json +mumdia bin + lib: fragment index, stages/, predictor + rescorer traits, + sidecar dispatch; main.rs is a thin CLI over the lib +``` + +The two ML predictors are traits with a deterministic native fallback and an +optional Python sidecar, selected by config (the sidecar replaces the native +path without changing callers, `predict.rs:1-8`). The rescorer is not a trait: a +free function plus the `RescorerKind` enum. + +``` +RtPredictor (trait, predict.rs:13) + |-- NativeRt (predict.rs:25, additive RT-coefficient model, deterministic) + \-- DeepLC (Python sidecar; predict_frag.rt_predictor = deeplc) + +FragmentPredictor (trait, predict.rs:19) + |-- NativeFrag (predict.rs:73, heuristic b/y intensities, deterministic) + \-- MS2PIP (Python sidecar; predict_frag.predictor = ms2pip) + +Rescorer (free fn + RescorerKind enum, no trait) + native_tda (percolator_lite) | nn_torch | mokapot | entrapment + (the last three are Python sidecars via rescore.python) +``` + +## Inputs and outputs + +The pipeline consumes an mzML file plus one of two library sources, and produces +a fixed set of Parquet artifacts plus JSON sidecars. Most primary stage Parquets +receive a small `.report.json` with row counts, parameters, hashes, and +timing. Coverage is deliberately partial: report TSVs, schema/PIN files, some +diagnostic/optional Parquets, and several Python-written outputs do not have one. +See docs/03 and docs/12 rather than assuming a report always exists. + +External inputs: +- `--mzml ` (mzML; `.raw`/`.d` must be converted upstream by msconvert). +- FASTA mode: `--fasta ` (digested into the library). +- Library-input mode: `--lib-precursors ` + `--lib-fragments ` + (a prebuilt library, for example an imported DIA-NN speclib). + +Inputs on disk: a pre-built E. coli test library is already present under `lib/`: +`lib_precursors.parquet` + `lib_fragments.parquet` (an imported DIA-NN speclib +with fragment intensities and iRT) plus `lib_precursors_ft.parquet` (the DeepLC +fine-tuned iRT variant, the recommended `--lib-precursors` per docs/18 A4), and a +cached `seed_psms.parquet` (+ `.masscal.json`). See `docs/19_getting_started.md` +for the local sidecar environments and two copy-pasteable end-to-end runs (a +zero-dependency native FASTA run and the best-sensitivity library run). + +Parquet artifacts and their column schemas as written in the code: + +`spectra_ms1.parquet` (`convert.rs:180`): `scan_index` u32, `rt_seconds` f64, +`mz` list, `intensity` list. + +`spectra_ms2.parquet` (`convert.rs:207`): `scan_index` u32, `id` str, +`rt_seconds` f64, `window_id` u32, `window_target` f64, `window_lower` f64, +`window_upper` f64, `precursor_mz` opt, `precursor_charge` opt, +`mz` list, `intensity` list. + +`isolation_windows.parquet` (`convert.rs:224`): `window_id` u32, `target` f64, +`lower` f64, `upper` f64. + +`ms2_to_ms1.parquet` (`convert.rs:234`): `ms2_scan_index` u32, +`ms1_scan_index` i32 (the preceding MS1 scan; -1 if none). + +`peptides.parquet` (digest; FASTA mode only): fully-tryptic stripped peptides +plus their paired decoys. + +`peptidoforms.parquet` (peptidoforms; FASTA mode only): concrete peptidoforms +with fixed/variable mods and charges as ProForma-lite strings. + +`fragment_library_precursors.parquet` (`predict_frag.rs:189`): `candidate_id` +u32, `peptidoform_id` u32, `base_peptide_id` u32, `peptidoform` str, `charge` +i32, `precursor_mz` f64, `predicted_irt` f32, `label` str +(`target`/`decoy`), `protein` str, `n_fragments` i32. `candidate_id` is a +contiguous 0-based index sorted by precursor m/z (the fragment-index build +precondition). + +`fragment_library_fragments.parquet` (`predict_frag.rs:204`): `candidate_id` +u32, `mz` f64, `predicted_intensity` f32, `name` str, `ion_type` str, `ordinal` +i32, `frag_charge` i32. + +`seed_psms.parquet` (`search_seed.rs:220`): `candidate_id` u32, `peptidoform` +str, `charge` i32, `precursor_mz` f64, `base_peptide_id` u32, `protein` str, +`label` str, `score` f64 (hyperscore), `spectrum_q` f64 (per-spectrum +target-decoy q), `observed_rt` f64, `predicted_irt` f32, `matched_peaks` i32, +`scan_index` u32. Sidecar `seed_psms.parquet.masscal.json` carries the per-run +mass recalibration consumed by extract. + +`run_windows.parquet` (`rt_im_train.rs:185`): `candidate_id` u32, `rt_pred_cal` +f64, `rt_lo` f64, `rt_hi` f64, `im_pred_cal`/`im_lo`/`im_hi` opt (IM columns +null in 3D). Sidecar `cal.json` records the fitted RT calibration. + +`psms_extracted.parquet` (`extract.rs:1359`): `candidate_id` u32, `apex_rt` f64, +`apex_im` opt, `apex_intensity` f32, `n_matched_fragments` i32, +`n_predicted_fragments` i32, `coelution_run` i32, `rt_pred_cal` f64, +`precursor_mz` f64, `charge` i32, `label` str, `base_peptide_id` u32, +`peptidoform` str, `protein` str, `predicted_irt` f32, `contested_frac` f64, +`ms1_isom1`/`ms1_mono`/`ms1_iso1`/`ms1_iso2` opt. Optional columns: +`contested_count_frac`, `apportioned_frac` when `emit_contested_features` +(`extract.rs:1383`); `gate_apex`, `gate_peak_spectral`, `gate_coelution`, +`gate_spectral_entropy` when `emit_gate_diagnostics` (`extract.rs:1389`). + +`chromatograms.parquet` (`extract.rs:1399`): `candidate_id` u32, `frag_name` +str, `frag_mz` f64, `frag_obs_mz` f64, `predicted_intensity` f32, `rt` +largelist, `intensity` largelist. `LargeList` (64-bit offsets) is +required because total list-value count can exceed the 32-bit `ListArray` limit +when extraction accepts a very large candidate set. + +`.peaks.parquet` (`extract.rs:1419`, written only when +`extract.retain_top_peaks > 1`): `candidate_id` u32, `peak_rank` i32, `apex_rt` +f64, `start_rt` f64, `end_rt` f64, `evidence_count` f64, `area` f64. This is a +diagnostic sidecar not yet scored downstream (see docs/18 A6, and Gotchas below). + +`features.parquet` (`features.rs:828`): bookkeeping columns `candidate_id`, +`label`, `base_peptide_id`, `peptidoform`, `protein`, `apex_rt`, `elution_lo`, +`elution_hi`, `precursor_mz`, `prelim_score`, followed by one f64 column per +active feature name (`features.rs:839`). Sidecar `features.parquet.schema.json` +records the ordered feature-column list and its hashed `schema_id`; `run.pin` is +the Percolator input written deterministically alongside. + +`psms_competed.parquet` (`compete.rs:118`): the same bookkeeping columns plus +every feature column carried through unchanged (`compete.rs:128`), for surviving +rows only. Sidecar `psms_competed.parquet.schema.json` carries the schema +forward for rescore. + +`psms_scored.parquet` (schema version 3): `candidate_id` u32, +`peptidoform` str, `charge` i32, `label` str, `protein` str, `base_peptide_id` +u32, carried `apex_rt`/`elution_lo`/`elution_hi`, `score` f64, `q_value` f64, +`peptide_q_value` f64, `protein_group` str, +`pg_q_value` f64, `global_q_value` f64, `prelim_score` f64, `source` u32, +`run_psm_q` f64, `experiment_psm_q` f64, `precursor_q` f64. The q-value columns +are independent per level, not a rollup (see Gotchas). + +`peptide_quant.parquet` (schema version 2): `candidate_id`, `base_peptide_id`, +`peptidoform`, `charge`, `protein_group`, nullable `quantity`, `quant_status`, +`n_fragments_used`, and nullable applied integration apex/bounds. +`protein_group_quant.parquet` (schema version 2): `protein_group`, nullable +`quantity`, `quant_status`, and `n_peptides` (unique positive base peptides). +Optional `fragment_quant.parquet` and +`peak_bounds` (`quant.rs:278`) diagnostics. `quant-lfq` emits a +protein-by-run matrix (`quant.rs:479`): `protein_group`, `run` i32, `quantity`, +`n_features` i32. + +`candidate_audit.parquet` (`audit.rs:180`, written when +`extract.emit_candidate_audit` or via `mumdia audit`): `run_id` str, +`precursor_id` u32, `modified_sequence` str, `charge` i32, +`target_decoy_label` str, `entrapment_label` bool, then the boolean stage-flags +`candidate_generated`, `traces_extracted`, `peak_generated`, `peak_selected`, +`variant_selected`, `target_decoy_winner`, `passed_precursor_fdr`, +`passed_peptide_fdr`, `reported`, and `rejection_reason` str (earliest loss +reason). + +Human-readable outputs: `peptides.tsv`, `proteins.tsv` (report), and +`manifest.json` (orchestrator). + +## CLI subcommands + +`main.rs` is a thin `clap` layer: the `Cmd` enum (`main.rs:20`) defines one +subcommand per stage plus the `run` orchestrator, the experiment-level +`align`/`mbr`, and the utilities `inspect`/`report`/`doctor`. Each stage arm +loads the config, hashes its canonical JSON into a `config_hash`, and calls the +stage `run`; every stage command is runnable standalone on prior outputs. Flags +are `--kebab-case`; a `num_args = 1..` flag accepts several values. + +Which subcommands read `--config` (verified against `main.rs`): + +| Takes `--config` | No `--config` | +|---|---| +| `digest`, `peptidoforms`, `predict-frag`, `search-seed`, `rt-im-train`, `extract`, `features`, `compete`, `rescore`, `quant`, `run`, `align`, `mbr`, `doctor` (optional) | `convert`, `quant-lfq`, `inspect`, `audit`, `report` | + +The five with no `--config` are fixed-behavior: `convert` always runs on +`Config::default()` (`main.rs:397`); `quant-lfq` takes typed string flags instead +(`main.rs:642-658`); `inspect`, `audit`, and `report` load no config at all +(`main.rs:712`, `main.rs:557`, `main.rs:715`). The full subcommand set, in source +order: + +| `mumdia ` | Inputs (flags) | Outputs (flags / artifacts) | +|---|---|---| +| `convert` (`main.rs:23`) | `--mzml`, `--out-dir`; caps `--max-spectra`, `--top-peaks-ms2`, `--top-peaks-ms1` (each `0` = uncapped) | `spectra_ms1`, `spectra_ms2`, `isolation_windows`, `ms2_to_ms1` in `--out-dir` | +| `digest` (`main.rs:43`) | `--fasta`, `--config` | `--out` = `peptides.parquet` | +| `peptidoforms` (`main.rs:52`) | `--peptides`, `--config` | `--out` = `peptidoforms.parquet` | +| `predict-frag` (`main.rs:61`) | `--peptidoforms`, `--work-dir` (default `sidecar_work`), `--config` | `--out-precursors`, `--out-fragments` (the library pair) | +| `search-seed` (`main.rs:75`) | `--ms2`, `--library-precursors`, `--library-fragments`, `--config` | `--out` = `seed_psms.parquet` (+ `.masscal.json`) | +| `rt-im-train` (`main.rs:88`) | `--seed-psms`, `--library-precursors`, `--config` | `--out-windows` = `run_windows.parquet`, `--out-cal` = `cal.json` | +| `extract` (`main.rs:101`) | `--ms2`, `--library-precursors`, `--library-fragments`, `--run-windows`, `--ms1` (opt), `--mass-cal` (opt), `--restrict-candidates` (opt allowlist), `--config` | `--out-psms` = `psms_extracted.parquet`, `--out-chrom` = `chromatograms.parquet` (+ `.peaks.parquet` when `retain_top_peaks>1`) | +| `features` (`main.rs:130`) | `--psms`, `--chromatograms`, `--seed` (opt), `--config` | `--out` = `features.parquet` (+ `.schema.json`), `--out-pin` = PIN | +| `compete` (`main.rs:146`) | `--features`, `--config` | `--out` = `psms_competed.parquet` (+ `.schema.json`) | +| `rescore` (`main.rs:155`) | `--competed` (1+ competed tables), `--config` | `--out` = `psms_scored.parquet` | +| `quant` (`main.rs:165`) | `--psms-scored`, `--chromatograms`, `--config` | `--out-peptide`, `--out-protein`, `--out-fragment` (opt), `--out-peak-bounds` (opt) | +| `quant-lfq` (`main.rs:184`) | `--inputs` (1+ per-run tables), `--method` (`maxlfq` default / `directlfq`), `--normalize` (`median_ratio` default / `median` / `none`) | `--out` = protein-by-run matrix; no `--config` | +| `run` (`main.rs:199`) | `--mzml`, `--out-dir`; `--fasta` xor (`--lib-precursors` + `--lib-fragments`); `--config`, `--profile`, `--max-spectra`, `--top-peaks-ms2` | the full chain + `manifest.json` | +| `align` (`main.rs:230`) | `--seeds` (1+ `seed_psms`, first = reference), `--config` | `--out` = `alignment.parquet` | +| `mbr` (`main.rs:240`) | `--scored` (experiment-wide `scored_combined`), `--psms` (1+ per-run in `source` order), `--frag` (0+ per-run `fragment_quant`), `--config` | `--out` = `transferred.parquet`, `--out-scored` (opt augmented scored) | +| `inspect` (`main.rs:261`) | positional `artifact` (any parquet) | schema + head + row count to stdout; no `--config` | +| `audit` (`main.rs:265`) | `--library-precursors`, `--psms`, `--competed`, `--scored`, `--q` (0.01), `--run-id` (`run`), `--entrapment-substr` (`""`) | `--out` = `candidate_audit.parquet`; no `--config` | +| `report` (`main.rs:292`) | `--scored`, `--peptide-quant` (opt), `--protein-quant` (opt), `--q` (0.01) | `peptides.tsv` + `proteins.tsv` in `--out-dir` | +| `doctor` (`main.rs:305`) | `--config` (opt) | probes the configured sidecar interpreters; nonzero exit if any FAIL | + +Per-subcommand specifics that are easy to miss: + +- `convert` is the only stage command with no `--config`; it always runs on + `Config::default()` (`main.rs:397`). Because the three caps are not part of the + config, `convert` folds them into the artifact `config_hash` (`main.rs:402`, + comment.md A2/C4) so two different caps do not collide on an identical hash. + `top_peaks_ms2` is an irreversible conversion-time cap that also affects + extraction/features/quant; `search_seed.top_n_peaks` is the seed-only + alternative. +- `quant-lfq`, `inspect`, `audit`, and `report` also take no `--config`. + `quant-lfq` validates its two string flags: `--method` must be `maxlfq` or + `directlfq` (`main.rs:649`) and `--normalize` is parsed by + `NormalizeMethod::from_token` (`config.rs:754`), erroring on anything but + `median_ratio`/`median`/`none` (`main.rs:652`). +- `search-seed` reads `extract.bucket_size` from the config (not a `search_seed` + field) for its fragment-index bucketing (`main.rs:473`, `run.rs:225`). +- `rescore` standalone accepts several `--competed` tables for experiment-wide + scoring and uses a fixed `sidecar_work` working directory (`main.rs:588`); + inside `run` it is passed exactly one table and a per-out-dir work directory. +- `align` uses `rt_im_train.q_train` for its training set and a hardcoded 100-knot + grid (`main.rs:666-667`); it needs >=2 seeds and is not part of the `run` chain. +- `mbr` (`main.rs:671`) is a wired standalone command, not part of `run`. It + hard-errors when `mbr.strategy = none` (`main.rs:680`), when fewer than two + `--psms` are supplied (`main.rs:685`), or when `mbr.python` is unset + (`main.rs:688`), then runs the `mbr_worker.py` sidecar with the `mbr.*` + thresholds (`main.rs:697`). It transfers identifications across a run set + (Stage D3); see the extend section for its stub status inside `run`. +- `doctor` (`main.rs:313`) probes three interpreters and prints `[ ok ]` / + `[FAIL]` / `[skip]` per line, exiting nonzero if any fail: `rescore.python` + (packages depend on the classifier: `torch,numpy,pandas,pyarrow` for `nn_torch`, + else `mokapot,sklearn,numpy,pandas,pyarrow`), `predict_frag.deeplc_python` + (`deeplc,numpy,pandas`), and `predict_frag.ms2pip_python` + (`ms2pip,numpy,pandas`). + +## How it works + +### The pipeline as a graph + +``` + FASTA mode library-input mode + (--fasta present) (--lib-precursors + --lib-fragments) + | | + digest peptides.parquet | + | | + peptidoforms peptidoforms.parquet | + | | + predict-frag fragment_library_precursors.parquet (supplied directly, digest/ + fragment_library_fragments.parquet peptidoforms/predict-frag skipped) + \________________________________________/ + | + lib_p (precursors) + lib_f (fragments) + | + --mzml ---> convert ---> spectra_ms1.parquet, spectra_ms2.parquet, + isolation_windows.parquet, ms2_to_ms1.parquet + | + spectra_ms2 + lib_p + lib_f --> search-seed --> seed_psms.parquet + seed_psms.parquet.masscal.json + | + [optional] lib_p + seed --> deeplc_finetune --> fragment_library_precursors_ft.parquet + (new table with updated predicted_irt; lib_p := *_ft) + | + seed + lib_p --> rt-im-train --> run_windows.parquet, cal.json + | + spectra_ms2 + lib_p + lib_f + run_windows + spectra_ms1 + masscal + --> extract --> psms_extracted.parquet, chromatograms.parquet + [.peaks.parquet if retain_top_peaks>1] + | + psms_extracted + chromatograms + seed --> features --> features.parquet, + features.parquet.schema.json, run.pin + | + features --> compete --> psms_competed.parquet (+ .schema.json) + | + psms_competed --> rescore --> psms_scored.parquet + | + [optional] lib_p + psms_extracted + competed + scored --> audit --> candidate_audit.parquet + | + psms_scored + chromatograms --> quant --> peptide_quant.parquet, + protein_group_quant.parquet, fragment_quant.parquet + | + psms_scored + quant --> report --> peptides.tsv, proteins.tsv + | + selected primary Parquets recorded --> manifest.json +``` + +### The two library sources and how run.rs branches + +The spectral library is the experiment-level, run-independent artifact pair +`(lib_p, lib_f)`. The orchestrator produces it in one of two ways, decided by a +`match` on `(p.lib_precursors, p.lib_fragments)` at `run.rs:98`: + +- Library-input mode (`(Some(lp), Some(lf))`, `run.rs:99`): the supplied library + is consumed directly. `digest`, `peptidoforms`, and `predict-frag` are skipped + and the FASTA is never read. The orchestrator only reads the two Parquet files + to record their row counts in the manifest, then uses their paths downstream + (`run.rs:107-125`). This is the highest-sensitivity path, used with an imported + DIA-NN speclib that already carries fragment intensities and iRT. + +- FASTA-digest mode (the `_ =>` arm, `run.rs:127`): the library is built from the + FASTA. `digest::run` writes `peptides.parquet` (`run.rs:132`), + `peptidoforms::run` writes `peptidoforms.parquet` (`run.rs:149`), and + `predict_frag::run` writes both library Parquet files (`run.rs:166`). The + returned `(lib_p, lib_f)` paths feed the rest of the chain. This path has zero + external runtime dependencies (native predictors). + +`preflight` (`run.rs:38`) validates the mode before any compute: exactly one of +the two source configurations must be present, and every required input path must +exist (`run.rs:42-61`). It also checks sidecar prerequisites: `finetune_deeplc` +requires `predict_frag.deeplc_python`; Mokapot and NnTorch require +`rescore.python`; entrapment requires `rescore.entrapment_marker`. `Config::validate` +rejects the same rescorer omissions at load time. `mumdia doctor` additionally +checks that the configured interpreters can import the required packages. + +### The per-run chain + +After the library is resolved, the orchestrator runs the per-run stages in order, +recording each output in the manifest with `record_artifact` (`run.rs:85-509`): + +1. `convert::run` (`run.rs:196`) reads the mzML through mzdata, centroids profile + spectra, caps peaks, synthesizes a full-range window for AIF/all-ion scans, + and writes the four spectra artifacts. It returns a `ConvertOutputs` struct of + paths (`convert.rs:96`). `run` forwards its `max_spectra` and `top_peaks_ms2` + but forces `top_peaks_ms1 = 0` (`run.rs:201`), so `run` never caps MS1 peaks; + only the standalone `convert` command can (`--top-peaks-ms1`). + +2. `search_seed::run` (`run.rs:219`) runs a native Sage-lite broad DIA search over + `spectra_ms2` against the fragment index for calibration (not final ID). It + writes `seed_psms.parquet` and the mass recalibration `masscal.json`. The seed + is iRT-independent, so it is computed once here on the base library and reused. + It borrows `extract.bucket_size` for the fragment-index bucketing + (`run.rs:225`), not a `search_seed` field. + +3. Optional DeepLC multitask fine-tune (`run.rs:242`, gated on + `rt_im_train.finetune_deeplc`): `sidecar::run_deeplc_finetune` adapts the RT + model to this run's confident seed PSMs and writes a new + `fragment_library_precursors_ft.parquet` whose `predicted_irt` values are + updated; the input library is not modified. It is driven by the `rt_im_train` + fields `finetune_epochs`, `finetune_patience`, `q_train` (the confident-seed + cutoff), and `finetune_batch` (`run.rs:259-262`; `finetune_batch = 0` + auto-scales the batch to the seed size). The `lib_p` binding is then rebound to + the fine-tuned file so rt-im-train and extract read it. `lib_f` is unchanged + (fine-tune touches iRT only). This step is nondeterministic (no fixed + torch/numpy seed). + +4. `rt_im_train::run` (`run.rs:284`) maps the run-independent iRT onto this run's + observed RT (LOESS or linear) and sets per-candidate RT windows from the + residual percentile, writing `run_windows.parquet` and `cal.json`. + +5. `extract::run` (`run.rs:303`) is the core stage: a peak-major cascade over the + inverted fragment index. It reads `spectra_ms2`, both library files, + `run_windows`, `spectra_ms1` (for MS1 isotope XICs), and the `masscal.json`; it + writes `psms_extracted.parquet` and `chromatograms.parquet`. `run` passes + `restrict_candidates: None` (no allowlist). + +6. `features::run` (`run.rs:335`) computes the config-selected feature vector plus + `prelim_score` per PSM and emits `features.parquet`, its schema sidecar, and + `run.pin`. It also reads `seed` for search-engine corroboration features. + +7. `compete::run` (`run.rs:354`) keeps the best candidate per competition group + before FDR counting, writing `psms_competed.parquet` and carrying the feature + schema forward. + +8. `rescore::run` (`run.rs:370`) does semi-supervised rescoring and native + target-decoy q-values at multiple contexts, writing `psms_scored.parquet`. In + `run` it is passed a single competed table (`std::slice::from_ref(&competed)`), so this is + single-run rescoring; the standalone `rescore` command accepts several tables + for experiment-wide scoring. + +9. Optional `audit::run` (`run.rs:405`, gated on `extract.emit_candidate_audit`) + reconstructs the per-candidate identification-loss ladder into + `candidate_audit.parquet`. It is a cheap join over the artifact chain and runs + no extraction. Inside `run` its parameters are fixed: `q_threshold = 0.01`, + `run_id = out_dir`, and an empty `entrapment_substr` (`run.rs:413-415`); the + standalone `audit` command exposes all three as flags. + +10. `quant::run` (`run.rs:422`) integrates fragment chromatograms and rolls up to + protein groups, writing `peptide_quant`, `protein_group_quant`, and + `fragment_quant`. + +11. `report::run` writes `peptides.tsv` and `proteins.tsv` from the scored table + plus quant, thresholded at `quant.q_threshold`. `run` reads the rescore + artifact report and uses its actual classifier in stdout and the manifest; + `psms_scored.parquet.report.json` remains the authoritative record. + +### The manifest + +`Manifest` (`manifest.rs:22`) has five fields: `mumdia_version` (the crate +version, `env!("CARGO_PKG_VERSION")`, stamped by `Manifest::new`), `config_json` +(the fully-resolved canonical config), `config_hash` (its blake3 hash), +`model_identities` (a `BTreeMap`), and `artifacts` (a +`BTreeMap`). `Manifest::new` (`manifest.rs:34`) seeds it +with the canonical config JSON and hash (`run.rs:89,93`). Selected primary +Parquet outputs are recorded by `record_artifact` (`mumdia-io`), which builds an +`ArtifactRecord` +(`manifest.rs:10`) with nine fields: `logical_name`, `path`, `format`, +`schema_name`, `schema_version`, `rows`, `content_hash`, `producing_stage`, and +`config_hash`. `Manifest::record` (`manifest.rs:44`) keys artifacts by +`logical_name`, so re-recording the same logical name overwrites and the map is +sorted, not insertion-ordered. The `producing_stage` is the stage name +(`digest`, `convert`, ...); in library-input mode the two library records carry +the synthetic stage `"library-input"` (`run.rs:114,122`). Four `model_identities` +keys are recorded: `rt_predictor`, `fragment_predictor`, `rescorer`, and +`feature_schema_id`. The rescorer identity comes from the rescore report, and the +fine-tuned precursor table is recorded when it is the table actually consumed +downstream. Calibration JSON, PIN/schema companions, report TSVs, and some +optional diagnostic sidecars are not manifest artifacts. The manifest is written last to +`/manifest.json` (`run.rs:501`) with `mumdia_io::json::write_json`. The +manifest is provenance recorded, not required: because inputs are +path-addressable, no stage depends on it to run (docs/18 B1). + +### Current best workflow + +The validated best-performing configuration (docs/18 A1 and its "current best +workflow") is library-input mode with: +- an imported DIA-NN library (fragment intensities + iRT), +- per-run DeepLC multitask fine-tune of iRT (`rt_im_train.finetune_deeplc = true` + + `predict_frag.deeplc_python`), which is essential in library mode because raw + DIA-NN iRT is locally noisy (~110 s MAD) and the fine-tune tightens it to + ~13-27 s (docs/18 A4), +- the Extended feature set (`features.set = extended`, applied by `--profile dia`, + `config.rs:1101`), +- the `nn_torch` PyTorch MLP rescorer (`rescore.classifier = nn_torch` + + `rescore.python`, with `rescore.strict = true`), which is the dominant + sensitivity lever and beats the native linear rescorer by ~8.5% (docs/18 A1), +- a loose default apex-intensity Pearson gate + (`extract.gate_mode = apex_pearson`, `extract.min_frag_corr ~= 0.2`), since + a strong nonlinear rescorer prefers recall and absorbs the loose-gate flood + (docs/18 A1/A2), +- the conversion-time MS2 peak cap explicitly kept at 300 on AIF + (`--top-peaks-ms2 300`; both conversion entry points otherwise default to + uncapped). This is distinct from seed-only `search_seed.top_n_peaks`. + +On `LFQ_Orbitrap_AIF_Ecoli_01.mzML` this historically reaches ~10,300 +precursor-shaped `(peptidoform, charge)` report rows selected at +`peptide_q_value <= 0.01`; it is not a precursor-q count or a universal preset. +The zero-dependency native FASTA-digest mode (~1,213 rows under the same report +definition) is the high-precision fallback. See docs/20 for the exact command, +acceptance gates, and quantification-specific choices. + +## Key types and functions + +| Name | file:line | What it does | +|---|---|---| +| `Cmd` (enum) | `main.rs:20` | One clap subcommand per stage; `Run` at `main.rs:199`. | +| `main` | `main.rs:386` | Parses the CLI, loads config, dispatches to the stage `run`. | +| `load_config` | `main.rs:376` | Reads a config JSON or returns `Config::default()`. | +| `doctor` | `main.rs:313` | Probes configured sidecar interpreters for required packages. | +| `RunParams` | `run.rs:18` | The orchestrator's inputs (config, fasta, mzml, out_dir, lib paths, caps). | +| `preflight` | `run.rs:38` | Validates mode, input existence, and sidecar prerequisites before compute. | +| `stages::run::run` | `run.rs:85` | The orchestrator: library branch, per-run chain, manifest write. | +| `Manifest` / `ArtifactRecord` | `manifest.rs:22` / `manifest.rs:10` | Provenance record for a chained run. | +| `artifact` module | `schema.rs:6` | Frozen `(logical name, schema version)` constants. | +| `Config::apply_profile` | `config.rs:1107` | Applies the `dia` preset (Extended features, apex window 5, RT prior 120 s). | +| `Config::canonical_json` | `config.rs:1116` | Canonical JSON for hashing into the manifest. | +| `NormalizeMethod::from_token` | `config.rs:754` | Parses `--normalize` for `quant-lfq` (`median_ratio`/`median`/`none`). | +| `Manifest::record` | `manifest.rs:44` | Inserts an `ArtifactRecord` keyed by `logical_name` (overwrites, sorted map). | +| `Col` (enum) | `table.rs:23` | Typed column for writing Parquet; `LargeListF32` for chromatograms. | +| `Table::read` + typed getters | `table.rs:204` | Reads a Parquet artifact and extracts typed columns. | + +## Configuration + +The orchestrator reads a single typed `Config` (`config.rs:973`) with per-stage +sections. Every field has `#[serde(default)]` and the struct uses +`deny_unknown_fields`, so a partial JSON overrides only named fields and an +unknown key is a hard error. `--profile dia` is applied on top of the loaded +config (`main.rs:607`) before `run`. The fields this overview area touches: + +| Field | Default | Effect | +|---|---|---| +| `rng_seed` | `0` | Seeds decoy generation and rescorer folds (determinism). | +| `digest.decoy.strategy` | `reverse` | Decoy scheme; `diann_shift` and `none` are rejected/no-op. | +| `peptidoforms.charge_min` / `charge_max` | `2` / `3` | Precursor charge range enumerated in FASTA mode. | +| `predict_frag.rt_predictor` | `native` | `native` or `deeplc` iRT source (recorded in manifest). | +| `predict_frag.predictor` | `native` | `native` or `ms2pip` fragment intensities. | +| `predict_frag.deeplc_python` | `None` | Interpreter for DeepLC; required if `finetune_deeplc`; probed by `doctor`. | +| `predict_frag.ms2pip_python` | `None` | Interpreter for MS2PIP; probed by `doctor` (`main.rs:332`). | +| `predict_frag.sidecar_script_dir` | `"scripts"` | Where sidecar workers are resolved from (used by `rescore`/`mbr`). | +| `search_seed.top_n_peaks` | `300` | Seed-only MS2 peak cap (keep 300 on AIF, docs/18 A3). | +| `extract.bucket_size` | (see `ExtractConfig`) | Fragment-index bucket width; `search-seed` reads it too (`main.rs:473`). | +| `rt_im_train.calibration_method` | `loess` | RT calibration; `none` is rejected by validation. | +| `rt_im_train.q_train` | (see `RtImTrainConfig`) | Confident-seed q cutoff for the DeepLC fine-tune and the `align` reference set. | +| `rt_im_train.finetune_deeplc` | `false` | Enables the per-run DeepLC fine-tune of iRT (best workflow). | +| `rt_im_train.finetune_epochs` / `finetune_patience` | (see config) | DeepLC fine-tune schedule (`run.rs:259-260`). | +| `rt_im_train.finetune_batch` | `0` | `0` auto-scales the fine-tune batch to seed size. | +| `extract.min_frag_corr` | `0.2` | Spectral-agreement gate threshold; loose default suits `nn_torch`. | +| `extract.gate_mode` | `apex_pearson` | Which spectral score the gate thresholds (`GateMode`, `config.rs:591`). | +| `extract.retain_top_peaks` | `1` | `>1` writes the `.peaks.parquet` sidecar (not yet scored). | +| `extract.emit_candidate_audit` | `false` | Emits `candidate_audit.parquet` in `run`. | +| `extract.emit_gate_diagnostics` | `false` | Adds the four `gate_*` columns to `psms_extracted`. | +| `features.set` | `minimal` | `minimal` (14) / `rich` (44) / `extended` (381, per the `feature_sets_sized` test `features.rs:1590-1598`); `dia` profile sets extended. | +| `compete.mode` | `winner_take_all` | Within-group competition resolution (`CompetitionMode`). | +| `rescore.classifier` | `native_tda` | Rescorer; `nn_torch` is the best lever, needs `rescore.python`. | +| `rescore.python` | `None` | Interpreter for mokapot / nn_torch / entrapment sidecars. | +| `mbr.strategy` | `none` | MBR mode; `none` makes the standalone `mumdia mbr` command error out (`main.rs:680`). | +| `mbr.python` | `None` | `mbr_worker.py` interpreter; required when `strategy != none` (`main.rs:688`). | +| `quant.q_threshold` | `0.01` (`QuantConfig`) | q-value threshold for the human-readable report. | + +The config surface was recently pruned of dead or unwired fields (see +`docs/02_config_and_data_model.md`): `threads`, `extract.{k_select, max_fragment_charge, +scan_scale, scan_window_mode}` + `ScanWindowMode`, `digest.decoy.{ratio, source}` ++ `DecoySource`, `search_seed.precursor_tol_ppm`, `rt_im_train.tolerance_regime` ++ `ToleranceRegime`, `FeatureSet::Custom`, `MatcherKind::Naive`, and +`CompetitionMode::from_token` were removed. Do not reintroduce them. Kept +deliberately as documented hooks: `DecoyStrategy::DiannShift` (deferred, +license-checked; rejected by validation), `CalibrationMethod::None` (rejected by +validation with a clear message), and the whole `mbr` section, which is wired to +the standalone `mumdia mbr` command (`main.rs:671`, calls `mbr_worker.py`) but is +not part of the `run` chain and needs >=2 runs. + +## Invariants, determinism, gotchas + +- Determinism is required (docs/18 B2, the determinism contract): seed the RNG, + keep numeric summation order fixed. A HashMap f32 sum shifting the apex once broke + reproducibility; use ordered maps or sorted iteration where floats are summed. + The DeepLC fine-tune is the one deliberate exception (nondeterministic; no fixed + torch seed), so a `run` with `finetune_deeplc = true` is not bit-reproducible. +- `candidate_id` must be a contiguous 0-based index sorted by precursor m/z; the + fragment index build in extract depends on it. An imported library must be + re-indexed by `make_reverse_decoys.py` to satisfy this. +- q-value columns in `psms_scored` are independent per level, not a rollup + (docs/18 A5): `q_value` (== `experiment_psm_q`), `run_psm_q` (per source), + `precursor_q` (peptidoform+charge), `peptide_q_value` (stripped sequence), + `pg_q_value`. Coarser grouping pools evidence, so peptide counts can exceed + precursor counts. Report at the correct context: `precursor_q` for a precursor + matrix, `run_psm_q` for cross-run library-mode quant. Do not threshold PSM q + then deduplicate. +- Every default-off knob keeps the production chain byte-identical when unset; + `retain_top_peaks=1`, `emit_candidate_audit=false`, `emit_gate_diagnostics=false`, + and the other sensitivity-program toggles all default to the legacy path. +- The extraction gate is a training-pool and null-curation lever, not feature + work (docs/18 A2). Loosening it floods the pool with decoy-enriched junk that + corrupts a linear rescorer; a nonlinear rescorer absorbs it. The gate optimum + therefore inverts by rescorer. +- The selected apex is the strongest peak only ~48-52% of the time (docs/18 A6); + the `.peaks.parquet` top-K sidecar exists to expose alternative peaks but + is not yet promoted through features/rescore. + +## How to extend / modify + +- To add a stage: implement it as `stages/.rs` with a `run(Params)` that + reads path-addressable inputs and writes Parquet through the `Col`/`Table` + layer plus an `.report.json`; add a `(logical name, version)` to + `schema.rs`; add a `Cmd` arm in `main.rs`; and thread it into `run.rs` with a + `record_artifact` call. Keep the stage runnable standalone. +- To add an artifact column: add the `Col::` in the writing stage, update the + reading stage's typed getter, and bump the schema version in `schema.rs` if the + change is not backward-compatible (`PSMS_SCORED` is already at version 3). +- To add a tuning profile: extend the `match` in `Config::apply_profile` + (`config.rs:1107`); it should only set existing typed config fields, never add + plumbing. +- To add a library source: extend the `match` on `(lib_precursors, + lib_fragments)` in `run.rs:98` and the corresponding `preflight` branch + (`run.rs:43`). The rest of the chain consumes `(lib_p, lib_f)` unchanged. +- To wire a new sidecar: follow the positional-CLI file contract + (`sidecar::resolve_script` + a worker in `scripts/`), gate it behind a config + field, and check its prerequisites in `preflight` and `doctor`. +- Stubs and unwired areas to be aware of: `mbr` has a wired standalone command + (`mumdia mbr` -> `mbr_worker.py`) but is a partial sidecar (not the full + experiment-wide flow) and is not in the `run` chain (needs >=2 runs); `align` is + likewise standalone-only (needs >=2 runs); ion mobility / 4D is a data-model + stub only (IM columns are always null in 3D); vendor readers other than mzML do + not exist. Do not document these as complete. diff --git a/docs/02_config_and_data_model.md b/docs/02_config_and_data_model.md new file mode 100644 index 0000000..f174e8d --- /dev/null +++ b/docs/02_config_and_data_model.md @@ -0,0 +1,743 @@ +# Config, mass model, constants, schema, manifest +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +This document covers the `mumdia-core` crate: the shared vocabulary that every +stage depends on but that contains no stage logic itself. Four concerns live +here. + +1. **Typed configuration** (`config.rs`): one serde structure with per-stage + sections and one strategy enum per algorithmic choice point. `Config` parses + from JSON, rejects unknown keys, and validates on load so a misconfiguration + fails at startup rather than producing a bogus run. +2. **Mass model** (`mass.rs` + `constants.rs`): the single source of residue + masses, physical constants, m/z conversion, ppm predicates, ProForma-lite + peptidoform parsing, and b/y fragment generation. No other crate defines a + mass constant or a ppm predicate. +3. **Frozen artifact schema ids** (`schema.rs`): `(logical name, version)` pairs + recorded in each artifact's `report.json` and in the `manifest.json` + `ArtifactRecord`. They are provenance only: no stage reads them back, and + `Table::read` (`table.rs:207-225`) does no version check, so a schema id is + never used to detect a mismatch or invalidate a downstream artifact. +4. **Run manifest** (`manifest.rs`): per-artifact provenance (content hash, + producing stage, config hash) written once by the `run` orchestrator. + +The crate is declared in `lib.rs:6-13` (`config`, `constants`, `error`, +`manifest`, `mass`, `rejection`, `schema`, `types`) and exposes +`version()` from `CARGO_PKG_VERSION` (`lib.rs:15-17`). + +## Files + +| Path | Role | +|---|---| +| `rust/mumdia/crates/mumdia-core/src/config.rs` | All config structs + every strategy enum; `Config::from_json`, `validate`, `apply_profile`, `canonical_json` (1188 lines) | +| `rust/mumdia/crates/mumdia-core/src/constants.rs` | Physical constants (`PROTON`, `WATER`, `AMMONIA`, `ISOTOPE_SPACING`), `residue_mass`, `mass_to_mz`, ppm predicates | +| `rust/mumdia/crates/mumdia-core/src/mass.rs` | `unimod_mass`, `IonType`, `Fragment`, `ParsedPeptidoform`, `parse_peptidoform`, b/y fragment generation | +| `rust/mumdia/crates/mumdia-core/src/schema.rs` | Frozen `(name, version)` ids for every artifact; `PSMS_SCORED` is v3, `PSMS_COMPETED`/`PEPTIDE_QUANT`/`PROTEIN_GROUP_QUANT` are v2, all others v1 | +| `rust/mumdia/crates/mumdia-core/src/manifest.rs` | `Manifest` + `ArtifactRecord` (provenance) | +| `rust/mumdia/crates/mumdia-core/src/types.rs` | `Peak`, `IsolationWindow`, `Label`, `Ms2Scan` | +| `rust/mumdia/crates/mumdia-core/src/rejection.rs` | `RejectionReason` ladder for the candidate-audit table | +| `rust/mumdia/crates/mumdia-core/src/error.rs` | `MassError`, `ConfigError` (thiserror) | +| `rust/mumdia/crates/mumdia-core/src/lib.rs` | Module wiring + `version()` | +| `rust/mumdia/crates/mumdia-io/src/lib.rs` | `record_artifact` (builds `ArtifactRecord` by hashing the file), `inspect` | +| `rust/mumdia/crates/mumdia-io/src/hash.rs` | `blake3_file`, `blake3_str` (content + config hashing) | +| `rust/mumdia/crates/mumdia-io/src/report.rs` | `ArtifactReport` written as `.report.json` | + +## Inputs and outputs + +`mumdia-core` produces no artifacts of its own. It defines the types the other +crates read and write. Two things it owns appear on disk: + +**`manifest.json`** (one per `run`, written at `run.rs:500-501`). Serialized +`Manifest`; fields (`manifest.rs:22-31`): + +| Field | Type | Meaning | +|---|---|---| +| `mumdia_version` | String | `CARGO_PKG_VERSION` at build time (`manifest.rs:36`) | +| `config_json` | String | Fully-resolved config, from `Config::canonical_json()` | +| `config_hash` | String | `blake3_str(canonical_json)` (`run.rs:89`) | +| `model_identities` | BTreeMap | `rt_predictor`, `fragment_predictor`, `rescorer`, `feature_schema_id` (`run.rs:489-498`) | +| `artifacts` | BTreeMap | one entry per produced artifact, keyed by logical name | + +**`ArtifactRecord`** (`manifest.rs:9-20`), one per artifact: +`logical_name`, `path`, `format` (always `"parquet"`), `schema_name`, +`schema_version`, `rows`, `content_hash` (blake3 of the file bytes), +`producing_stage`, `config_hash`. Built by `record_artifact` +(`mumdia-io/src/lib.rs:20-39`), which hashes the written Parquet file with +`blake3_file` (`hash.rs:8-20`, streamed in 64 KiB chunks). + +**`ArtifactReport`** / `.report.json` (`report.rs:11-24`) is written +alongside each artifact by its producing stage, not by core: `logical_name`, +`schema_name`, `schema_version`, `stage`, `rows`, `content_hash`, `params` +(resolved parameters), `stats` (key metrics), `model_identity`, `elapsed_ms`. +`write_for` appends `.report.json` to the artifact path (`report.rs:26-31`). + +### Artifact schema ids (`schema.rs:7-24`) + +Every id is a `(&str, u32)` constant. A stage stamps `.0` and `.1` into its +`ArtifactReport` and `ArtifactRecord`. Both are provenance records only; nothing +reads them back to validate a downstream read (`Table::read`, `table.rs:207-225`, +does no version check, and `write_table`, `table.rs:151-196`, does not stamp the +id into the Parquet file itself). + +| Constant | Logical name | Version | +|---|---|---| +| `SPECTRA_MS1` | `spectra_ms1` | 1 | +| `SPECTRA_MS2` | `spectra_ms2` | 1 | +| `ISOLATION_WINDOWS` | `isolation_windows` | 1 | +| `MS2_TO_MS1` | `ms2_to_ms1` | 1 | +| `PEPTIDES` | `peptides` | 1 | +| `PEPTIDOFORMS` | `peptidoforms` | 1 | +| `FRAGMENT_LIBRARY_PRECURSORS` | `fragment_library_precursors` | 1 | +| `FRAGMENT_LIBRARY_FRAGMENTS` | `fragment_library_fragments` | 1 | +| `SEED_PSMS` | `seed_psms` | 1 | +| `RUN_WINDOWS` | `run_windows` | 1 | +| `PSMS_EXTRACTED` | `psms_extracted` | 1 | +| `CHROMATOGRAMS` | `chromatograms` | 1 | +| `FEATURES` | `features` | 1 | +| `PSMS_COMPETED` | `psms_competed` | **2** | +| `PSMS_SCORED` | `psms_scored` | **3** | +| `PEPTIDE_QUANT` | `peptide_quant` | **2** | +| `PROTEIN_GROUP_QUANT` | `protein_group_quant` | **2** | +| `FRAGMENT_QUANT` | `fragment_quant` | 1 | + +`PSMS_SCORED` v3 carries the identification apex/bounds through rescoring so +quant can integrate the selected peak. Its column layout is +written by the rescore stage and is the schema a +downstream reader must expect: + +| Column | Type | Meaning | +|---|---|---| +| `candidate_id` | U32 | library candidate id | +| `peptidoform` | Str | ProForma-lite string | +| `charge` | I32 | precursor charge | +| `label` | Str | `target` / `decoy` | +| `protein` | Str | protein accession | +| `base_peptide_id` | U32 | interned stripped-peptide id (peptide-level q grouping) | +| `apex_rt` | F64 | identification apex used by downstream quant | +| `elution_lo` / `elution_hi` | F64 | identified elution bounds carried through | +| `score` | F64 | rescorer score | +| `q_value` | F64 | per-PSM q (pooled) | +| `peptide_q_value` | F64 | peptide-level q (global per base peptide) | +| `protein_group` | Str | protein-group key | +| `pg_q_value` | F64 | protein-group q | +| `global_q_value` | F64 | experiment-wide PSM q (== `q_value` for a single-run rescore) | +| `prelim_score` | F64 | pre-rescore feature-stage score | +| `source` | U32 | run index (0 for single-run; index into `--competed` for experiment-wide) | +| `run_psm_q` | F64 | per-run PSM FDR | +| `experiment_psm_q` | F64 | pooled PSM FDR | +| `precursor_q` | F64 | per (peptidoform+charge) FDR | + +The multi-context q columns and carried peak coordinates are the reasons for the +schema bumps; an older reader that assumes the prior column set will misread +them. `quant.q_filter` +(see [QuantConfig](#quantconfig-quantrs)) selects which of these q columns the +quant stage filters on. + +## How it works + +### Config load path + +`load_config` (`main.rs:376-384`) reads the `--config` file to a string and +calls `Config::from_json` (`config.rs:1009-1014`), which does +`serde_json::from_str` and then `validate()`. With no `--config` it returns +`Config::default()` (`config.rs:988-1005`), which is fully populated from the +per-field defaults. `--profile ` then calls `apply_profile` +(`main.rs:607`, `config.rs:1107-1121`) on top of the loaded config. + +`deny_unknown_fields` on every section (`config.rs:182`, `203`, `251`, ...) means +a typo like `{"digest":{"min_len":7,"bogus":1}}` is a parse error, verified by +`unknown_key_rejected` (`config.rs:1146-1149`). `#[serde(default)]` on every +section and field means a partial JSON overlays defaults: `{"digest":{"min_len":7}}` +keeps `max_len=50`, `charge_max=3`, `top_n_peaks=300` (`config.rs:1151-1159`). +The `t::()` helper (`config.rs:163-168`) is a terse `T::default()` used inside +the per-struct `Default` impls. + +### `validate()` hard-error checks (`config.rs:1019-1100`) + +Known invalid combinations are rejected at load so they never silently produce a +wrong result. This is targeted validation, not proof that every scientifically +poor setting is detectable. Defaults always pass. + +1. `digest.decoy.strategy == DiannShift` -> `Invalid`: the engine digest + produces zero decoys under it, giving an invalid target-decoy FDR + (`config.rs:1021-1029`). +2. `rt_im_train.calibration_method == None` -> `Invalid`: `None` would silently + fall through to the linear fit, so it is rejected and the user must pick + `linear` or `loess` (`config.rs:1030-1036`). +3. `extract.retain_top_peaks == 0` -> `Invalid`: must be `>= 1` (1 = legacy + single apex) (`config.rs:1037-1043`). +4. `extract.min_frag_corr` not finite or outside `[0, 1]` -> `Invalid` + (`config.rs:1044-1052`). 0 disables the gate. Verified by + `explicit_uncapped_seed_and_invalid_gate_are_distinguished` + (`config.rs:1180-1187`). +5. `rescore.folds < 2` -> `Invalid`: every PSM needs an out-of-fold score + (`config.rs:1053-1059`). +6. `rescore.num_iter == 0` -> `Invalid`: iterative model training needs at least + one iteration (`config.rs:1060-1064`). +7. `rescore.train_fdr` non-finite or outside `(0, 1]` -> `Invalid` + (`config.rs:1065-1072`). +8. Mokapot or NnTorch without `rescore.python` -> `Invalid` + (`config.rs:1073-1082`). +9. `classifier = percolator` -> `Invalid`: the adapter is not wired + (`config.rs:1083-1089`). +10. Entrapment without `rescore.entrapment_marker` -> `Invalid` + (`config.rs:1090-1098`). + +`canonical_json` (`config.rs:1125-1127`) serializes the fully-resolved config; +`run` hashes it with `blake3_str` into `config_hash` (`run.rs:89`) and stores the +JSON verbatim in the manifest. There is no separate pretty vs canonical form; it +is a plain `serde_json::to_string`. + +### `apply_profile` (`config.rs:1107-1121`) + +Only `dia` is defined. It sets `features.set = Extended`, +`extract.apex_count_window = 5`, `extract.apex_rt_prior_s = 120.0`. Any other name +is an `Invalid` error. All other extraction defaults stay at their conservative +baselines; the profile is a convenience shortcut, not a full preset file. + +### Mass model math (`mass.rs`, `constants.rs`) + +**Neutral mass** (`mass.rs:66-73`): `WATER + n_term_mod + c_term_mod` plus, per +residue, `residue_mass(r) + mod_delta`. `residue_mass` (`constants.rs:26-51`) is +the 20-standard-amino-acid table; `B, J, O, U, X, Z` return `None` (ambiguous / +non-standard) and cause `MassError::AmbiguousResidue` at parse time. Leucine and +isoleucine share `113.084064015` (`constants.rs:35-36`). `is_standard_residue` +(`constants.rs:54-56`) is the boolean form (`residue_mass(aa).is_some()`), used by +the digest to reject peptides containing a non-standard residue byte. + +**m/z conversion** (`constants.rs:59-62`): +`mass_to_mz(m, z) = (m + z * PROTON) / z`. `precursor_mz` (`mass.rs:76-78`) is +this over the neutral mass. + +**Fragment generation** (`mass.rs:82-127`): a forward prefix scan builds b ions +(`b2..b(n-1)`, dropping `b1`) and a reverse suffix scan builds y ions +(`y3..y(n-1)`, dropping `y1` and `y2`) because b1/y1/y2 are low-information +(`mass.rs:93`, `mass.rs:113`). Each fragment m/z is +`(residue_sum_incl_terminus + z * PROTON) / z` (`mass.rs:97`, `mass.rs:116`). +`Fragment` (`mass.rs:45-53`) carries `ion_type`, `ordinal`, `charge`, `mz`, and a +stable `name` (`b3`, or `y5^2` for charge 2, `frag_name` at `mass.rs:130-136`). +`IonType` (`mass.rs:29-42`) is the `B`/`Y` enum; `IonType::symbol()` returns the +lowercase `'b'`/`'y'` used in the fragment name. These two series are the only +ones the MVP scores (see docs/18_findings_and_decisions.md). + +**ppm predicates.** Three exist and they differ by which mass normalizes the +difference; a maintainer must not treat them as interchangeable. + +- `ppm_diff(observed, theoretical)` (`constants.rs:66-68`): + `1e6 * (observed - theoretical) / theoretical`. **Signed**, normalized by the + **theoretical** mass. Used for reporting a mass error and for + `ppm_match(obs, theo, tol) = |ppm_diff| <= tol` (`constants.rs:72-74`). +- `ppm_bounds(mz, tol)` (`constants.rs:78-81`): returns `(mz - d, mz + d)` with + `d = mz * tol * 1e-6`. **Symmetric window centered on the query** `mz`, + normalized by the query itself. +- `within_ppm(a, b, tol)` (`constants.rs:92-96`): `lo = min(a,b)`, `hi = max(a,b)`, + true iff `hi - lo <= tol * 1e-6 * lo`. **Min-relative**, symmetric in argument + order. This is the canonical fragment-index predicate (fragindex_spec 2.1): + it is algebraically `hi/lo <= 1 + delta` and `ln(hi) - ln(lo) <= ln(1 + delta)`, + the last form being what makes log-space binning exact and is proven exact + against the log-bin +/-1 probe. `within_ppm_three_forms_agree` + (`constants.rs:102-126`) checks the three forms round identically away from the + edge; `within_ppm_edges` (`constants.rs:128-136`) checks edge inclusivity and + argument-order symmetry. + +The three normalize by theoretical (`ppm_diff`), by query center (`ppm_bounds`), +and by the smaller mass (`within_ppm`) respectively, so they disagree at the +tolerance edge. Index probing must use `within_ppm`; do not substitute a +`ppm_bounds` window there. + +**UniMod subset** (`mass.rs:13-26`): `Carbamidomethyl` 57.021463735, +`Oxidation` 15.994914620, `Acetyl` 42.010564684, `Phospho` 79.966331090, +`Deamidated` 0.984016106, `Methyl` 14.015650064, `Dimethyl` 28.031300128, +`Carbamyl` 43.005813726. An unknown name is `MassError::UnknownModification` +(`mass.rs:221`), never a silent zero. + +**ProForma-lite parse** (`parse_peptidoform`, `mass.rs:143-205`): optional +N-terminal `[Mod]-`, residues each optionally followed by `[Mod]`, optional +trailing `-[Mod]`. A `[Mod]` is a UniMod name or a signed float such as +`[+15.9949]` (`parse_bracket`, `mass.rs:209-225`, tries `unimod_mass` first then +`f64` parse). Non-alphabetic characters and ambiguous residues are errors. A +second mod at the same residue position accumulates (`+=`, `mass.rs:191`); the +data model has one `mods[i]` slot per residue plus separate terminal deltas +(`ParsedPeptidoform`, `mass.rs:56-62`). + +## Key types and functions + +| Name | file:line | What it does | +|---|---|---| +| `Config` | config.rs:971-1005 | Top-level config; 10 stage sections + `mbr` + `rng_seed` | +| `Config::from_json` | config.rs:1009-1014 | Parse (deny unknown) then `validate` | +| `Config::validate` | config.rs:1019-1100 | Ten hard-error checks | +| `Config::apply_profile` | config.rs:1107-1121 | `dia` preset only | +| `Config::canonical_json` | config.rs:1125-1127 | Serialize for manifest hashing | +| `residue_mass` | constants.rs:26-51 | 20-AA monoisotopic table; `None` for ambiguous | +| `is_standard_residue` | constants.rs:54-56 | `residue_mass(aa).is_some()` boolean form | +| `mass_to_mz` | constants.rs:59-62 | Neutral mass -> m/z | +| `ppm_diff` / `ppm_match` | constants.rs:66-74 | Theoretical-relative signed ppm + tolerance test | +| `ppm_bounds` | constants.rs:78-81 | Query-centered symmetric m/z window | +| `within_ppm` | constants.rs:92-96 | Min-relative canonical index predicate | +| `parse_peptidoform` | mass.rs:143-205 | ProForma-lite parser | +| `ParsedPeptidoform::fragments` | mass.rs:82-127 | b/y ions, drops b1/y1/y2 | +| `IonType` | mass.rs:29-42 | `B`/`Y` enum; `symbol()` -> `'b'`/`'y'` | +| `unimod_mass` | mass.rs:13-26 | 8-name UniMod subset | +| `Manifest` / `ArtifactRecord` | manifest.rs:9-51 | Run provenance; `new`/`record`/`get` (manifest.rs:33-51) | +| `record_artifact` | mumdia-io/src/lib.rs:20-39 | Build `ArtifactRecord`, hash file | +| `RejectionReason` | rejection.rs:19-113 | Ordered identification-loss ladder | +| `Label` | types.rs:33-51 | Target/Decoy; `.pin()` = +1/-1, `.is_decoy()` | + +### Core data types (`types.rs`) + +Ion mobility is `Option`/nullable throughout so one model serves 3D and 4D runs; +the MVP is 3D so every IM field is `None` (`types.rs:1-3`). + +| Type | file:line | Fields / behavior | +|---|---|---| +| `Peak` | types.rs:9-13 | `mz` f64, `intensity` f32, `ion_mobility` `Option` (`None` for Orbitrap DIA) | +| `IsolationWindow` | types.rs:17-30 | `target_mz`/`lower_mz`/`upper_mz` f64, `im_lower`/`im_upper` `Option`; `covers(mz)` is inclusive m/z containment (`types.rs:26-29`) | +| `Label` | types.rs:33-51 | `Target`/`Decoy` (serde snake_case); `pin()` -> +1/-1 (Percolator), `is_decoy()` | +| `Ms2Scan` | types.rs:54-62 | `scan_index` u32, `id` String, `rt_seconds` f64, `window`, m/z-sorted `peaks` `Vec` | + +### Error types (`error.rs`) + +Both enums derive `thiserror::Error`; the `#[error(...)]` string is the Display +message. Misconfiguration and bad input fail loudly (see +docs/18_findings_and_decisions.md). + +| Variant | file:line | Raised when | +|---|---|---| +| `MassError::Parse(String)` | error.rs:8-9 | peptidoform parse failure: stray `-`, non-alphabetic char, unclosed `[`, or no residues (mass.rs:172, 177, 197, 214) | +| `MassError::AmbiguousResidue(char)` | error.rs:10-11 | a residue with no monoisotopic mass (B/J/O/U/X/Z) at parse (mass.rs:183) | +| `MassError::UnknownModification(String)` | error.rs:12-13 | a `[...]` mod that is neither a UniMod name nor a parseable signed float (mass.rs:221) | +| `ConfigError::Parse(String)` | error.rs:18-19 | `serde_json` failure inside `Config::from_json` (config.rs:1011) | +| `ConfigError::Invalid(String)` | error.rs:20-21 | a `validate()` rejection or unknown `--profile` (config.rs:1022-1098, 1115) | + +### Candidate-audit reason ladder (`rejection.rs`) + +`RejectionReason` is the ordered identification-loss ladder written to the +candidate-audit table (`emit_candidate_audit` / `mumdia audit`). Each candidate's +row records the EARLIEST stage at which it was lost, so the aggregate answers +"where was each DIA-NN-only precursor first lost?" without conflating later +stages. The serialized spelling is `SCREAMING_SNAKE_CASE` and equals +`code()`; `code()` (rejection.rs:50-71) is the stable string written to +Parquet/JSON without a serde round-trip. `stage_order()` (rejection.rs:76-97) is +the ladder position (0 = earliest); `earliest(a, b)` (rejection.rs:106-112) keeps +the smaller `stage_order` and `Reported` never overrides a real rejection; +`is_rejection()` (rejection.rs:100-102) is true for any non-`Reported` variant. + +| `stage_order` | Variant / `code()` | Stage lost at | +|---|---|---| +| 0 | `PeptideNotGenerated` / `PEPTIDE_NOT_GENERATED` | search space (A) | +| 1 | `ModificationNotAllowed` / `MODIFICATION_NOT_ALLOWED` | search space (A) | +| 2 | `ChargeOutOfRange` / `CHARGE_OUT_OF_RANGE` | search space (A) | +| 3 | `PrecursorMzOutOfRange` / `PRECURSOR_MZ_OUT_OF_RANGE` | search space (A) | +| 4 | `NoValidFragments` / `NO_VALID_FRAGMENTS` | search space (A) | +| 5 | `WrongIsolationWindow` / `WRONG_ISOLATION_WINDOW` | search space (A) | +| 6 | `RtPruned` / `RT_PRUNED` | candidate generation (B) | +| 7 | `CandidateCapReached` / `CANDIDATE_CAP_REACHED` | candidate generation (B) | +| 8 | `NoFragmentTraces` / `NO_FRAGMENT_TRACES` | extraction (C, D) | +| 9 | `NoPeakGroup` / `NO_PEAK_GROUP` | extraction (C, D) | +| 10 | `PeakNotSelected` / `PEAK_NOT_SELECTED` | peak/peptide ranking (E) | +| 11 | `OutcompetedByTarget` / `OUTCOMPETED_BY_TARGET` | competition (G) | +| 12 | `OutcompetedByDecoy` / `OUTCOMPETED_BY_DECOY` | competition (G) | +| 13 | `FailedPrecursorFdr` / `FAILED_PRECURSOR_FDR` | FDR + reporting (H) | +| 14 | `FailedPeptideFdr` / `FAILED_PEPTIDE_FDR` | FDR + reporting (H) | +| 15 | `RemovedDuringReporting` / `REMOVED_DURING_REPORTING` | FDR + reporting (H) | +| 255 | `Reported` / `REPORTED` | sentinel: reached the final report (not a loss) | + +## Configuration + +Every section is `#[serde(default, deny_unknown_fields)]`. Below, each field is +listed with its default and effect. Fields marked **stub** or **default-off** are +called out explicitly. + +### Strategy enums (live variants) + +| Enum | file:line | Variants (default in bold) | Notes | +|---|---|---|---| +| `DecoyStrategy` | config.rs:17-28 | **`reverse`**, `scramble`, `diann_shift`, `none` | `diann_shift` rejected by validate; `none` produces no decoys (invalid FDR) | +| `MatcherKind` | config.rs:38-42 | `bucketed`, **`fragindex`** | fragment-matcher backend for search-seed + extract | +| `Enzyme` | config.rs:46-52 | **`trypsin_p`**, `trypsin` | cut after K/R, with/without before-P | +| `CalibrationMethod` | config.rs:56-61 | **`loess`**, `linear`, `none` | `none` rejected by validate (falls through to linear) | +| `FeatureSet` | config.rs:65-75 | **`minimal`** (14), `rich` (44), `extended` (381) | superset battery in `stages/features/`; count asserted by the `feature_sets_sized` test (`features.rs:1590-1598`) | +| `RtPredictorKind` | config.rs:79-86 | **`native`**, `deeplc` | DeepLC is a Python sidecar | +| `FragPredictorKind` | config.rs:90-96 | **`native`**, `ms2pip` | MS2PIP is a Python sidecar | +| `RescorerKind` | config.rs:100-123 | **`native_tda`**, `mokapot`, `nn_torch`, `percolator`, `entrapment` | see RescoreConfig | +| `PeakClaim` | config.rs:132-157 | **`none`**, `winner_predicted_intensity`, `proportional`, `coelution_winner`, `coelution_proportional`, `coelution_winner_margin` | shared-peak apportionment | +| `GateMode` | config.rs:551-574 | **`apex_pearson`**, `peak_spectral`, `spectral_entropy`, `coelution`, `combined` | which spectral score `min_frag_corr` thresholds | +| `UnknownModPolicy` | config.rs:244-248 | **`error`**, `skip` | unknown-mod behavior | +| `CompetitionMode` | config.rs:674-691 | **`winner_take_all`**, `none`, `features_only`, `unique_evidence`, `margin_gated` | within-group resolution | +| `CompeteGroupBy` | config.rs:695-702 | **`precursor`**, `apex`, `peptidoform_charge` | competition grouping key | +| `RollupMethod` | config.rs:706-712 | **`top_n_sum`**, `sum` | protein rollup | +| `PeakWindowMode` | config.rs:717-729 | **`per_candidate`**, `consensus` | quant integration window | +| `NormalizeMethod` | config.rs:737-750 | `none`, **`median_ratio`**, `median` | cross-run LFQ normalization; `from_token` at 754-761 | +| `QuantQColumn` | config.rs:772-789 | **`peptide_q`**, `precursor_q`, `psm_q`, `run_psm_q` | which q column quant filters on | +| `MbrStrategy` | config.rs:845-855 | **`none`**, `empirical_library`, `rt_transfer`, `full` | **stub**: only `none` runs; the rest are config hooks (Stage D3 not wired) | +| `DecoyTransfer` | config.rs:864-869 | **`permuted_rt`**, `reverse_sequence`, `both` | MBR false-transfer null (**stub**, MBR-only) | + +#### Variant semantics (behaviorally-rich enums) + +The table names every variant; the ones below carry distinct behavior a +maintainer must not confuse. Line refs are `config.rs`. + +**`MatcherKind`** (30-42). `Fragindex` (default) is the log-bin CSR matcher: on +narrow-window DIA it is ~1.95x faster in search-seed and ~1.26x in extract with +essentially unchanged IDs (HYE B_01 peptides -0.1%). `Bucketed` is the previous +`Library::page_search` path, retained for A/B comparison and for the AIF +full-range-window case, where the min-relative vs query-relative predicate +difference shifts IDs more. + +**`RescorerKind`** (100-123). `NativeTda` (default): native semi-supervised +linear rescorer + target-decoy q, always available. `Mokapot` (105-106): mokapot +Python sidecar over the PIN. `NnTorch` (107-113): PyTorch semi-supervised MLP +sidecar (`nn_rescore_worker.py`), a nonlinear Percolator/mokapot-style rescorer +with CV folds + iterative positive re-selection; on the E.coli benchmark it beats +linear mokapot on the same PIN and gains further when the extraction gate is +opened; requires `rescore.python` to point at a torch interpreter. `Percolator` +(114-115): external `percolator.exe` over the PIN. `Entrapment` (116-122): treats +foreign-proteome PSMs (marked by `entrapment_marker`) as real negatives, trains a +nonlinear GBM sidecar (out-of-fold by base peptide) or a native linear fallback, +and reports entrapment-calibrated q-values; the chimeric false matches that +in-silico decoys under-model appear as real negatives here. + +**`PeakClaim`** (125-157) apportions one observed MS2 peak that matches the +fragments of several co-isolated, co-eluting candidates (near-universal in +wide-window DIA: ~98% of fragment m/z collide within tolerance), to stop a +chimeric candidate borrowing a real peptide's peak wholesale. `None` (default, +legacy): every claimant gets the full peak intensity. `WinnerPredictedIntensity`: +winner-take-all by highest predicted intensity for the matching fragment. +`Proportional`: split by predicted intensity. `CoelutionWinner` (two-pass): a +first pass builds each candidate's per-scan summed-matched-intensity elution +profile, then the peak goes to the claimant most eluting at that scan (best +corroborated by its OTHER fragments), not the one predicting the brightest ion. +`CoelutionProportional` (two-pass): split by per-scan elution-profile height. +`CoelutionWinnerMargin` (two-pass): winner-take-all only when the top eluter's +profile height dominates the runner-up by `peak_claim_margin`, else the peak +stays shared (as `None`). + +**`GateMode`** (547-574) selects which spectral-agreement score `min_frag_corr` +thresholds, all computed at the gate from data in hand. `ApexPearson` (default, +legacy): Pearson of observed-vs-predicted fragment intensities at the single apex +scan (one chimeric scan can dominate). `PeakSpectral`: Pearson of the +peak-integrated observed spectrum (each fragment summed over the elution-peak +scans) vs predicted; the standard library-dot-product measure. `SpectralEntropy`: +Li spectral-entropy similarity of the sqrt-transformed apex-scan intensities +(`spectral_entropy_similarity_sqrt`); the full-feature gate search found it the +single best gate discriminator (AUC 0.826 / matched-pool recall 69.8% vs apex +Pearson 0.781 / 64.5%). `Coelution`: predicted-intensity-weighted mean co-elution +correlation of each matched fragment's XIC to the signature reference over the +peak (temporal agreement, orthogonal to intensity). `Combined`: require BOTH +peak-integrated Pearson >= `min_frag_corr` AND co-elution >= `gate_coelution_min`. + +**`CompetitionMode`** (674-691). `WinnerTakeAll` (default, legacy): keep only the +top `prelim_score` per group. `None`: keep every candidate (FDR handles +ambiguity). `FeaturesOnly`: same retained set as `None`, with conflict/contested +features carrying the interference signal into rescoring (the name documents +intent for the experiment matrix). `UniqueEvidence`: keep a loser only when +`unique_fragment_count >= unique_evidence_min_fragments`, else winner-take-all +fallback (also falls back to WTA when the feature column is absent). +`MarginGated`: remove a loser only when `winner_score - loser_score >= margin`, +else keep it (conservative removal for the low-FDR region). Target/decoy labels +stay in the competition key in every mode, so a target never competes against its +own decoy. + +**`MbrStrategy`** (838-855) and **`DecoyTransfer`** (857-869) are stubs / config +hooks (Stage D3 not wired; all require >= 2 runs). `MbrStrategy`: `None` (default) +reproduces the chain byte-for-byte; `EmpiricalLibrary` builds the cross-run +consensus anchor library only (M1); `RtTransfer` adds cross-run expected-RT +transfer extraction (M2/M3); `Full` adds requantification (M5). `DecoyTransfer` is +the MBR false-transfer null (M4): `PermutedRt` (default) transfers real precursors +to a decoupled wrong expected RT; `ReverseSequence` transfers reverse/scramble +decoys at the same expected RT; `Both` combines them. + +**`CompeteGroupBy`** (695-702). `Precursor` (default) groups charge/modification +variants of one base peptide, separately within targets and within decoys; +`Apex` additionally buckets by rounded apex RT (`apex_rt_tolerance_s`); +`PeptidoformCharge` keeps each distinct peptidoform+charge as its own group +(precursor-level, as DIA-NN/Spectronaut report), recovering sibling forms the +base-peptide grouping collapses. The label stays in the key in every mode: a +target never directly competes against its paired decoy in the `compete` stage. +This does not remove target-decoy comparison from FDR: `rescore` later selects +best representatives at each q-value unit and estimates the target-decoy null. + +**`PeakWindowMode`** (714-729). `PerCandidate` (default): each candidate's quant +window is anchored at the identified apex when available, with a summed-XIC +fallback for older scored artifacts; its descent bounds can still be stretched by +interference or collapse on sparse peaks. `Consensus`: the median left/right +half-widths of confident peptides applied around each candidate's identified +apex. Consensus widths are estimated independently inside each quant invocation, +not shared across runs. + +**`QuantQColumn`** selects which q column quant filters on. `PeptideQ` (default) +uses `peptide_q_value`. `PrecursorQ` uses `precursor_q` and is valid only for a +single-run rescore; after pooled rescoring that grouped q is experiment-wide. +`PsmQ` uses pooled `q_value`. `RunPsmQ` uses per-source `run_psm_q` and is the +run-local choice after an experiment-wide rescore. Quant has no source selector: +slice the scored table by `source` before pairing it with each run's +chromatograms. Changing the q column does not perform that slice. + +**`RollupMethod`** (706-712): `TopNSum` (default) sums the top-N most abundant +peptides per protein group (`top_n_peptides`); `Sum` sums all group peptides. +**`NormalizeMethod`** (735-761, `quant-lfq` CLI token, not a config field): +`MedianRatio` (default) is a DESeq-style median-of-ratios size factor over +complete-case features, robust to a minority of changing features (does not flatten +a spike-in design's real fold changes); `Median` aligns each run's median intensity +(simpler, less robust to composition shifts); `None` uses raw areas. + +### `DigestConfig` (config.rs:181-200) + +| Field | Default | Effect | +|---|---|---| +| `enzyme` | `trypsin_p` | cleavage rule | +| `missed_cleavages` | 2 | max missed cleavages | +| `min_len` | 5 | min peptide length | +| `max_len` | 50 | max peptide length | +| `decoy.strategy` | `reverse` | decoy scheme (`DecoyConfig`, config.rs:170-179) | + +### `PeptidoformsConfig` (config.rs:202-231) + +| Field | Default | Effect | +|---|---|---| +| `fixed_mods` | `[{C, Carbamidomethyl}]` | applied to every matching residue | +| `variable_mods` | `[{M, Oxidation}]` | optionally applied | +| `max_variable_mods` | 1 | max simultaneous variable mods | +| `charge_min` | 2 | lowest precursor charge | +| `charge_max` | 3 | highest precursor charge | +| `unknown_modification` | `error` | `error` or `skip` | + +`ResidueMod` (config.rs:233-240) is `{residue: char, name: String}` where `name` +is a UniMod name. The doc comment reserves `residue: '*'` for "any" and notes +terminal mods are handled separately in the MVP (config.rs:236). +`deny_unknown_fields` applies but the struct has no `#[serde(default)]` (unlike +every other section), so both keys are required inside a `ResidueMod` entry. + +### `PredictFragConfig` (config.rs:250-282) + +| Field | Default | Effect | +|---|---|---| +| `predictor` | `native` | fragment-intensity source (native heuristic or MS2PIP) | +| `rt_predictor` | `native` | iRT source (native or DeepLC) | +| `charge2_from_precursor_charge` | 2 | add charge-2 fragments for precursor charge >= this | +| `top_n_fragments` | 6 | fragments kept per candidate | +| `ms2pip_model` | `"HCD"` | MS2PIP model name | +| `ms2pip_python` | `None` | interpreter for MS2PIP sidecar | +| `deeplc_python` | `None` | interpreter for DeepLC sidecar | +| `sidecar_script_dir` | `"scripts"` | directory holding worker scripts | + +### `SearchSeedConfig` (config.rs:284-320) + +| Field | Default | Effect | +|---|---|---| +| `fdr_seed` | 0.01 | seed FDR cutoff for calibration anchors | +| `fragment_tol_ppm` | 20.0 | fragment match tolerance | +| `report_psms` | 5 | max PSMs reported per spectrum | +| `min_matched_peaks` | 4 | min matched fragments per seed PSM | +| `top_n_peaks` | 300 | probe only the N most intense MS2 peaks (0 = all) | +| `matcher` | `fragindex` | matcher backend | +| `two_pass_mass_cal` | `false` | **default-off** robust two-pass mass calibration (P3.1) | + +### `RtImTrainConfig` (config.rs:322-387) + +| Field | Default | Effect | +|---|---|---| +| `calibration_method` | `loess` | RT calibration (`none` rejected by validate) | +| `q_train` | 0.01 | q cutoff for calibration anchors | +| `p_rt` | 0.95 | residual percentile for the RT window | +| `rt_window_multiplier` | 1.0 | scales the RT half-window | +| `min_seed_for_calibration` | 50 | min anchors before calibrating | +| `loess_span` | 0.3 | LOESS local-fit fraction | +| `fallback_rt_window_s` | 120.0 | fixed window when calibration cannot fit | +| `finetune_deeplc` | `false` | **default-off** DeepLC multitask fine-tune (nondeterministic; needs `deeplc_python`) | +| `finetune_epochs` | 25 | fine-tune epoch cap (early stopping usually halts earlier) | +| `finetune_patience` | 10 | early-stopping patience | +| `finetune_batch` | 0 | 0 = auto-scale batch to seed size | +| `adaptive_rt_window` | `false` | **default-off** per-region residual window (P3.2/P3.3) | +| `adaptive_rt_bins` | 12 | RT bins for the adaptive window | +| `rt_window_min_s` | 1.0 | lower clamp on any RT half-window | + +### `ExtractConfig` (config.rs:389-545) + +The largest section; the core extraction stage. Cascade thresholds and apex +selection dominate. + +| Field | Default | Effect | +|---|---|---| +| `fixed_scan_window` | 3 | scan-window floor around the apex | +| `frag_tol_ppm` | 20.0 | fragment tolerance | +| `prec_tol_ppm` | 20.0 | precursor tolerance | +| `presence_min_matched` | 3 | tier-(b) min matched fragment count | +| `presence_min_fragments` | 3 | min distinct fragments for acceptance | +| `presence_min_coelution` | 2 | min simultaneously-present fragments over the run | +| `min_frag_corr` | 0.2 | tier-(d) spectral-agreement gate (0 disables; must be in [0,1]) | +| `min_matched_fraction` | 0.0 | tier-(c) min fraction of predicted fragments observed | +| `apex_top_fragments` | 0 | signature-fragment apex: sums the observed intensity of the top-K predicted fragments per scan; `0` falls back to a default of 3 (`extract.rs:1054-1058`), not all-matched | +| `apex_rt_prior_s` | 0.0 | Gaussian RT prior sigma on apex (0 = off) | +| `apex_count_tol` | 1 | fragment-count apex slack | +| `apex_count_window` | 1 | rolling distinct-fragment count width (1 = none; profile `dia` sets 5) | +| `emit_window_grid` | `true` | zero-filled window-grid chromatograms | +| `bucket_size` | 8192 | m/z bucket size (power of two) | +| `peak_claim` | `none` | shared-peak apportionment mode | +| `emit_contested_features` | `false` | **default-off** `contested_frac` feature (forces two-pass) | +| `peak_claim_margin` | 2.0 | dominance factor for `coelution_winner_margin` | +| `matcher` | `fragindex` | matcher backend | +| `min_coelution_run` | 0 | **default-off** min consecutive co-elution scans | +| `ms1_rescue` | `false` | **default-off** MS1-isotope rescue of gate failures | +| `retain_top_peaks` | 1 | K peak groups per candidate (1 = legacy; must be >= 1) | +| `emit_candidate_audit` | `false` | **default-off**; in `run` gates the separate `audit` stage that writes `candidate_audit.parquet` (`run.rs:405-406`); no stage writes `.audit.parquet`; no-op for standalone `mumdia extract` (P0.3) | +| `apex_evidence_rank` | `false` | **default-off** evidence-count apex | +| `emit_gate_diagnostics` | `false` | **default-off** four gate-score columns | +| `gate_mode` | `apex_pearson` | which spectral score `min_frag_corr` thresholds | +| `gate_coelution_min` | 0.5 | second threshold for `gate_mode = combined` | + +The `min_frag_corr` default was relaxed from a historical 0.5 to 0.2 +(config.rs:517-522). Every default-off knob is explicitly documented as +requiring entrapment/target-decoy FDR validation before use. + +### `FeaturesConfig` (config.rs:576-624) + +| Field | Default | Effect | +|---|---|---| +| `set` | `minimal` | feature set (minimal / rich / extended) | +| `coelution_corr_threshold` | 0.9 | co-elution correlation cutoff | +| `prec_tol_ppm` | 20.0 | precursor tolerance for MS1 features | +| `bound_features` | `true` | restrict trace features to the elution peak | +| `bound_peak_fraction` | 1/3 | peak-boundary descent fraction of apex height | +| `bound_peak_grace` | 0 | consecutive sub-threshold scans to bridge | +| `bound_from_confident` | `true` | learn one global peak width from confident seed PSMs | +| `bound_confident_pct` | 50.0 | percentile of confident half-widths as the shared width | + +### `CompeteConfig` (config.rs:626-664) + +| Field | Default | Effect | +|---|---|---| +| `group_by` | `precursor` | competition grouping key | +| `apex_rt_tolerance_s` | 5.0 | RT bucket for `apex` grouping | +| `mode` | `winner_take_all` | within-group resolution | +| `margin` | 0.0 | score margin for `margin_gated` | +| `unique_evidence_min_fragments` | 2 | min unique fragments for `unique_evidence` | +| `emit_competition_audit` | `false` | **default-off** writes `.compete_audit.parquet` | + +### `QuantConfig` (config.rs:791-836) + +| Field | Default | Effect | +|---|---|---| +| `q_threshold` | 0.01 | cutoff applied to the q column selected by `q_filter` | +| `top_n_fragments` | 3 | fragments summed per peptidoform | +| `top_n_peptides` | 3 | peptides summed per protein group (TopNSum) | +| `rollup` | `top_n_sum` | protein rollup method | +| `bound_peak` | `true` | integrate only over the detected peak window | +| `peak_fraction` | 1/6 | descent threshold for the peak-window walk | +| `peak_grace` | 1 | zig-zag grace (bridge N sub-threshold scans) | +| `peak_window_mode` | `per_candidate` | per-candidate vs consensus window | +| `reliable_q` | 0.001 | confident-set q for the consensus width | +| `q_filter` | `peptide_q` | `peptide_q`, `precursor_q` (single-run only), `psm_q`, or `run_psm_q` | + +### `RescoreConfig` (config.rs:913-965) + +| Field | Default | Effect | +|---|---|---| +| `classifier` | `native_tda` | rescorer backend | +| `folds` | 3 | CV folds | +| `train_fdr` | 0.01 | positive-selection FDR | +| `num_iter` | 10 | semi-supervised iterations (native) | +| `python` | `None` | interpreter for a Python rescorer (mokapot/nn_torch/entrapment) | +| `percolator_bin` | `None` | percolator.exe path | +| `entrapment_marker` | `None` | protein substring marking spike-in negatives (required for `entrapment`) | +| `entrapment_exclude` | `None` | substring that de-marks (own-species) | +| `entrapment_contaminant_markers` | `[]` | substrings that keep a spike-in hit as a real target | +| `entrapment_ratio` | 1.0 | N_real_lib / N_entrap_lib scaling | +| `strict` | `true` | fail on a rescorer sidecar failure or unsupported classifier; set false only for explicit compatibility fallback | + +### `MbrConfig` (config.rs:871-911), stub + +Config hooks only; the MBR stage (D3) is not wired into the run chain. Fields: +`strategy` (`none` default), `q_anchor` 0.01, `min_anchor_runs` 2, +`q_transfer` 0.01, `rt_window_s` 20.0, `decoy_transfer` `permuted_rt`, +`consensus_corr_min` 0.0, `requant_all` `false`, `python` `None`. `mbr` is the +one top-level section with an explicit extra `#[serde(default)]` attribute +(config.rs:985-986). With `strategy = none` the chain is byte-identical to no +MBR. + +### Top-level `Config` (config.rs:971-1005) + +`rng_seed` (default 0) plus the ten stage sections and `mbr`. `rng_seed` +seeds every RNG (decoy scramble, CV fold assignment) for determinism. + +### Removed / not present + +The recent cleanup deleted several dead enums that older docs still mention +(`DecoySource`, `ToleranceRegime`, `ScanWindowMode::PeakWidthDerived`, +`FeatureSet::Custom`). They are **not** in the current `config.rs`. Do not +re-add or document them; if you see them referenced elsewhere, that reference is +stale. + +## Invariants, determinism, gotchas + +- **Single source of masses.** No crate outside `mumdia-core` defines a residue + mass, `PROTON`, `WATER`, `AMMONIA`, or `ISOTOPE_SPACING` + (`constants.rs:10-22`). `ISOTOPE_SPACING = 1.003354835` is the 13C-12C mass + difference used for MS1 envelope spacing. `PROTON = 1.007276466812` is the + physically correct proton mass, deliberately not DIA-NN's H-atom value + 1.007825035 (`constants.rs:8-10`). All constants are own-derived from + CODATA/AME; nothing is copied from another engine (clean-room boundary). +- **Three ppm predicates are not interchangeable.** `within_ppm` (min-relative) + is the only one the fragment index may use; `ppm_bounds` (query-centered) and + `ppm_diff`/`ppm_match` (theoretical-relative) disagree at the tolerance edge. + Substituting one for another shifts which fragments match at the boundary. +- **Validation is targeted and fail-loud for known invalid states.** Invalid + decoy/calibration settings, impossible rescorer coverage, missing required + sidecar settings, and invalid numeric bounds are rejected. It is not a general + scientific validator. `DecoyStrategy::None` is still accepted but produces no + decoys and therefore no valid target-decoy FDR; treat it as diagnostic-only. +- **`deny_unknown_fields` everywhere.** A misspelled key fails the whole load, so + a config file cannot silently ignore a setting the author intended. +- **Determinism.** `rng_seed` seeds all randomness. `canonical_json` is a stable + `serde_json::to_string` (field order fixed by struct declaration order), so the + same config hashes to the same `config_hash` across runs. `manifest.artifacts` + and `model_identities` are `BTreeMap`s (`manifest.rs:28-30`), so manifest key + order is deterministic. Note `finetune_deeplc` fine-tuning is itself + nondeterministic (no torch seed) and breaks byte-identical reproducibility when + enabled. +- **Fragment generation drops b1/y1/y2** unconditionally (`mass.rs:93`, + `mass.rs:113`); a peptide shorter than 2 residues yields no fragments + (`mass.rs:85-87`). +- **`ParsedPeptidoform::neutral_mass` uses `.expect`** on residue masses + (`mass.rs:70`), safe only because `parse_peptidoform` already rejected + ambiguous residues. Constructing a `ParsedPeptidoform` by hand with a + non-standard residue byte would panic. +- **Second mod at the same position accumulates** (`mass.rs:191`), which is a + behavior difference from engines that drop it; terminal mods are separate + fields, not part of `mods[i]`. +- **Recent correctness changes bumped four schemas.** `PSMS_COMPETED` and + `PEPTIDE_QUANT`/`PROTEIN_GROUP_QUANT` are v2; `PSMS_SCORED` is v3. Other + registry entries remain v1. Readers must honor the registry rather than assume + one version globally. +- **`content_hash` is the file's blake3, not the logical content.** Any byte + change (compression, column order) changes the hash; it is a change detector, + not a canonical-content identity. + +## How to extend / modify + +- **Add a config field.** Add it to the section struct, give it a value in that + struct's `Default` impl (every field must have one; `#[serde(default)]` is at + the struct level), and document the effect inline. Do not hardcode a choice the + config could express (project convention). If the field can be set to a value + that would silently corrupt results, add a `validate()` check + (`config.rs:1019-1100`) that rejects it with `ConfigError::Invalid`. +- **Add a strategy variant.** Extend the enum, keep `#[serde(rename_all = + "snake_case")]`, and (if it is not implementable yet) reject it in `validate` + the way `DiannShift` and `CalibrationMethod::None` are, rather than letting it + fall through. State default-off status in the doc comment. +- **Add a UniMod modification.** Add a `name => mass` arm to `unimod_mass` + (`mass.rs:13-26`). Use the PSI-MS/UniMod monoisotopic delta so the Python + sidecar adapters map the name. No table is copied from another tool. +- **Add an artifact / bump a schema.** Add or bump the constant in + `schema.rs:7-24`. Bump the version whenever the column set changes + (as `PSMS_SCORED` went to 3). Update the producing stage's `ArtifactReport` + (`schema_name`, `schema_version`) and its `record_artifact` call so the manifest + and the sidecar report agree. +- **Add a manifest field.** Extend `Manifest` or `ArtifactRecord` + (`manifest.rs`); both are plain serde structs. Keep new maps as `BTreeMap` for + deterministic key order. +- **Never edit `plan.md`** (gitignored spec, project rule). Keep validated + numbers consistent across `README`, `COMPARISON.md`, and `CLAUDE.md`. diff --git a/docs/03_io_layer.md b/docs/03_io_layer.md new file mode 100644 index 0000000..d4f5403 --- /dev/null +++ b/docs/03_io_layer.md @@ -0,0 +1,447 @@ +# IO layer: Col/Table, Parquet, report.json, hashing, inspect +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +The `mumdia-io` crate is the on-disk contract layer for the whole engine. Every +stage reads its path-addressable inputs and writes its outputs through this +crate, so no stage hand-rolls Arrow `RecordBatch`es or touches the Parquet +reader/writer directly. The crate provides five things: + +1. A small typed column model (`Col`) and a read-back table (`Table`) over + Arrow + Parquet, so a stage declares its schema as a `Vec` and reads it + back by column name with typed getters (`table.rs`). +2. Parquet write (`write_table`) and read (`Table::read`), SNAPPY-compressed, + the open self-describing interstage format (see the interstage contract in + `docs/18_findings_and_decisions.md`, B1). +3. The per-artifact `.report.json` sidecar (`ArtifactReport` in + `report.rs`) so a stage can be evaluated (row counts, resolved params, key + distributions, model identity, timing) without loading the full table. +4. blake3 content hashing of files and strings (`hash.rs`), which feeds the + `content_hash` on every artifact record and the `config_hash` recorded on + each manifest `ArtifactRecord` (`manifest.rs:19`) and on the manifest header + (`manifest.rs:27`). These hashes are provenance only: nothing in the engine + reads them back to invalidate or skip a downstream artifact, and `run` always + recomputes the full chain. (`report.json` itself carries no `config_hash` + field; see the report fields below.) +5. The `mumdia inspect ` implementation (`inspect` in `lib.rs`): + schema + head sample + row count for any Parquet file. + +This crate reads no `Config` fields of its own. It is a mechanism layer; the +concrete per-artifact column schemas live in each stage's `write_table` call, +and the frozen schema identifiers live in `mumdia-core` (`schema.rs`). The only +"configuration" here is compile-time (SNAPPY compression, a 64 KiB hash buffer, +the schema-id tuples). + +## Files + +| path | role | +|---|---| +| `rust/mumdia/crates/mumdia-io/src/lib.rs` | crate root: `init_logging`, `record_artifact`, `inspect`; re-exports the modules | +| `rust/mumdia/crates/mumdia-io/src/table.rs` | `Col` enum (write side), `write_table`, `Table` (read side) and the typed getters | +| `rust/mumdia/crates/mumdia-io/src/report.rs` | `ArtifactReport` struct + `write_for` (the `.report.json` sidecar) | +| `rust/mumdia/crates/mumdia-io/src/hash.rs` | `blake3_file`, `blake3_str` | +| `rust/mumdia/crates/mumdia-io/src/json.rs` | `write_json`, `read_json` (pretty JSON via serde) | +| `rust/mumdia/crates/mumdia-core/src/schema.rs` | frozen `(logical name, schema version)` tuples for every artifact | +| `rust/mumdia/crates/mumdia-core/src/manifest.rs` | `ArtifactRecord` / `Manifest` (populated from `record_artifact`) | + +## Inputs and outputs + +The IO layer is schema-agnostic: it does not itself consume or produce a fixed +set of named artifacts. It is the read/write mechanism that every stage uses. +What is fixed here is the artifact **identity registry** and the shape of the +two JSON sidecars. + +### Artifact schema registry (`mumdia-core/src/schema.rs:7-24`) + +Each artifact carries a logical schema name and version. The intent is that a +stage could validate its inputs and refuse to apply a model under a mismatched +schema, but no such read-side check is implemented: `schema_version` is only +ever written into the report and the manifest, never read back and compared. +They are `pub const` tuples in the `artifact` submodule, referenced as +`mumdia_core::schema::artifact::PEPTIDES` and so on. The tuples are +`(name, version)`: + +| logical name | version | producing stage | +|---|---|---| +| `spectra_ms1` | 1 | convert | +| `spectra_ms2` | 1 | convert | +| `isolation_windows` | 1 | convert | +| `ms2_to_ms1` | 1 | convert | +| `peptides` | 1 | digest | +| `peptidoforms` | 1 | peptidoforms | +| `fragment_library_precursors` | 1 | predict-frag / library-input | +| `fragment_library_fragments` | 1 | predict-frag / library-input | +| `seed_psms` | 1 | search-seed | +| `run_windows` | 1 | rt-im-train | +| `psms_extracted` | 1 | extract | +| `chromatograms` | 1 | extract | +| `features` | 1 | features | +| `psms_competed` | 2 | compete | +| `psms_scored` | 3 | rescore | +| `peptide_quant` | 2 | quant | +| `protein_group_quant` | 2 | quant | +| `fragment_quant` | 1 | quant | + +There are 18 registered artifacts. Four have been version-bumped past 1: +`psms_competed`, `peptide_quant`, and `protein_group_quant` are at version 2, and +`psms_scored` is at version 3; every other artifact is still version 1. The +version is recorded in the report and the manifest but is not consumed anywhere: +no stage reads a prior artifact's version back, so there is no schema-mismatch +gate on read. A reader could compare the recorded version by hand, but the engine +does not. + +### Example concrete column schemas + +The IO crate stores no schema definitions; a stage's `write_table(path, vec![ +Col::… ])` is the schema. Two examples read from the actual code: + +`isolation_windows` (`stages/convert.rs:215-226`), one row per distinct window: +`window_id: u32`, `target: f64`, `lower: f64`, `upper: f64`. +`ms2_to_ms1` (`stages/convert.rs:228-234`): `ms2_scan_index: u32`, +`ms1_scan_index: i32`. + +`peptides` (`stages/digest.rs:286-298`): `id: u32`, `peptide: utf8`, +`protein: utf8`, `start: i32`, `end: i32`, `label: utf8`, `target_id: i32`, +`decoy_strategy: utf8`. + +To see any artifact's real schema, run `mumdia inspect ` (see +below) rather than trusting a doc; the schema is authoritative on disk. + +### JSON sidecars produced + +- `.report.json` (written for most primary stage Parquets via + `ArtifactReport::write_for`, `report.rs:28`). Coverage is not universal: + schema/PIN files, report TSVs, and several optional or Python-written + artifacts have no report. Fields below. +- `manifest.json` (written once by the `run` orchestrator from the collected + `ArtifactRecord`s, `stages/run.rs`). This crate supplies the per-record + builder `record_artifact` (`lib.rs:20`); it does not write the manifest file. + +## How it works + +### Write side: `Col` -> Arrow -> Parquet + +`Col` (`table.rs:23-42`) is an enum with one variant per supported column type. +Each variant carries `(String name, Vec)`. The scalar variants are +`I64`, `I32`, `U32`, `F64`, `F32`, `Bool`, `Str`; the nullable variants are +`OptF64`, `OptF32`, `OptI32`, `OptStr` (each a `Vec>`); the list +variants are `ListF32`, `ListF64`, and `LargeListF32`. `LargeListF32` +(`table.rs:41`) is encoded as an Arrow `LargeList` with 64-bit offsets, needed +when the total list-value count across all rows can exceed the ~2.1 billion +limit of a 32-bit `ListArray` offset buffer (for example per-fragment +chromatograms when extraction accepts a very large candidate set). The `Opt*` +variants exist to back conditional and ion-mobility columns under the +missing-value policy (`table.rs:5-6`; see `docs/18_findings_and_decisions.md`); +ion-mobility columns are written null throughout the 3D MVP. + +`Col` has four private helpers used by the writer (all non-`pub`): +- `name()` (`table.rs:45`): the column name, via a match over every variant. +- `len()` (`table.rs:64`): the row count of the inner `Vec`. +- `field()` (`table.rs:83`): the Arrow `Field`. Scalar variants are + `nullable = false`; the `Opt*` and all list variants are `nullable = true`. + List inner items are declared `Field::new("item", Float32/64, true)`. +- `into_array()` (`table.rs:107`): the consuming conversion into an `ArrayRef`. + It **moves** the `Vec` into the Arrow array instead of cloning, so the column + data is copied only once during a write. Scalar `Vec` and `Vec>` + go straight through `PrimitiveArray::from`; list variants are built with a + `ListBuilder`/`LargeListBuilder`, appending each row's slice and then + `append(true)` (a present, possibly empty, list). + +`write_table(path, cols)` (`table.rs:151`) is the single write entry point and +returns the row count as `u64`: +1. Reject an empty column set (`table.rs:152`). +2. Reject duplicate column names via a `HashSet` (`table.rs:157-165`). Arrow + allows duplicate names but readers resolve a name to the first match, which + would silently hide the second column, so this is a hard error. +3. Take `nrows` from column 0 and require every column to match + (`table.rs:166-176`); a mismatch is a hard error naming the offending column. +4. Build the `Schema` from `field()` over all columns (`table.rs:177-178`), + then consume the columns into `ArrayRef`s (`table.rs:181`). The `fields` + vector is captured before the consume so the schema still has everything. +5. `RecordBatch::try_new` (`table.rs:182`), create parent dirs + (`create_dir_all(...).ok()`, best-effort, `table.rs:185-187`), create the + file, build `WriterProperties` with `Compression::SNAPPY` (`table.rs:189-191`), + and write a single batch through `ArrowWriter`, then `close()` + (`table.rs:192-194`). One `write_table` call produces exactly one row group / + one logical batch. + +### Read side: Parquet -> `Table` -> typed `Vec` + +`Table` (`table.rs:200-204`) holds the `Arc`, the `Vec`, +and `nrows`. `Table::read(path)` (`table.rs:207`) opens the file, builds a +`ParquetRecordBatchReaderBuilder`, captures the schema, then iterates the reader +collecting every batch and summing `num_rows()`. It errors with context +`"opening {path}"` if the file cannot be opened and `"reading parquet {path}"` if +the builder cannot parse the Parquet footer. The whole file is materialized +into memory; there is no streaming or predicate pushdown. + +Column access is by name. `idx(name)` (`table.rs:235`) resolves a name to a +column index via `schema.index_of`, returning a descriptive error listing all +column names if the name is absent. The typed getters each downcast every +batch's column to the concrete Arrow array type and concatenate across batches +into one `Vec`. They error if the downcast fails, so the type is checked at read +time. The message wording is per getter: `"column '' is not +f64|f32|i64|i32|u32|bool"` for the scalar getters (and `opt_f64` reuses the f64 +message), `"column '' is not utf8"` for `str` (note: `utf8`, not `str`), +and for `list_f32` either `"column '' is not a list"` when the column is +neither a `List` nor a `LargeList` (`table.rs:426`) or `"list '' inner is +not f32"` when the inner element array is not f32 (`table.rs:403`). + +The getters and their exact null behaviour: + +- `f64` (`table.rs:241`), `f32` (`table.rs:261`): fast path when + `null_count() == 0` uses `extend_from_slice(a.values())`; otherwise iterate + and map a null to `f64::NAN` / `f32::NAN`. Nulls become NaN. +- `i64` (`table.rs:281`), `i32` (`table.rs:301`), `u32` (`table.rs:321`): fast + path on no nulls; otherwise iterate pushing `a.value(k)` **without checking + `is_null`**. A null therefore comes through as the underlying buffer value + (typically 0), not as a sentinel. See the gotcha below. +- `bool` (`table.rs:341`): always iterates `a.value(k)`; never checks null. +- `str` (`table.rs:357`): iterates; a null maps to an empty `String`. +- `opt_f64` (`table.rs:377`): the only null-preserving getter. Returns + `Vec>`, mapping a null to `None`. +- `list_f32` (`table.rs:396`): reads an f32 list column and accepts **both** + `List` (32-bit offsets) and `LargeList` (64-bit offsets) encodings, so a + chromatogram artifact written by `Col::ListF32` or `Col::LargeListF32` reads + back through the same call. An outer null list becomes an empty `Vec`; a + present list is materialized via `f.values().to_vec()`. + +`column_names()` (`table.rs:227`) returns the schema field names in order. + +### Hashing (`hash.rs`) + +`blake3_file(path)` (`hash.rs:8`) streams the file in 64 KiB chunks +(`[0u8; 1 << 16]`) through a `blake3::Hasher` and returns the hex digest. This +is the artifact `content_hash`. It is fallible: it returns `Result` and errors +with context `"hashing {path}"` if the file cannot be opened or a read fails. +`blake3_str(s)` (`hash.rs:23`) is a one-shot hex digest of a string, used for the +`config_hash`; it is infallible and returns a plain `String` rather than a +`Result`. The engine derives the +config hash from `Config::canonical_json()` (`config.rs:1125`, a plain +`serde_json::to_string`), for example at `main.rs:417`. The `convert` command is +a deliberate exception: because `--max-spectra`, `--top-peaks-ms2`, and +`--top-peaks-ms1` are CLI caps that change the spectra output but are not part +of `Config`, they are folded into the hash with a unit-separator (`\u{1f}`) +alongside the canonical config JSON so two different caps do not collapse to the +same `config_hash` (`main.rs:402-404`). + +### JSON (`json.rs`) + +Both JSON sidecars (`.report.json` and `manifest.json`) and every JSON +scalar the engine persists go through this module. +`write_json(path, value)` (`json.rs:7`) best-effort-creates the +parent directory (`create_dir_all(...).ok()`, `json.rs:8-9`, same best-effort +policy as `write_table`), serializes with `serde_json::to_string_pretty` +(pretty-printed, human-diffable), and writes the file, erroring with context +`"writing json {path}"` on an I/O failure. +`read_json(path)` (`json.rs:16`) is the counterpart used to +load configs and JSON sidecars back; it errors with context `"reading json +{path}"` if the file cannot be read and `"parsing json {path}"` if +deserialization fails. Serialized key order follows the type's serde field order, +which is why `ArtifactReport.stats` uses a `BTreeMap` (below) to keep key order +deterministic. + +### Report sidecar (`report.rs`) + +`ArtifactReport` (`report.rs:11-24`) is the per-artifact JSON summary; it derives +`Clone`, `Debug`, `Serialize`, `Deserialize` (`report.rs:10`). A stage +constructs it and calls `write_for(artifact_path)` (`report.rs:28`), which +appends `.report.json` to the artifact path and writes it via +`json::write_json` (pretty-printed). Fields: + +| field | type | meaning | +|---|---|---| +| `logical_name` | String | logical artifact name (usually the schema name) | +| `schema_name` | String | schema name from the `schema.rs` tuple | +| `schema_version` | u32 | schema version from the tuple | +| `stage` | String | producing stage, e.g. `"digest"` | +| `rows` | u64 | row count returned by `write_table` | +| `content_hash` | String | `blake3_file` of the artifact just written | +| `params` | `serde_json::Value` | the resolved parameters the stage actually used | +| `stats` | `BTreeMap` | summary key distributions / metrics (ordered) | +| `model_identity` | `Option` | sidecar / predictor identity, else `None` | +| `elapsed_ms` | u128 | wall-clock ms for the stage | + +`stats` is a `BTreeMap` (not a `HashMap`) so the JSON key order is +deterministic. Concrete example: `digest` records `params` with enzyme, missed +cleavages, length bounds, decoy strategy, rng seed, and `max_decoy_attempts`, +and `stats` with `n_targets`, `n_decoys`, `decoy_collision_retries`, and +`dropped_target_decoy_pairs` (`stages/digest.rs:301-331`; the collision-retry and +dropped-pair counters were added with the collision-safe decoy resolver in +e7d7fa5). `convert` has a +stage-local helper `write_reports` (`stages/convert.rs:269-290`) that writes one +report per artifact with a shared `params` and empty `stats`; note this +`write_reports` lives in `convert.rs`, it is not part of the `mumdia-io` public +API. Stages with report coverage build their `ArtifactReport` directly (see the +`write_for` call sites in `align`, `compete`, `extract`, `search_seed`, +`predict_frag`, `peptidoforms`, `rt_im_train`, `rescore`, `features`, `quant`). + +### `record_artifact` and the manifest + +`record_artifact(logical_name, schema, path, rows, stage, config_hash)` +(`lib.rs:20`) builds an `ArtifactRecord` (`manifest.rs:10-20`, in +`mumdia_core::manifest`; derives `Clone`/`Debug`/`Serialize`/`Deserialize`) for +the run manifest. The record has nine fields: `logical_name`, `path`, and `rows` +are copied straight from the arguments; `format` is hard-coded to `"parquet"` +(`lib.rs:31`); `schema_name`/`schema_version` come from the `schema` tuple; +`content_hash` is `blake3_file(path)` of the file just written; `producing_stage` +comes from the `stage` argument (the struct field and the argument are named +differently); and `config_hash` is the argument. The +`run` orchestrator calls it for selected primary Parquet artifacts and records +those into the `Manifest` (`stages/run.rs`, many call sites around +`record_artifact(...)`), which is then serialized to `manifest.json`. +Calibration JSON, PIN/schema companions, TSVs, and some diagnostics are not +manifest records. Standalone single-stage invocations write their normal +sidecars, when implemented, but not a manifest. + +### `inspect` (`lib.rs:43`) + +`inspect(path)` reads the whole table via `Table::read`, then builds a string: +`artifact: `, `rows: `, a `schema:` block listing each field as +` : ` with a ` (nullable)` suffix when the field is nullable, +and a `head:` block. The head is the first up-to-10 rows sliced from the +**first batch only** (`first.slice(0, min(num_rows, 10))`, `lib.rs:57-59`), +formatted with `arrow::util::pretty::pretty_format_batches`. The result is +appended only inside an `if let Ok(p) = ...` (`lib.rs:60`), so if pretty-print +fails the head is silently omitted; there is no `else` branch. The CLI +command `Cmd::Inspect { artifact }` (`main.rs:712`) simply prints the returned +string (`main.rs:713`). + +### Logging + +`init_logging()` (`lib.rs:13`) initializes `tracing_subscriber` once, honouring +`RUST_LOG` and defaulting to `info`, with `with_target(false)`. It uses +`try_init` so a second call is a no-op rather than a panic. + +## Key types and functions + +| name | file:line | what it does | +|---|---|---| +| `Col` (enum) | `table.rs:23` | typed write-side column: scalar, `Opt*`, and list variants | +| `Col::field` | `table.rs:83` | Arrow `Field`; scalars non-nullable, `Opt*`/lists nullable | +| `Col::into_array` | `table.rs:107` | consuming move of the `Vec` into an `ArrayRef` (copy once) | +| `write_table` | `table.rs:151` | validate + write one SNAPPY Parquet batch; returns row count | +| `Table` (struct) | `table.rs:200` | read-back table: schema, batches, nrows | +| `Table::read` | `table.rs:207` | read a Parquet file fully into memory | +| `Table::column_names` | `table.rs:227` | schema field names, in order | +| `Table::f64` / `f32` | `table.rs:241` / `261` | float getters; null -> NaN | +| `Table::i64`/`i32`/`u32` | `table.rs:281`/`301`/`321` | integer getters; null NOT checked (-> buffer value) | +| `Table::bool` | `table.rs:341` | bool getter; null NOT checked | +| `Table::str` | `table.rs:357` | string getter; null -> `""` | +| `Table::opt_f64` | `table.rs:377` | only null-preserving getter; -> `Vec>` | +| `Table::list_f32` | `table.rs:396` | f32 list getter; reads `List` and `LargeList`; null row -> empty `Vec` | +| `ArtifactReport` | `report.rs:11` | per-artifact JSON summary struct | +| `ArtifactReport::write_for` | `report.rs:28` | write `.report.json` | +| `blake3_file` | `hash.rs:8` | streamed blake3 hex digest of a file (`content_hash`) | +| `blake3_str` | `hash.rs:23` | one-shot blake3 hex digest of a string (`config_hash`) | +| `write_json` / `read_json` | `json.rs:7` / `16` | pretty serde JSON write / read, creating parent dirs on write | +| `record_artifact` | `lib.rs:20` | build an `ArtifactRecord` (format hard-coded `"parquet"`) | +| `inspect` | `lib.rs:43` | schema + head(<=10, first batch) + row count as a string | +| `init_logging` | `lib.rs:13` | `tracing` init honouring `RUST_LOG`, default `info` | + +## Configuration + +This subsystem reads no `Config` fields. It has no config surface of its own, so +the recent pruning of dead config fields in `mumdia-core::config` did not touch +it. Its behaviour is fixed at compile time: + +- Compression is `SNAPPY`, hard-coded in `write_table` (`table.rs:189-191`); it + is not configurable and there is no other codec path. +- The hash read buffer is 64 KiB (`hash.rs:11`). +- Schema identifiers are the constants in `mumdia-core/src/schema.rs:7-24`; a + new artifact requires a new tuple there, not a config change. +- Dependency features are pinned in the workspace `Cargo.toml`: `arrow` v59 with + `["prettyprint"]` (needed by `inspect`), `parquet` v59 with + `default-features = false, features = ["arrow", "snap"]` (pure-Rust snap, no + cmake/C toolchain), `blake3` v1. Do not re-enable Parquet default features; + they pull in a C compression backend that breaks the pure-Rust build + constraint (see CLAUDE.md environment gotchas). + +## Invariants, determinism, gotchas + +- **Determinism.** `write_table` writes columns in the order given and a single + batch, so byte layout is stable for stable input. `ArtifactReport.stats` is a + `BTreeMap`, so JSON key order is fixed. `config_hash` comes from + `Config::canonical_json` (serde field order) so it is reproducible. blake3 is + deterministic. All of this is required by the determinism contract (see + `docs/18_findings_and_decisions.md`, B2). +- **Integer/bool null gotcha.** `i64`/`i32`/`u32`/`bool` getters do not check + `is_null`; in the presence of nulls they return the raw buffer value (usually + 0 / false), silently. This is safe today only because the MVP integer/bool + columns are written non-nullable (the `I*`/`Bool` variants set + `nullable = false`). If you write a nullable integer column via `OptI32`, + reading it back with `i32()` will erase the null distinction. Only `opt_f64` + is null-preserving on the read side. This is a known gap (CLAUDE.md + "Correctness": add null-aware getters). +- **Read-side coverage asymmetry.** The write side has `OptF32`, `OptI32`, + `OptStr`, `ListF64` variants with no matching null-aware / typed reader. + `OptF32` reads back via `f32()` (null -> NaN, acceptable), `OptStr` via + `str()` (null -> `""`), `OptI32` via `i32()` (null erased), and there is no + `list_f64` reader at all. Add the corresponding getter before relying on a + round-trip of those variants. +- **Inner-list nulls are not represented.** List item fields are declared + nullable, but the builders only ever `append(true)` present lists and never + append inner-element nulls; `list_f32` reads inner values with + `values().to_vec()` and cannot surface an inner null. Only the outer + list-level null (empty `Vec`) is modeled. +- **Duplicate column names are rejected** at write time (`table.rs:157-165`) + because Arrow readers resolve to the first match and would hide the rest. +- **Length equality is enforced**: all columns must have identical length or + `write_table` errors (`table.rs:166-176`). +- **Everything is loaded into memory.** `Table::read` collects all batches and + `inspect` reads the full table just to print 10 rows; there is no streaming + path. For very large artifacts this is a real memory cost. `inspect`'s head is + taken from the first batch only, so a file with tiny leading batches shows few + rows even when later batches are large. +- **`inspect` swallows pretty-print errors** (`lib.rs:60`, an `if let Ok(p)` + with no `else`): a formatting failure omits the head silently rather than + erroring, so absence of a head block is not proof of an empty table. +- **`create_dir_all` on write is best-effort** (`.ok()`, `table.rs:186`); a real + permission failure surfaces later at `File::create`, not at the mkdir. +- **`record_artifact` hard-codes `format = "parquet"`** (`lib.rs:31`); it is not + suitable for a non-Parquet artifact without a change there. +- **LargeList transparency.** `list_f32` intentionally reads both `List` and + `LargeList`, so an artifact whose encoding differs between two builds (32- vs + 64-bit offsets) still reads identically; do not assume a fixed offset width + when consuming chromatograms. +- **Test coverage is one round-trip.** The crate's only unit test is + `roundtrip_mixed_columns` (`table.rs:438`), which writes then reads back + `U32`/`F64`/`Str`/`OptF64`/`ListF32`/`LargeListF32`, asserting among other + things that a `LargeListF32` column cross-reads through `list_f32`. `hash.rs`, + `json.rs`, `report.rs`, and `lib.rs` (`inspect`, `record_artifact`, + `init_logging`) have no unit tests, and the integer/bool null gotcha and the + `Opt*`/`ListF64` reader gaps above are consequently not exercised by the suite. + +## How to extend / modify + +- **Add a new column type.** Add a variant to `Col` (`table.rs:23`) and extend + the four matches: `name`, `len`, `field`, `into_array`. Decide nullability in + `field`. Then add the matching read-side getter on `Table` following the + downcast + concatenate pattern of the existing getters, and be explicit about + null handling (prefer an `opt_*` return over a silent sentinel for anything + that can be null in practice). +- **Add a null-aware integer/string getter.** This is the standing correctness + item. Mirror `opt_f64` (`table.rs:377`): iterate `is_null(k)` and return + `Vec>`. Do not change the existing non-optional getters' signatures; + add new ones so current call sites are unaffected. +- **Register a new artifact.** Add a `(name, version)` tuple to + `mumdia-core/src/schema.rs` and pass it to `write_table` (schema = the + `Vec` you write) plus the `ArtifactReport`/`record_artifact` calls. Bump + the version only on a breaking column-schema change (as was done for + `psms_scored`, now at version 3, and for `psms_competed` / `peptide_quant` / + `protein_group_quant`, now at version 2) and document the change. There is no + read-side version + check today, so the bump is provenance only; if you need a hard gate, add the + check on the read path (it does not exist yet). +- **Change compression.** It is a one-line change in `write_table` + (`table.rs:190`); keep it to a codec available under the pinned pure-Rust + Parquet features, and re-measure round-trip and file size. +- **Extend the report.** Add a field to `ArtifactReport` (`report.rs:11`). Keep + `stats` a `BTreeMap` for deterministic key order. Since it is + `Serialize`/`Deserialize`, adding a field is backward compatible only if it is + `Option` or has a serde default; otherwise old `.report.json` files fail to + deserialize. +- **Reuse the inspector.** Any tool that needs a schema/head view should call + `mumdia_io::inspect` rather than re-opening Parquet, so the output format stays + consistent with the `mumdia inspect` CLI command. diff --git a/docs/04_convert.md b/docs/04_convert.md new file mode 100644 index 0000000..69ce60a --- /dev/null +++ b/docs/04_convert.md @@ -0,0 +1,348 @@ +# convert (Stage 0): mzML to normalized spectra +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +`convert` is Stage 0 of the pipeline (PLAN.md Stage 0). It is the single point in +the engine that touches the vendor mass-spectrometry format. It reads one mzML +run through the `mzdata` crate and writes a normalized, self-describing spectra +artifact set (four Parquet files) that every downstream stage consumes instead of +the raw file. Everything after this stage (`search-seed`, `rt-im-train`, +`extract`, `features`, `quant`) reads spectra only through +`crates/mumdia/src/spectra.rs`, never through `mzdata`. Consequences: the +vendor-format dependency is isolated here, and any format quirk (profile vs +centroid, AIF/all-ion windows, missing precursor) must be resolved at this stage +because later stages assume the normalized shape. + +The MVP is mzML-only and 3D. Ion-mobility columns are therefore absent from the +artifacts. convert writes no IM columns at all; the in-memory `Peak`/ +`IsolationWindow` types carry `Option` IM fields (`Peak.ion_mobility` at +`crates/mumdia-core/src/types.rs:12`; `IsolationWindow.im_lower`/`im_upper` at +`:21`/`:22`), and the read side (`spectra.rs`) fills them with `None` +(`spectra.rs:74`, `:87-88`). + +## Files + +| Path | Role | +|---|---| +| `rust/mumdia/crates/mumdia/src/stages/convert.rs` | The whole stage: mzML reading, centroiding, peak capping, window synthesis, artifact writing. | +| `rust/mumdia/crates/mumdia/src/main.rs` (`Cmd::Convert`, lines 22-41, 390-414) | CLI subcommand; builds the provenance `config_hash` and calls `convert::run`. | +| `rust/mumdia/crates/mumdia/src/stages/run.rs` (lines 196-203) | The `run` orchestrator's call into convert (top_peaks_ms1 hardcoded to 0). | +| `rust/mumdia/crates/mumdia/src/spectra.rs` | The read-back side: `load_ms1` / `load_ms2` turn the artifacts back into in-memory scans for downstream stages. | +| `rust/mumdia/crates/mumdia-core/src/schema.rs` (lines 7-10) | Frozen `(logical name, version)` identifiers for the four output artifacts. | +| `rust/mumdia/crates/mumdia-io/src/table.rs` | `Col` / `write_table` typed Parquet writer used to emit the artifacts. | +| `rust/mumdia/crates/mumdia-io/src/report.rs` | `ArtifactReport`, the `.report.json` sidecar written per output. | +| `rust/mumdia/crates/mumdia-core/src/types.rs` | `Peak`, `Ms2Scan`, `IsolationWindow` (used on the read-back side). | + +## Inputs and outputs + +Input: one mzML file path (`--mzml`). `mzdata` is pinned at version `0.65` with +`default-features = false, features = ["mzml", "miniz_oxide"]` +(`rust/mumdia/Cargo.toml:26`), so only mzML is compiled in; `miniz_oxide` gives +pure-Rust gzip for compressed mzML. No other input is read. The stage takes no +config file (see Configuration). + +Outputs: four Parquet artifacts written to `--out-dir`, each with a sibling +`.report.json`. All are written with SNAPPY compression via +`write_table` (`table.rs:151`). List columns (`mz`, `intensity`) are Arrow +`List` (the `Col::ListF32` variant); both the outer list column and its +inner `item` element are marked nullable in the Arrow schema (`table.rs:84` +builds the nullable inner `item` field, `table.rs:98` the nullable list column), +but convert writes neither as null. An empty scan is a non-null empty list +(`ListBuilder::append(true)` at `table.rs:124`). + +### `spectra_ms1.parquet` (`SPECTRA_MS1`, schema v1; written at `convert.rs:171`) + +| Column | Type | Meaning | +|---|---|---| +| `scan_index` | `u32` | Global monotonic index over all spectra in the run. | +| `rt_seconds` | `f64` | Retention time in seconds (mzdata minutes x 60). | +| `mz` | `List` | Centroided, m/z-ascending peak m/z (widened to f64 on read). | +| `intensity` | `List` | Peak intensities, aligned to `mz`. | + +### `spectra_ms2.parquet` (`SPECTRA_MS2`, schema v1; written at `convert.rs:198`) + +| Column | Type | Meaning | +|---|---|---| +| `scan_index` | `u32` | Global monotonic index (shares the same counter as MS1). | +| `id` | `Utf8` | Native mzML spectrum id string (`spec.id()`), kept for traceability/USI. | +| `rt_seconds` | `f64` | Retention time in seconds. | +| `window_id` | `u32` | Index into `isolation_windows.parquet`, dedup by (lower, upper). | +| `window_target` | `f64` | Isolation window center m/z (0.0 for AIF/all-ion). | +| `window_lower` | `f64` | Isolation window lower bound m/z. | +| `window_upper` | `f64` | Isolation window upper bound m/z (1.0e6 for AIF/all-ion). | +| `precursor_mz` | `f64?` (nullable) | Selected precursor m/z, or null when absent. | +| `precursor_charge` | `i32?` (nullable) | Precursor charge, or null when absent. | +| `mz` | `List` | Centroided, m/z-ascending fragment m/z. | +| `intensity` | `List` | Fragment intensities, aligned to `mz`. | + +### `isolation_windows.parquet` (`ISOLATION_WINDOWS`, schema v1; written at `convert.rs:215`) + +| Column | Type | Meaning | +|---|---|---| +| `window_id` | `u32` | First-seen id (0-based) of a distinct (lower, upper) window. | +| `target` | `f64` | Window center m/z. | +| `lower` | `f64` | Window lower bound m/z. | +| `upper` | `f64` | Window upper bound m/z. | + +### `ms2_to_ms1.parquet` (`MS2_TO_MS1`, schema v1; written at `convert.rs:228`) + +| Column | Type | Meaning | +|---|---|---| +| `ms2_scan_index` | `u32` | MS2 scan_index. | +| `ms1_scan_index` | `i32` | scan_index of the most recent preceding MS1, or `-1` if none seen yet. | + +The `-1` sentinel (not null) is the "no preceding MS1" marker; it occurs for MS2 +scans acquired before the first MS1 in the run (`convert.rs:160`). + +## How it works + +Control flow is `run` (`convert.rs:103-267`), a single linear pass over the mzML +reader plus four table writes. + +1. **Open the run.** `mzdata::MZReader::open_path(p.mzml)` (`convert.rs:107`) + returns an iterator over spectra in acquisition order. The out directory is + created first (`convert.rs:105`). + +2. **Iterate spectra.** The loop zips the reader with an infinite `(0_u32..)` + range and `.enumerate()` (`convert.rs:120`), so `scan_index` (the range value) + and `count` (the enumerate index) advance together for every spectrum the + reader yields, before the MS-level dispatch. If `max_spectra > 0` and `count` + reached it, break (`convert.rs:121-123`). Because `scan_index` is drawn from the + range regardless of MS level, it is a run-global monotonic counter. Retention + time is `spec.start_time() * 60.0` because mzdata returns start time in minutes + and the artifact stores seconds (`convert.rs:124`). + +3. **Dispatch on MS level** (`convert.rs:125`): + - **MS1** (`convert.rs:126-133`): centroid+cap peaks with `top_peaks_ms1`, + push into the MS1 accumulators, and record `last_ms1_index = scan_index` so + subsequent MS2 scans can point back to it. + - **MS2** (`convert.rs:134-161`): centroid+cap peaks with `top_peaks_ms2`, then + resolve the isolation window and precursor (details below), and append the + `(ms2_scan_index, ms1_scan_index)` mapping row. + - **Any other level** (MS3, etc.): ignored by the `_ => {}` arm + (`convert.rs:162`), but it still consumed a `scan_index`, so the per-level + tables have globally unique but non-contiguous indices. + +4. **Centroiding** happens inside `peaks_of` (`convert.rs:56`, generic over + `SpectrumLike`). It first pulls the raw arrays via `spec.raw_arrays()` -> + `mzs()` (f64) and `intensities()` (f32) (`convert.rs:57-64`). Each access is + `.map(|c| c.to_vec()).unwrap_or_default()`, so a spectrum whose `raw_arrays()` + is `None`, or that is missing either the m/z or intensity array, degrades to + empty vectors and therefore an empty peak list rather than an error. If + `spec.signal_continuity() == SignalContinuity::Profile` + (`convert.rs:65`) it calls `centroid` (`convert.rs:19`); already-centroided + spectra pass through unchanged. `centroid` does simple local-maxima detection + with 3-point parabolic m/z refinement: + - If fewer than 3 samples, return the input as-is (`convert.rs:21-23`). + - Compute a relative noise floor `floor = max_intensity * 1e-4` + (`convert.rs:24-25`), i.e. 0.01% of the base peak. This threshold is + hardcoded, not a config field. + - For each interior sample `i` in `1..n-1` with neighbors `y0, y1, y2` + (`convert.rs:28-34`): keep it only if `y1 > floor` and it is a local maximum + under the asymmetric test `y1 >= y0 && y1 > y2` (left inclusive, right + strict, so a flat-topped pair keeps the left sample once). Otherwise skip. + - Parabolic apex refinement on m/z (`convert.rs:35-43`): with + `denom = y0 - 2*y1 + y2`, the sub-sample offset is + `delta = 0.5 * (y0 - y2) / denom` when `|denom| > 1e-12`, else 0. The local + m/z spacing is `spacing = (mz[i+1] - mz[i-1]) * 0.5`, and the refined center + is `cm = mz[i] + delta * spacing`. Only the m/z is refined; the emitted + intensity is the raw apex sample `y1`, not a parabola-interpolated height + (`convert.rs:44-45`). + - If no local maximum survived, `centroid` returns the original profile arrays + as a fallback (`convert.rs:47-51`). This is a safety net; a pathological + profile scan can therefore leak raw profile samples downstream. + +5. **Filter, cap, sort** (still in `peaks_of`, `convert.rs:70-83`): the m/z and + intensity vectors are joined with `mz.into_iter().zip(inten)` + (`convert.rs:71-73`), which stops at the shorter of the two, so a length + mismatch silently drops the tail of the longer array rather than erroring. + Drop peaks with intensity `<= 0`. If `top_n > 0` and there are more than + `top_n` peaks, + sort descending by intensity and truncate to `top_n` (`convert.rs:76-79`). + Then always sort ascending by m/z (`convert.rs:80`). Finally, cast m/z to + `f32` for output (`*m as f32`, `convert.rs:81`) while intensity stays `f32`. + Storing observed m/z as f32 halves peak storage; the read side widens back to + f64 (`spectra.rs:72`, `spectra.rs:137`). At the ppm tolerances used in DIA + matching, f32 m/z (~7 significant digits) is adequate for observed peaks; + library/theoretical m/z stay f64. + +6. **Isolation window and precursor resolution** (`convert.rs:136-148`): read + `spec.precursor()` and clone its `isolation_window`. If a real window is + present (not both bounds zero), use `(target, lower_bound, upper_bound)` + (`convert.rs:139-141`); mzdata exposes these three window fields as `f32`, and + each is widened with `as f64` before storage, so the stored window columns are + f64 even though the source precision is f32. Otherwise, the AIF / all-ion path + synthesizes a + full-range window `(target=0.0, lower=0.0, upper=1.0e6)` (`convert.rs:142-143`). + This `_` arm fires both when the quadrupole reported a zero-width window + (AIF/all-ion acquisition) and when there is no precursor at all, so any MS2 + with no usable window is treated as covering the entire m/z range. The + downstream `IsolationWindow::covers` (inclusive on both bounds) then returns + true for every fragment (`types.rs:27`). The precursor m/z and charge come from + the first precursor ion + or are `None` (`convert.rs:145-148`). Window synthesis and precursor extraction + are independent code paths: a scan that reports a zero-width window but still + carries a precursor ion gets the synthesized full-range window + (`window_target = 0.0`) together with a non-null `precursor_mz`/ + `precursor_charge`, so `window_target = 0.0` does not imply a null precursor. + +7. **Distinct isolation windows** (`convert.rs:181-196`): a `HashMap` keyed by the + raw bit patterns of `(window_lower, window_upper)` via `f64::to_bits` + (`convert.rs:188`) assigns a first-seen `window_id`. The key uses bits, not + float equality, so identical window bounds always collapse to the same id and + the id order is the acquisition order of first appearance. Each MS2 row records + its `window_id` in `win_id_col`. + +8. **Write the four tables** (`convert.rs:171-234`) with `write_table`, then + `write_reports` (`convert.rs:269-290`) emits one `ArtifactReport` per file: + `logical_name` and `schema_name` both set to the artifact's `schema.0` name + (`convert.rs:276-277`), `schema_version` = `schema.1`, `stage = "convert"`, row + count, a blake3 content hash of the written file (`convert.rs:281`), the same + resolved params for all four (`mzml`, `max_spectra`, `top_peaks_ms2`, + `top_peaks_ms1`, `config_hash`), and `elapsed_ms` (shared across the four + reports, measured once for the whole stage). convert leaves the report's `stats` + empty (`Default::default()`, an empty `BTreeMap`) and `model_identity` `None` + (`convert.rs:283-284`), since it applies no model and computes no summary + distributions. The report is written next to the artifact as + `.report.json` (`report.rs:28-31`). The function returns + `ConvertOutputs` with the four paths for chaining (`convert.rs:261-266`). + +Note the artifacts are written in acquisition order. RT-sorting is deferred to the +read side: `spectra::load_ms2` / `load_ms1` sort by `rt_seconds` after loading +(`spectra.rs:95`, `spectra.rs:158`). + +## Key types and functions + +| Name | file:line | What it does | +|---|---|---| +| `centroid` | `convert.rs:19` | Local-maxima centroiding with parabolic m/z refinement and a relative noise floor. | +| `peaks_of` | `convert.rs:56` | Profile-detect + centroid, drop non-positive intensity, top-N cap, m/z sort, cast m/z to f32. | +| `ConvertParams` | `convert.rs:86` | Inputs: `mzml`, `out_dir`, `max_spectra`, `top_peaks_ms2`, `top_peaks_ms1`, `config_hash`. | +| `ConvertOutputs` | `convert.rs:96` | Returned paths: `ms1`, `ms2`, `isolation_windows`, `ms2_to_ms1`. | +| `run` | `convert.rs:103` | The stage entry point; single pass over mzML, then four table writes. | +| `write_reports` | `convert.rs:269` | Writes the per-artifact `report.json` sidecars. | +| `artifact::SPECTRA_MS1/_MS2/ISOLATION_WINDOWS/MS2_TO_MS1` | `schema.rs:7-10` | Frozen `(name, version)` schema identifiers, all v1. | +| `Col` / `write_table` | `table.rs:23` / `table.rs:151` | Typed columns and the SNAPPY Parquet writer; validates equal lengths and rejects duplicate names. | +| `ArtifactReport` | `report.rs:11` | The report struct written next to each artifact. | +| `load_ms2` / `load_ms1` | `spectra.rs:20` / `spectra.rs:100` | Read-back into `Ms2Scan` / `Ms1Scan`, RT-sorted, m/z widened to f64; per-scan peak count is `mf.len().min(iff.len())` (`spectra.rs:68`), tolerant of an m/z vs intensity length mismatch. | +| `Ms1Scan` / `Ms2Scan` | `spectra.rs:12` / `types.rs:55` | Read-back structs. `Ms1Scan` (scan_index, rt_seconds, mz, intensity) is defined in `spectra.rs`, not `types.rs`; `Ms2Scan` (adds `id`, `window`, `peaks`) is in `types.rs`. | + +## Configuration + +convert reads no `Config` fields, and its subcommand has no `--config` flag +(`main.rs:22-41`). The stage function signature does not take a `Config`; the +`config_hash` it receives is only recorded for provenance. The CLI wrapper still +loads the default config via `load_config(&None)` (`main.rs:397`) purely to seed +that hash: `config_hash = blake3(cfg.canonical_json() + separators + caps)` +(`main.rs:402-405`), so the default config's canonical JSON is embedded in the +hash even though no config field alters the output. + +| CLI flag | Default | Effect | +|---|---|---| +| `--mzml` | (required) | Input mzML path. | +| `--out-dir` | (required) | Output directory for the four artifacts. | +| `--max-spectra` | `0` (all) | Read at most N spectra, for fast iteration. Counts all MS levels. | +| `--top-peaks-ms2` | `0` (uncapped) | Keep at most N most-intense MS2 peaks per scan. Irreversible conversion-time cap. | +| `--top-peaks-ms1` | `0` (uncapped) | Keep at most N most-intense MS1 peaks per scan. Irreversible. | + +Defaults are asserted by `conversion_caps_default_to_uncapped` (`main.rs:750`) and +the explicit-cap case by `explicit_conversion_cap_is_preserved` (`main.rs:790`). +The `run` orchestrator exposes `--max-spectra` and `--top-peaks-ms2` but not +`--top-peaks-ms1`; it hardcodes `top_peaks_ms1: 0` when calling convert +(`run.rs:201`). The MS2 cap is documented as "irreversible" because it discards +peaks before they reach extraction, features, and quantification; for a seed-only +peak limit use `search_seed.top_n_peaks` instead (`main.rs:31-35`, +`config.rs:298`). + +Provenance handling of the caps: because the caps change the spectra output but +are not part of the `Config`, `main.rs:402-405` folds them into the blake3 +`config_hash` with a unit-separator (`\u{1f}`) so two different caps do not +collapse to an identical hash. The same values are recorded in the convert report +`params` (`convert.rs:245-251`). + +The centroid noise floor (`1e-4` relative, `convert.rs:25`) and the full-range AIF +window value (`1.0e6`, `convert.rs:143`) are hardcoded constants, not config +fields. There is no `ScanWindowMode` or centroiding strategy knob for this stage; +the project config was recently pruned of dead fields, and convert exposes no +strategy enum. Any change to centroiding or window synthesis is a code change +here, not a config toggle. + +## Invariants, determinism, gotchas + +- **Determinism.** The output is a deterministic function of the input file and + the CLI caps. `scan_index` is assigned in reader order; `window_id` is assigned + in first-appearance order via a bit-keyed map, not float equality. The + intensity-descending truncation sort (`convert.rs:77`) uses Rust's stable + `sort_by`, so ties preserve the incoming (m/z-ascending) order; the final + m/z-ascending sort (`convert.rs:80`) fixes output order regardless. +- **`scan_index` is run-global, not per-level.** MS3 and other unhandled levels + still advance the range-derived counter (`convert.rs:120`), so the MS1 and MS2 tables + have unique but non-contiguous indices. Do not assume contiguity; use + `ms2_to_ms1.parquet` to relate MS2 to its parent MS1. +- **AIF and no-precursor collapse to the same full-range window.** The `_` arm at + `convert.rs:143` handles both a zero-width reported window (AIF/all-ion) and a + missing precursor. If a future non-AIF format reports a genuinely absent window, + it will be silently treated as full-range. `window_target = 0.0` is the marker + for a synthesized window. +- **Observed m/z is f32 on disk.** `convert.rs:81` casts to f32; `spectra.rs` + widens back to f64. This is intentional (storage) and fine at DIA ppm + tolerances, but do not round-trip observed m/z through convert expecting f64 + precision. +- **Intensity is the raw apex sample.** Parabolic refinement adjusts m/z only; the + reported intensity is `y1` (`convert.rs:45`), not an interpolated peak height. +- **Centroid fallback can leak profile samples.** If no local maximum clears the + floor, `centroid` returns the original profile arrays (`convert.rs:47-51`). + Rare, but a downstream stage could then see profile-shaped data for that scan. +- **List columns are nullable in the Arrow schema but never null in practice.** + An empty scan is written as a non-null empty list; the read side treats null and + empty identically (`spectra.rs:55`, `table.rs:419`). +- **`partial_cmp().unwrap()` on sorts** (`convert.rs:77`, `:80`, + `spectra.rs` sorts) would panic on NaN. Convert filters intensity `<= 0` before + sorting and does not sort on m/z NaN in practice, so this is safe for real mzML + but is a latent trap if malformed data ever reaches it. +- **Out-dir creation errors are swallowed.** `std::fs::create_dir_all(p.out_dir) + .ok()` (`convert.rs:105`) discards a creation failure; a genuinely unwritable + out-dir does not fail here but surfaces later as a file-create error from + `write_table` (`table.rs:188`). +- **Observability.** The stage is otherwise side-effect-free apart from its file + writes; it emits two `tracing::info!` records, one on open + (`convert.rs:106`) and one on completion carrying the MS1/MS2/window counts and + `elapsed_ms` (`convert.rs:254-260`). +- **`elapsed_ms` is shared, not per-artifact.** A single `Instant` started at + `convert.rs:104` times the whole stage; the same value is written into all four + `report.json` sidecars, so per-artifact timing cannot be read from them. +- **Test coverage.** Only the two CLI-parsing tests above exercise this area. The + centroiding math, window synthesis, and artifact writing have no stage-level + unit test (see CLAUDE.md "test gaps"). MS1 extraction and mass-calibration paths + that depend on convert output are exercised only in full runs. + +## How to extend / modify + +- **Add a vendor format** (Thermo `.raw`, Bruker `.d`/TDF): this stage is the only + place to touch. Either extend `mzdata` features or add a reader that yields the + same per-spectrum interface, and keep the four output schemas byte-compatible so + no downstream stage changes. Convert must stay the sole vendor-format touch + point. +- **Ion mobility / 4D (diaPASEF).** The artifact schemas here are 3D. Adding IM + means new nullable columns on `spectra_ms2` (and the isolation-window IM bounds + already modeled as `Option` in `types.rs:21`), plus populating `Peak.ion_mobility` + on the read side. Bump the affected schema versions in `schema.rs` when columns + change, since the version guards downstream model/schema matching. +- **Change centroiding.** Edit `centroid` (`convert.rs:19`). If the choice should + be user-selectable, add a config field and strategy enum in `mumdia-core` + (per the project convention that every algorithmic choice is a typed config + field) rather than a second hardcoded branch, and thread it through + `ConvertParams`. Remember the noise floor and parabolic step are currently + hardcoded. +- **Change the AIF window sentinel.** The full-range bound `1.0e6` + (`convert.rs:143`) and `window_target = 0.0` marker are relied on by extraction's + `IsolationWindow::covers`. Changing either requires auditing the extract stage. +- **Add an artifact column.** Add the column to the relevant `write_table` call, + add a matching getter/reader in `spectra.rs`, and bump the schema version in + `schema.rs`. `write_table` rejects duplicate names and mismatched lengths, so + every new column vector must match the row count. +- **Preserve provenance semantics.** Any new conversion-time parameter that + changes the output but is not part of `Config` must be folded into the + `config_hash` key in `main.rs` (as the caps are), or two different settings will + produce artifacts with an identical hash. diff --git a/docs/05_digest_peptidoforms.md b/docs/05_digest_peptidoforms.md new file mode 100644 index 0000000..8a03d55 --- /dev/null +++ b/docs/05_digest_peptidoforms.md @@ -0,0 +1,428 @@ +# digest (Stage A) and peptidoforms (Stage A2) + +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +These two stages build the run-independent, experiment-wide peptide search space +from a FASTA file, before any spectra are touched. They run once per experiment +and their outputs are reused across all runs. + +- **digest** (Stage A, PLAN.md) performs a fully-tryptic in-silico digest of the + input proteins, deduplicates the resulting peptides by stripped sequence, and + mints a collision-checked, paired target-decoy null (reverse or seeded + scramble). Output is one `peptides.parquet` table of stripped target and decoy + sequences with a `target`/`decoy` label. +- **peptidoforms** (Stage A2) expands each stripped peptide into concrete + peptidoforms by enumerating fixed and variable modifications and precursor + charge states, emitting each as a ProForma-lite string with UniMod modification + names. Output is one `peptidoforms.parquet` table. + +Neither stage reads spectra or config-run parameters (tolerances, RT, etc.); they +depend only on the FASTA and the `digest` / `peptidoforms` config sections. In the +library-input path (`--lib-precursors`/`--lib-fragments`) both stages are skipped +entirely; the imported library already carries sequences, modifications, charges, +and decoys (see `run.rs` library-input branch, and CLAUDE.md "DIA-NN library +recipe"). + +## Files + +| Path | Role | +|---|---| +| `rust/mumdia/crates/mumdia/src/stages/digest.rs` | Stage A: FASTA parse, tryptic digest, dedup, decoy generation, `peptides.parquet` writer | +| `rust/mumdia/crates/mumdia/src/stages/peptidoforms.rs` | Stage A2: fixed/variable mod + charge enumeration, ProForma-lite emission, `peptidoforms.parquet` writer | +| `rust/mumdia/crates/mumdia-core/src/config.rs` | `DigestConfig`, `DecoyConfig`, `PeptidoformsConfig`, `ResidueMod`, `Enzyme`, `DecoyStrategy`, `UnknownModPolicy`, and `Config::validate` | +| `rust/mumdia/crates/mumdia-core/src/constants.rs` | `is_standard_residue` / `residue_mass` (the 20-residue allowlist that gates the digest) | +| `rust/mumdia/crates/mumdia-core/src/mass.rs` | `unimod_mass` (the UniMod name allowlist that validates modification names) | +| `rust/mumdia/crates/mumdia-core/src/schema.rs` | `artifact::PEPTIDES` and `artifact::PEPTIDOFORMS` logical names + schema versions | +| `rust/mumdia/crates/mumdia/src/main.rs` | CLI wiring: `Cmd::Digest` (main.rs:415), `Cmd::Peptidoforms` (main.rs:426) | +| `rust/mumdia/crates/mumdia/src/stages/run.rs` | Orchestrator wiring in the FASTA-build branch (run.rs:132, run.rs:149) | + +## Inputs and outputs + +**digest** +- Consumes: a FASTA file path (`DigestParams.fasta`, digest.rs:177). Parsed by + `read_fasta` (digest.rs:25) into `(accession, sequence)` pairs. FASTA parsing is + minimal: `>` starts a record, the accession is the first whitespace-delimited + token after `>`, and sequence lines are concatenated, trimmed, and upper-cased + (`to_ascii_uppercase`, digest.rs:41). Lowercase residues are therefore + normalized rather than silently discarded by the later allowlist check. There is + still no special handling of `*` stop characters; they simply fail the + standard-residue allowlist during digest. An unreadable FASTA path is the one + error case in the parser (`with_context("reading fasta {path}")`, digest.rs:26); + a file with no `>` records is not an error, it simply yields zero proteins and + an empty (but valid) `peptides.parquet`. +- Produces: `peptides.parquet` (logical name `peptides`, schema version 1, from + `artifact::PEPTIDES`, schema.rs:11) plus a sibling `.report.json` + (`ArtifactReport`, digest.rs:312). Report `stats` carries `n_targets`, + `n_decoys`, `decoy_collision_retries`, and `dropped_target_decoy_pairs` + (digest.rs:302-311); `params` records enzyme, missed cleavages, length bounds, + decoy strategy, `rng_seed`, and `max_decoy_attempts` (digest.rs:319-326). + +`peptides.parquet` column schema (written at digest.rs:288-297): + +| Column | Type | Meaning | +|---|---|---| +| `id` | u32 | Monotonic row id, assigned interleaved (target then its decoy) | +| `peptide` | str | Stripped sequence (target sequence, or the rewritten decoy sequence) | +| `protein` | str | `;`-joined accessions for targets; `DECOY_` + the same join for decoys | +| `start` | i32 | 0-based start offset of the peptide in the protein of first occurrence | +| `end` | i32 | 0-based end offset (exclusive) of that first occurrence | +| `label` | str | Literal `"target"` or `"decoy"` | +| `target_id` | i32 | `-1` for a target; for a decoy, the `id` of its paired target | +| `decoy_strategy` | str | Lowercased strategy name (e.g. `reverse`, `scramble`) | + +**peptidoforms** +- Consumes: `peptides.parquet` (`PeptidoformsParams.peptides`, peptidoforms.rs:180), + read via `Table::read` (peptidoforms.rs:189). It reads columns `id`, `peptide`, + `protein`, `label`, `target_id`. +- Produces: `peptidoforms.parquet` (logical name `peptidoforms`, schema version 1, + from `artifact::PEPTIDOFORMS`, schema.rs:12) plus `.report.json`. Report + `params` record fixed/variable mods, `max_variable_mods`, and the charge range + (peptidoforms.rs:275); `stats` is empty. + +`peptidoforms.parquet` column schema (written at peptidoforms.rs:255-264): + +| Column | Type | Meaning | +|---|---|---| +| `id` | u32 | Monotonic peptidoform row id (fresh counter across all output rows) | +| `peptide_id` | u32 | The digest `id` of the row this peptidoform came from | +| `base_peptide_id` | u32 | The digest `id` of the underlying **target** peptide (for decoys, the paired target id; for targets, the own id) | +| `peptide` | str | Stripped sequence, copied through unchanged | +| `peptidoform` | str | ProForma-lite string with UniMod names in brackets, e.g. `PEPC[Carbamidomethyl]M[Oxidation]K` | +| `charge` | i32 | Precursor charge (one row per charge in `[charge_min, charge_max]`) | +| `label` | str | `"target"` / `"decoy"`, copied from the digest row | +| `protein` | str | Copied from the digest row | + +## How it works + +### digest (digest.rs:184 `run`) + +1. **Parse FASTA** with `read_fasta` (digest.rs:25). +2. **Per-protein cleavage** in `cleavage_sites` (digest.rs:52). Sites is seeded + with `0`, then for each residue that is `K` or `R` a cut index `i+1` is pushed; + Trypsin/P (`Enzyme::TrypsinP`, the default) always cuts, classic `Enzyme::Trypsin` + suppresses the cut when the next residue is `P` (`before_p`, digest.rs:57). The + sequence length is appended as a final site if not already present + (digest.rs:67). Sites are strictly increasing, so peptides are contiguous + half-open `[start, end)` spans. +3. **Peptide enumeration** in `digest_protein` (digest.rs:75). For every ordered + pair of sites `(i, j)` with `i < j`, the number of missed cleavages is + `j - i - 1`; the inner loop `break`s once that exceeds `cfg.missed_cleavages` + (digest.rs:81), so at most `missed_cleavages + 1` fragments are joined. Length + bounds `min_len`/`max_len` filter on residue count (digest.rs:86). A peptide + is dropped entirely if any residue is not one of the 20 standard amino acids + (`is_standard_residue`, digest.rs:90 -> constants.rs:54), which is how `X`, `B`, + `Z`, `U`, and `*` get excluded (lowercase is already upper-cased in + `read_fasta`, so it is not dropped here). +4. **Dedup by stripped sequence** (digest.rs:189-209). Three structures: a + `HashMap>` mapping peptide to accessions, a `HashMap` from + peptide to its first `(start, end)`, and an insertion-order `Vec` + named `order`. On a repeat occurrence only the accession list is extended (and + only if the accession is not already present, digest.rs:200); the first + occurrence records position and pushes the peptide into `order`. Iteration for + output is over `order`, so **insertion order is the emission order** and is + deterministic (a protein's peptides in sequence order, proteins in FASTA + order). The borrow-then-clone pattern (digest.rs:199) is a deliberate + allocation optimization, not a semantic detail. +5. **Row assembly and decoy minting** (digest.rs:211-275). A single monotonic + `next_id` counter assigns ids. The decoy is resolved **before** the target row + is written: for each target peptide in `order`, if the strategy is `Reverse` or + `Scramble` (digest.rs:224-227), call `collision_safe_decoy` (digest.rs:228). If + it returns `None` (no collision-free permutation within `MAX_DECOY_ATTEMPTS`), + the whole pair is dropped, the target row is not written, and + `dropped_target_decoy_pairs` is incremented (digest.rs:240-243). Otherwise emit + the target row (`label = "target"`, `target_id = -1`, digest.rs:258-259) and + then the decoy row immediately after with a fresh id, `label = "decoy"`, + `target_id = tid` (the target's id), and the `DECOY_`-prefixed protein string + (digest.rs:263-274). Ids therefore interleave target/decoy in pairs. A + `used_decoys` set (digest.rs:216) tracks emitted decoy sequences for + cross-decoy uniqueness, and `decoy_collision_retries` accumulates the retry + count reported in `stats`. +6. **Write** the eight columns with `write_table` (digest.rs:286) and emit the + `ArtifactReport` (digest.rs:312). + +**Decoy generation.** `collision_safe_decoy` (digest.rs:137) is the wrapper the +run loop calls; it drives the lower-level transform `make_decoy` (digest.rs:101) +and guarantees the decoy differs from its target, matches no target sequence, and +is unique among emitted decoys. The configured transform is tried first +(`attempt == 0`); on any collision it retries with an independently reseeded +`Scramble` up to `MAX_DECOY_ATTEMPTS` (64, digest.rs:22), returning the sequence +and the retry count (digest.rs:144-156). If every attempt collides, it returns +`None` and the pair is dropped. Tests `collision_safe_decoy_avoids_targets_and_other_decoys` +(digest.rs:382) and `impossible_low_complexity_decoy_drops_pair` (digest.rs:403) +pin the retry and drop behavior. + +`make_decoy` (digest.rs:101) itself performs one transform. Peptides shorter than +3 residues return `None` (digest.rs:104); through `collision_safe_decoy`'s `?` +this drops the pair rather than emitting an unpaired target. Both realized +strategies keep the C-terminal residue fixed and rewrite the interior `b[..n-1]`: +- `DecoyStrategy::Reverse` (digest.rs:108): reverse the first `n-1` residues, then + re-append the original last residue. Reversing all-but-last keeps the enzyme's + C-terminal `K`/`R` in place while moving the N-terminus. Test + `reverse_decoy_keeps_cterm` (digest.rs:367) pins `PEPTIDER -> EDITPEPR`. +- `DecoyStrategy::Scramble` (digest.rs:114): a deterministic Fisher-Yates shuffle + of the first `n-1` residues. The PRNG state is seeded per peptide as + `rng_seed ^ fnv1a(pep)` (digest.rs:117) and advanced with `splitmix64` + (digest.rs:159). The loop runs `i` from `len-2` down to `1` and swaps index `i` + with a uniform `j` in `0..=i` (digest.rs:118-122), so every interior position + including index 0 (the N-terminal residue) can move while the last residue is + re-appended untouched. Test `scramble_is_deterministic` (digest.rs:374) pins + reproducibility and the fixed C-terminus. `collision_safe_decoy` also mixes an + `attempt`-indexed constant into this seed (digest.rs:145) so each retry is a + distinct but deterministic scramble. +- `DecoyStrategy::DiannShift` and `DecoyStrategy::None` return `None` + (digest.rs:126-127); `DiannShift` is unrealized here by design (the comment + notes it would be a predict-frag fragment-shift, PLAN.md Section 11) and is + additionally rejected by `Config::validate` (config.rs:1021) because it would + yield zero decoys and an invalid FDR. + +### peptidoforms (peptidoforms.rs:186 `run`) + +1. **Validate rules up front** (`validated_rules`, peptidoforms.rs:125, called at + peptidoforms.rs:188 before the table is read). This performs all config + validation once, so a bad config fails fast: + - charge range: `charge_min >= 1` and `charge_min <= charge_max`, else a hard + `anyhow::bail!` (peptidoforms.rs:126-138); + - fixed and variable mod names are resolved by `known_mods` (peptidoforms.rs:85), + which rejects a `*` residue (wildcard/terminal mods are unimplemented, + peptidoforms.rs:92-98) and looks each name up in `unimod_mass` (mass.rs:13, + peptidoforms.rs:99). An unknown name is handled per `unknown_modification`: + `Error` bails with `unknown modification '' in config` + (peptidoforms.rs:101-103); `Skip` warns and drops the rule + (peptidoforms.rs:104-112); + - stacked fixed mods on one residue are rejected (peptidoforms.rs:145-153); + - exact-duplicate variable rules are deduplicated with a warning + (peptidoforms.rs:156-166); + - a residue carrying both a fixed and a variable mod is rejected + (fixed-variable stacking, peptidoforms.rs:167-172). + + The `unimod_mass` allowlist (mass.rs:14-24) currently recognizes exactly eight + names: `Carbamidomethyl`, `Oxidation`, `Acetyl`, `Phospho`, `Deamidated`, + `Methyl`, `Dimethyl`, `Carbamyl`. Extending the set is a `mass.rs` edit (see + "How to extend"). +2. **Read** the digest table (peptidoforms.rs:189-194): columns `id`, `peptide`, + `protein`, `label`, `target_id`. +3. **Per digest row** (peptidoforms.rs:201): compute `base_peptide_id` as + `target_id` when it is `>= 0` (the row is a decoy) else the row's own `id` + (peptidoforms.rs:203-207). +4. **Fixed mods** (peptidoforms.rs:210-217): for each residue index, if any + validated fixed rule's `residue` char matches, record `(index, name)`. Fixed + mods apply to every matching residue and are always present. +5. **Variable-mod candidate sites** (peptidoforms.rs:219-228): the same + residue-char scan collects, per modified site, a `Vec` of the alternative mod + names configured for that residue (`var_sites: Vec<(usize, Vec<&str>)>`). A + single residue can therefore offer more than one variable alternative. +6. **Combination enumeration** (`variable_combos`, peptidoforms.rs:58): returns the + empty subset plus every size-`k` subset of candidate **sites** for + `k = 1..=max_variable_mods.min(n)`, using an in-place combinatorial index + advance (peptidoforms.rs:69-79); for each selected set, `expand_site_choices` + (peptidoforms.rs:34) enumerates every combination of the per-site alternatives, + picking at most one alternative per site so two mods never stack on one residue. + When each site has a single alternative the count is `1 + sum_{k=1..min(max,n)} + C(n,k)`; test `combos_bounded` (peptidoforms.rs:302) pins `n=3, max=1 -> 4` and + `max=2 -> 7`, and `same_site_alternatives_never_stack_and_are_deterministic` + (peptidoforms.rs:329) pins the multi-alternative case. +7. **Emit** (peptidoforms.rs:231-250): for each combo, merge fixed mods and the + combo, sort by position (peptidoforms.rs:234), build the ProForma-lite string + with `proforma` (peptidoforms.rs:19), and emit one output row per charge in + `[charge_min, charge_max]` (peptidoforms.rs:239). A `seen_forms` set + (peptidoforms.rs:230, 236) skips any duplicate ProForma string within a + peptide. Each row gets a fresh `id`, the source `peptide_id`, the computed + `base_peptide_id`, and the copied `label`/`protein`. + +`proforma` (peptidoforms.rs:19) walks the stripped sequence and, at each residue +index, appends `[]` for the **first** mod found at that position +(`mods.iter().find`, peptidoforms.rs:23). Because validation now forbids two mods +on one residue (see gotchas), this first-match is a defensive fallback rather than +a live silent-drop path. + +## Key types and functions + +| Name | file:line | What it does | +|---|---|---| +| `digest::run` | digest.rs:184 | Stage A entry point; parse, digest, dedup, mint decoys, write `peptides.parquet` | +| `read_fasta` | digest.rs:25 | Minimal FASTA parser to `(accession, sequence)` pairs; upper-cases sequence | +| `cleavage_sites` | digest.rs:52 | Trypsin/P vs Trypsin cut-site indices for one sequence | +| `digest_protein` | digest.rs:75 | Enumerate length-bounded, missed-cleavage-bounded, standard-residue-only peptides | +| `collision_safe_decoy` | digest.rs:137 | Wrap `make_decoy` with target/other-decoy collision checks and up to `MAX_DECOY_ATTEMPTS` reseeded-scramble retries; `None` drops the pair | +| `make_decoy` | digest.rs:101 | One transform: reverse or seeded-scramble decoy of the interior, C-term fixed; `None` if `len < 3` or strategy `None`/`DiannShift` | +| `splitmix64` | digest.rs:159 | Deterministic PRNG step for the scramble shuffle | +| `fnv1a` | digest.rs:167 | Per-peptide hash mixed into the scramble seed | +| `DigestParams` | digest.rs:176 | `fasta`, `out`, `cfg: &DigestConfig`, `rng_seed`, `config_hash` | +| `peptidoforms::run` | peptidoforms.rs:186 | Stage A2 entry point; validate rules, enumerate mods+charges, write `peptidoforms.parquet` | +| `validated_rules` | peptidoforms.rs:125 | Validate charges and fixed/variable rules (stacking, overlap, dedup, unknown-mod policy) before expansion | +| `known_mods` | peptidoforms.rs:85 | Resolve one mod list against `unimod_mass` under `UnknownModPolicy`; reject `*` residue | +| `proforma` | peptidoforms.rs:19 | Build ProForma-lite string from stripped peptide + `(position, name)` mods | +| `variable_combos` | peptidoforms.rs:58 | Enumerate modified-site subsets up to `max_var`, expanding per-site alternatives | +| `expand_site_choices` | peptidoforms.rs:34 | Enumerate the per-site alternative assignments for one selected set of sites | +| `PeptidoformsParams` | peptidoforms.rs:179 | `peptides`, `out`, `cfg: &PeptidoformsConfig`, `config_hash` | +| `unimod_mass` | mass.rs:13 | Name -> monoisotopic delta allowlist (the set of mods that validate) | +| `is_standard_residue` | constants.rs:54 | The 20-residue allowlist gating digest output | + +## Configuration + +`digest` reads `DigestConfig` (config.rs:183). `peptidoforms` reads +`PeptidoformsConfig` (config.rs:204). Both structs use `#[serde(default, +deny_unknown_fields)]`, so an unknown key is a load error and any omitted field +takes the default below. `rng_seed` (config.rs:974, default `0`) is a top-level +`Config` field, passed to digest as `DigestParams.rng_seed` (main.rs:422, +run.rs:136). + +| Field | Section | Default | Effect | +|---|---|---|---| +| `enzyme` | digest | `TrypsinP` | `trypsin_p` cuts after K/R including before P; `trypsin` suppresses the cut before P | +| `missed_cleavages` | digest | `2` | Max missed cleavages; joins at most `missed_cleavages + 1` tryptic fragments | +| `min_len` | digest | `5` | Minimum peptide length (residues) | +| `max_len` | digest | `50` | Maximum peptide length (residues) | +| `decoy.strategy` | digest | `reverse` | `reverse` / `scramble` realized; `diann_shift` and `none` produce no decoy (`diann_shift` rejected by `validate`) | +| `fixed_mods` | peptidoforms | `[{C: Carbamidomethyl}]` | Applied to every matching residue | +| `variable_mods` | peptidoforms | `[{M: Oxidation}]` | Enumerated as optional subsets | +| `max_variable_mods` | peptidoforms | `1` | Max simultaneous variable mods per peptidoform | +| `charge_min` | peptidoforms | `2` | Lowest precursor charge emitted | +| `charge_max` | peptidoforms | `3` | Highest precursor charge emitted | +| `unknown_modification` | peptidoforms | `error` | `error` (default) hard-fails on an unknown mod name; `skip` warns and drops the rule. Both branches are wired (see gotchas) | + +`rng_seed` is `0` by default (config.rs:991) and only affects the `scramble` +decoy strategy; `reverse` is independent of the seed. + +Config was recently pruned of dead fields in this area. `ResidueMod` +(config.rs:235) is only `{ residue: char, name: String }`; there is no +position/terminal specifier field. Unlike the enclosing `DigestConfig` / +`PeptidoformsConfig`, `ResidueMod` carries `#[serde(deny_unknown_fields)]` but +**not** `#[serde(default)]` (config.rs:233-234), so both `residue` and `name` are +mandatory in every mod entry; an entry missing either key is a load error rather +than a defaulted value. The doc comment on `residue` (config.rs:236) still +mentions `*` for "any / terminal handled separately in MVP", but that handling +was never implemented; the current code instead rejects a `*` residue with a hard +error at validation (see gotchas). There is no `DecoySource` field on +`DigestConfig` (the "dead/unwired config surface" list in CLAUDE.md refers to +types elsewhere, not here). `DecoyConfig` (config.rs:172) carries only +`strategy`. + +## Invariants, determinism, gotchas + +- **Determinism.** Emission order is FASTA order then in-protein sequence order, + preserved by the `order` vector (digest.rs:206), not by HashMap iteration. Ids + are a single monotonic counter. Scramble is fully deterministic for a fixed + `rng_seed` because the PRNG is seeded per peptide from the sequence + (digest.rs:117); `collision_safe_decoy`'s retry reseeding is `attempt`-indexed + (digest.rs:145), so which pairs drop is deterministic too. No floats are summed + in either stage, so the numeric-order determinism concerns from other stages do + not apply. +- **Target/decoy pairing.** A decoy's `target_id` points at its target's `id` and + the pair is emitted adjacently; targets carry `target_id = -1`. Downstream FDR + keeps the label in competition keys (see CLAUDE.md FDR / compete), so this + pairing must stay intact. A target whose decoy cannot be built collision-free is + dropped together with its would-be decoy (digest.rs:240-243), so the emitted + target and decoy populations stay one-to-one. `peptidoforms` propagates the link + via `base_peptide_id` (peptidoforms.rs:203). +- **Residue allowlist.** Peptides containing any non-standard residue are dropped + wholesale at digest time (digest.rs:90); they never reach peptidoforms. This is + the only place ambiguity codes (`X`/`B`/`Z`/`U`) are handled. Lowercase is not + an ambiguity case here: `read_fasta` upper-cases sequence lines first + (digest.rs:41), so a lowercase-but-standard residue survives. +- **Collision-checked decoys.** `collision_safe_decoy` (digest.rs:137) does + guarantee the decoy differs from its target, equals no target sequence, and is + unique among emitted decoys, retrying with reseeded scrambles up to + `MAX_DECOY_ATTEMPTS` (64). When no collision-free permutation exists (a + homopolymer or otherwise low-complexity sequence) the whole pair is dropped and + counted in `dropped_target_decoy_pairs`, so a decoy is never silently equal to + some target. Peptides with `len < 3` return `None` from `make_decoy` + (digest.rs:104) and are likewise dropped as a pair; with the default + `min_len = 5` this only bites if a caller lowers `min_len`. +- **No terminal modifications, and `*` is rejected.** `ResidueMod` matches on an + exact residue char (peptidoforms.rs:211, peptidoforms.rs:220). There is no + N-term/C-term matching, so protein/peptide N-terminal acetylation and similar + terminal mods cannot be enumerated. A `*` residue is no longer silently + unmatched: `known_mods` hard-fails on it up front (peptidoforms.rs:92-98, + message `unsupported '*' residue ...`). The `mass.rs` parser does model terminal + mod deltas, but Stage A2 never produces terminal-mod ProForma strings. +- **Two mods on one residue are prevented at validation.** `validated_rules` + rejects two fixed mods on the same residue (peptidoforms.rs:145-153) and a + fixed + variable mod on the same residue (peptidoforms.rs:167-172), and + `variable_combos`/`expand_site_choices` pick at most one alternative per site so + variable alternatives never stack (test + `same_site_alternatives_never_stack_and_are_deterministic`, peptidoforms.rs:329). + Under any config that loads, the merged `mods` list therefore never holds two + entries at one position, so `proforma`'s first-match `find` (peptidoforms.rs:23) + cannot silently drop a second mod. This supersedes the older "second mod at same + position dropped" limitation. +- **`unknown_modification` policy is wired.** The config field takes `error` / + `skip` (config.rs:212, `UnknownModPolicy`, config.rs:244) and `known_mods` + honors it via `validated_rules` (peptidoforms.rs:140-141): `Error` bails on an + unknown mod name (peptidoforms.rs:101-103); `Skip` logs a warning and drops the + offending rule (peptidoforms.rs:104-112). Test + `skip_policy_removes_unknown_modification` (peptidoforms.rs:350) pins the `skip` + path. +- **`config_hash` param is unused.** Both `DigestParams.config_hash` + (digest.rs:181) and `PeptidoformsParams.config_hash` (peptidoforms.rs:183) are + threaded from the CLI/orchestrator but never read inside either `run`. The + content hash written to the report comes from `blake3_file` over the output + Parquet (digest.rs:318, peptidoforms.rs:274), not from `config_hash`. +- **`decoy_strategy` string is derived by Debug-formatting the enum** + (`format!("{:?}", ...).to_lowercase()`, digest.rs:260). It is a display string, + not a parseable round-trip; changing enum variant names changes the column + value. +- **Charge range is validated.** `validated_rules` rejects `charge_min < 1` and an + empty range `charge_min > charge_max` with a hard error before any row is + processed (peptidoforms.rs:126-138); the `charge_min..=charge_max` emission + (peptidoforms.rs:239) is therefore always non-empty. Test + `invalid_charges_and_wildcard_modifications_are_rejected` (peptidoforms.rs:381) + pins the rejection. +- **Library-input mode bypasses both stages.** When a prebuilt library is passed, + `run` takes the library-input branch (run.rs:98) and never calls + `digest`/`peptidoforms`; the FASTA-build branch is the `_ =>` arm at run.rs:127. + Do not assume `peptides.parquet` / `peptidoforms.parquet` exist for every run. + +## How to extend / modify + +- **New enzyme.** Add a variant to `Enzyme` (config.rs:46) and a match arm in + `cleavage_sites` (digest.rs:58). Keep the site list strictly increasing and + terminated at `seq.len()`; the rest of `digest_protein` is enzyme-agnostic. + Semi-tryptic search would require changing how spans are enumerated in + `digest_protein` (digest.rs:75), not just the cut list. +- **New decoy scheme.** Add a `DecoyStrategy` variant (config.rs:17) and a match + arm in `make_decoy` (digest.rs:107). Preserve the C-terminal residue and + determinism (seed the PRNG from the sequence as scramble does, digest.rs:117). + The target/other-decoy collision guard is applied for free by + `collision_safe_decoy` (digest.rs:137) around any transform, but only its + `attempt == 0` path uses the new strategy; every retry falls back to `Scramble`, + so add the new variant to the run-loop `matches!` (digest.rs:224-227) if it + should be attempted at all. Realizing `DiannShift` means removing the `validate` + rejection (config.rs:1021) and implementing a fragment-m/z shift at predict-frag, + not a sequence rewrite here. +- **New modification.** Add the UniMod name and monoisotopic delta to + `unimod_mass` (mass.rs:13); it is the single allowlist both the Stage A2 + validation (`known_mods`, peptidoforms.rs:99) and the mass model consult. Then + reference it by name in `fixed_mods`/`variable_mods`. +- **Support a second mod per position or terminal mods.** Change `proforma` + (peptidoforms.rs:19) to emit all mods at a position (e.g. concatenate multiple + `[...]` groups) instead of `find`-ing the first, relax the stacking/overlap and + `*` checks in `validated_rules` (peptidoforms.rs:92-98, peptidoforms.rs:145-172), + and extend `ResidueMod` (config.rs:235) plus the residue-matching loops + (peptidoforms.rs:210-228) with a position/terminal specifier. The mass model in + `mass.rs` already carries `n_term_mod` fields to build on. +- **Schema changes.** Adding or reordering `peptides.parquet` / + `peptidoforms.parquet` columns is a schema change; bump the version in + `artifact::PEPTIDES` / `artifact::PEPTIDOFORMS` (schema.rs:11-12) and update the + downstream readers (`Table::read` callers in predict-frag and later stages). +- **Testing.** Existing unit tests live at the bottom of each file + (digest.rs:343, peptidoforms.rs:291). digest: `trypsin_p_cleaves_after_kr` + (digest.rs:348) pins Trypsin/P cut sites; `reverse_decoy_keeps_cterm` + (digest.rs:367) and `scramble_is_deterministic` (digest.rs:374) pin the fixed + C-terminus plus scramble reproducibility; + `collision_safe_decoy_avoids_targets_and_other_decoys` (digest.rs:382) pins the + retry-on-collision behavior; and `impossible_low_complexity_decoy_drops_pair` + (digest.rs:403) pins the pair-drop when no collision-free decoy exists. + peptidoforms: `proforma_places_mods` (peptidoforms.rs:296) pins bracket + placement; `combos_bounded` (peptidoforms.rs:302) and + `default_rules_and_form_order_are_preserved` (peptidoforms.rs:315) pin the subset + counts and legacy order; `same_site_alternatives_never_stack_and_are_deterministic` + (peptidoforms.rs:329) pins per-site alternative expansion; + `skip_policy_removes_unknown_modification` (peptidoforms.rs:350) pins the `skip` + policy; `fixed_stacking_and_fixed_variable_overlap_are_rejected` + (peptidoforms.rs:362) and `invalid_charges_and_wildcard_modifications_are_rejected` + (peptidoforms.rs:381) pin the validation rejections. There is no stage-level test + that round-trips through Parquet or exercises `run` end to end (CLAUDE.md "test + gaps"); add one when changing the output schema. diff --git a/docs/06_predict_frag_index_matchers.md b/docs/06_predict_frag_index_matchers.md new file mode 100644 index 0000000..27516f7 --- /dev/null +++ b/docs/06_predict_frag_index_matchers.md @@ -0,0 +1,490 @@ +# predict-frag, the library, the inverted fragment index, matchers + +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +This subsystem produces the run-independent spectral library and the data +structures that make fragment matching cheap. It has three parts: + +1. **predict-frag** (Stage C, `stages/predict_frag.rs`): turn concrete + peptidoforms into a library of precursor and b/y fragment m/z with predicted + fragment intensities and predicted iRT, keep the top-N fragments per + candidate, and assign each candidate a `candidate_id` equal to its rank in + precursor-m/z order. The output is two Parquet artifacts that are constant for + the whole run (they do not depend on the mzML being searched). + +2. **the library loader** (`index.rs`, `Library::load`): read those two Parquet + artifacts back into a Structure-of-Arrays in-memory model, group fragments by + candidate, build the bucketed peak-major inverted fragment index, and enforce + the preconditions the matchers rely on. + +3. **the matchers** (`matchers/`): given an observed peak m/z, a ppm tolerance, + and an isolation-window candidate range, return the predicted fragments that + fall within tolerance. Two backends exist: the default log-bin CSR `fragindex` + (`matchers/fragindex.rs`, spec in `fragindex_spec.md`) and the fallback + bucketed `Library::page_search`. A naive band-join (`matchers/naive.rs`) is the + correctness oracle, not a production path. + +The predictor traits (`predict.rs`) and the sidecar file contract (`sidecar.rs`) +sit under predict-frag: they are the boundary between the native (zero external +dependency) intensity/RT models and the optional Python predictors MS2PIP and +DeepLC. + +## Files + +| path | role | +|---|---| +| `rust/mumdia/crates/mumdia/src/stages/predict_frag.rs` | Stage C: build the library (parse, fragments, intensity, iRT, top-N, sort, write Parquet) | +| `rust/mumdia/crates/mumdia/src/predict.rs` | `RtPredictor`/`FragmentPredictor` traits + `NativeRt`/`NativeFrag` fallbacks | +| `rust/mumdia/crates/mumdia/src/sidecar.rs` | Python sidecar clients (MS2PIP, DeepLC, DeepLC fine-tune, MBR) over the file contract | +| `rust/mumdia/crates/mumdia/src/index.rs` | `Library` (SoA model + bucketed inverted index), `load()` preconditions, `page_search`, `candidate_range`, `deconvolve` | +| `rust/mumdia/crates/mumdia/src/matchers/mod.rs` | matcher module tree; `MatcherKind` selects the backend | +| `rust/mumdia/crates/mumdia/src/matchers/binning.rs` | `LogBins`: log-space bin geometry (fragindex_spec Section 2.2) | +| `rust/mumdia/crates/mumdia/src/matchers/fragindex.rs` | `FragIndex` CSR index, `SeedScratch` epoch-stamped accumulator, `probe_peak`, equivalence-gate scorer | +| `rust/mumdia/crates/mumdia/src/matchers/naive.rs` | band-join reference for the equivalence gate | +| `rust/mumdia/crates/mumdia-core/src/constants.rs` | `within_ppm` (min-relative predicate), `ppm_bounds` (query-relative), `PROTON` | +| `fragindex_spec.md` | language-agnostic algorithm spec the fragindex matcher implements | + +## Inputs and outputs + +### Consumed + +**peptidoforms** (Stage A2 output), read at `predict_frag.rs:52-58`: + +| column | type | note | +|---|---|---| +| `id` | u32 | peptidoform id | +| `base_peptide_id` | u32 | stripped-peptide id | +| `peptidoform` | str | ProForma-lite string with UniMod names | +| `charge` | i32 | precursor charge | +| `label` | str | `"target"` or `"decoy"` | +| `protein` | str | protein accession | + +### Produced (two Parquet artifacts + a `report.json` each) + +**fragment_library_precursors** (schema `("fragment_library_precursors", 1)`, +`schema.rs:13`), written at `predict_frag.rs:197-211`: + +| column | type | +|---|---| +| `candidate_id` | u32 (dense 0..N in precursor-m/z order) | +| `peptidoform_id` | u32 | +| `base_peptide_id` | u32 | +| `peptidoform` | str | +| `charge` | i32 | +| `precursor_mz` | f64 | +| `predicted_irt` | f32 | +| `label` | str | +| `protein` | str | +| `n_fragments` | i32 (kept fragment count after top-N) | + +**fragment_library_fragments** (schema `("fragment_library_fragments", 1)`, +`schema.rs:14`), written at `predict_frag.rs:212-223`: + +| column | type | +|---|---| +| `candidate_id` | u32 (foreign key into precursors) | +| `mz` | f64 (fragment m/z at its own `frag_charge`) | +| `predicted_intensity` | f32 | +| `name` | str (e.g. `b2`, `y3`) | +| `ion_type` | str (`b` or `y`) | +| `ordinal` | i32 (residue-count ordinal of the fragment) | +| `frag_charge` | i32 (1, or 2 when enabled) | + +Each artifact also gets an `ArtifactReport` (`predict_frag.rs:234-252`) recording +row count, blake3 content hash, params (`top_n`, `ms2pip_model`, `rt_predictor`, +`fragment_predictor`), a shared `stats` map (`candidates`, `fragments`, +`parse_errors`, `predict_frag.rs:226-229`), and `model_identity` (the concatenated +RT + fragment model ids, `predict_frag.rs:119`). Both reports share the same +`stats`/`model_identity`; only `rows` differs per artifact (`predict_frag.rs:239-243`). +The schema ids are the constants `artifact::FRAGMENT_LIBRARY_PRECURSORS` / +`artifact::FRAGMENT_LIBRARY_FRAGMENTS` (`schema.rs:13-14`). `run` itself returns +`(n_prec, n_frag)` (`predict_frag.rs:50,261`). + +The library-input path (`--lib-precursors`/`--lib-fragments`) skips Stage C and +feeds externally built Parquet with these exact schemas directly into +`Library::load`. + +`predict_frag::run` is driven by `PredictFragParams` (`predict_frag.rs:24-31`): +`peptidoforms` (input path), `out_precursors` / `out_fragments` (the two output +paths), `cfg` (`&PredictFragConfig`), `work_dir` (scratch dir for sidecar +Parquet), and `config_hash` (carried for provenance). + +## How it works + +### predict-frag (`stages/predict_frag.rs:50`) + +**Phase A: parse and enumerate fragments** (`predict_frag.rs:60-110`). Each +peptidoform row is parsed with `parse_peptidoform` and fragmented independently, +so rows are mapped in parallel with rayon (`into_par_iter`, line 72). The closure +returns `RowOut::Raw`, `RowOut::ParseErr`, or `RowOut::Empty`. Fragment charges +are chosen per row: charge 1 always, charge 2 added when the precursor charge is +at least `charge2_from_precursor_charge` (`predict_frag.rs:78-82`). `collect` +preserves row order, and the sequential fold at `predict_frag.rs:104-110` +reproduces the exact serial `raws` order and `n_parse_err` count. The parsed form +is stored on the `Raw` struct so intensity/iRT assignment reuses it instead of +re-parsing (`predict_frag.rs:45-47`). + +**iRT assignment** (`assign_rt`, `predict_frag.rs:265`). Native path calls +`NativeRt::predict_irt` per candidate. DeepLC path deduplicates by peptidoform +string (RT is charge-independent, `predict_frag.rs:281-290`), runs the sidecar +once over the unique set, then maps results back. Peptidoforms DeepLC returns no +prediction for are anchored at `irt = 0.0` and counted; if any are missing a +`tracing::warn!` fires (`predict_frag.rs:303-308`). This is the DeepLC-miss iRT +warning: it makes the silent "unmatched peptidoform gets iRT 0.0" failure visible, +because an iRT-0 anchor collapses the RT window onto the gradient origin and +misplaces the candidate at extraction. The DeepLC branch requires `deeplc_python` +and errors otherwise (`predict_frag.rs:275-277`); its returned model id is the +hardcoded string `"deeplc-4.0-mt"` (`predict_frag.rs:309`), not a trait +`identity()` (the sidecar path has no `RtPredictor` impl to query). + +**intensity assignment** (`assign_intensities`, `predict_frag.rs:315`). Native +path calls `NativeFrag::predict_intensities`. MS2PIP path runs the sidecar over +all candidates; an empty whole-map result is a hard error (`bail`, +`predict_frag.rs:342-344`). Per candidate, MS2PIP supplies charge-1 b/y +intensities keyed by `(ion_byte, ordinal)`; charge-2 fragments (which MS2PIP does +not emit) fall back to the native model (`predict_frag.rs:350-363`). MS2PIP +values (TIC-fraction scale, roughly 0.02-0.3) and the native charge-2 fallback +(max-normalized, roughly 0.19-0.5) live on different scales, so each charge group +is max-normalized to its own peak before they compete for top-N slots +(`predict_frag.rs:365-384`); otherwise the larger-scale group would always win the +truncation. There are two distinct MS2PIP-miss fallbacks: a candidate MS2PIP +returns nothing for (absent from the map, or an empty per-candidate map) falls +back wholesale to native (`predict_frag.rs:387-389`), whereas a single charge-1 +fragment whose `(ion_byte, ordinal)` key MS2PIP omits gets intensity `0.0`, not +the native value (`unwrap_or(&0.0)`, `predict_frag.rs:359`); a fragment at +`0.0` can then be dropped by top-N. MS2PIP requires `ms2pip_python` and errors +otherwise (`predict_frag.rs:325-327`); its model id is `format!("ms2pip-{model}")` +(`predict_frag.rs:392`). + +**top-N and candidate_id** (`predict_frag.rs:121-139`). For each candidate, +fragments are ranked by predicted intensity descending, truncated to +`top_n_fragments`, then re-sorted ascending to restore stored order; an in-place +forward-swap gather compacts the kept fragments without reallocating. Candidates +left with zero fragments are dropped (`retain`, line 139). Then `raws` is sorted +by `precursor_mz` with a stable `sort_by` (`predict_frag.rs:142`), and +`candidate_id` is assigned by `enumerate` over that order +(`predict_frag.rs:174-175`). This is the single most load-bearing invariant of +the whole subsystem: `candidate_id` is the dense precursor-m/z rank, which is what +lets the index recover the isolation-window candidate slice by binary search and +lets `candidate_id` directly index the dense accumulator. + +### The native predictor models (`predict.rs`) + +Both native fallbacks are deterministic and Python-free, so the engine runs with +zero external dependencies. `NativeRt` (`predict.rs:55-70`) sums a self-derived +per-residue hydrophobicity coefficient (`rt_coeff`, `predict.rs:27-53`, clean-room, +not a borrowed vector), adds `0.01 * mod_mass` per modified residue, and adds a +`sqrt(length)` term so very long peptides do not elute infinitely late; `identity` +is `native-rt-v1`. `NativeFrag` (`predict.rs:75-105`) weights y ions at 1.0 and b +ions at 0.75, scales by a mid-sequence positional factor `1 - 0.5*|ordinal-L/2|/(L/2)` +(mid-sequence fragments are more intense), halves charge-2 fragments, then +max-normalizes the whole vector to its peak; `identity` is `native-frag-v1`. The +predictor traits `RtPredictor` / `FragmentPredictor` (`predict.rs:13-22`) each +expose `predict*` plus `identity`; only the native structs implement them (the +sidecar paths bypass the traits and emit their own id strings, above). + +### The sidecar file contract (`sidecar.rs`) + +Sidecars follow a positional-CLI Parquet contract (no JSON request file): write an +input Parquet, invoke `python script arg...`, read an output Parquet keyed by id. + +- `run_ms2pip` (`sidecar.rs:42-78`): input columns `id` (u32), `peptidoform` + (str), `charge` (i32); output columns `id`, `ion_type` (str), `ordinal` (i32), + `intensity` (f32). Returns `candidate_id -> (ion_byte, ordinal) -> intensity`; + `ion_byte` is the first byte of `ion_type` (`b'?'` if empty, `sidecar.rs:72`). + Argv is `[input, output, model]` (`sidecar.rs:63`). +- `run_deeplc` (`sidecar.rs:81-105`): input columns `id`, `peptidoform`; output + columns `id`, `predicted_rt` (f32). Returns `id -> predicted_rt`. Argv is + `[input, output]`. +- `run_deeplc_finetune` (`sidecar.rs:111-155`): fine-tunes the DeepLC RT model on + confident seed PSMs and writes a new output precursor table with updated + `predicted_irt`; it does not modify the input library. + Positional contract `deeplc_finetune.py --epochs E + --patience P --q-train Q --batch B`. Invoked by `run` between search-seed and RT + calibration, not by predict-frag. +- `run_mbr` (`sidecar.rs:162-213`): match-between-runs transfer (Stage D3, needs + >= 2 runs). Positional contract `mbr_worker.py + [flags]`, where `psms_csv` is the per-run psms paths joined by + `,` in `source` order; the launcher always passes `--q-anchor`, + `--min-anchor-runs`, `--q-transfer`, and `--seed`, adds `--out-scored` when an + output-scored path is supplied, and adds `--frag-csv` / `--consensus-corr-min` + only when fragments are supplied and the threshold is > 0. +- `run_worker` (`sidecar.rs:217-233`) is the shared launcher; it bails if the + process exits non-zero. `utf8 = true` sets `PYTHONUTF8=1` and + `PYTHONIOENCODING=utf-8` (DeepLC/Keras/torch crash on the Windows cp1252 console); + it is on for the DeepLC and fine-tune calls, off for MS2PIP and MBR. + +### Library load and the bucketed inverted index (`index.rs:54`) + +`Library::load(precursors, fragments, bucket_size)` reads both Parquet artifacts, +validates the label column (only `"target"`/`"decoy"` allowed, via +`fdr::validate_labels`, `index.rs:65`), and rebuilds the per-candidate fragment +arrays grouped and contiguous (`index.rs:100-128`), so `cand_frags(cid)` returns +three parallel slices (m/z, predicted intensity, name) for one candidate +(`index.rs:209-218`); `n_candidates()` is the candidate count (`index.rs:204`). +Each row becomes a `Candidate` (`index.rs:23-35`) with `candidate_id`, +`peptidoform_id`, `base_peptide_id`, `peptidoform`, `charge`, `precursor_mz`, +`predicted_irt`, `protein`, `frag_start`/`n_frag` (the slice bounds into the flat +fragment arrays), and `is_decoy`, which is derived from the `label` string +(`label == "decoy"`, `index.rs:122`); the string label is not otherwise retained. +`load` then builds the bucketed inverted index (`index.rs:160-188`): + +1. Emit one `(frag_mz as f32, candidate_id, frag_int)` entry per fragment. +2. Globally sort entries by fragment m/z with `par_sort_by` (parallel stable + sort, identical result to the serial stable sort, `index.rs:170`). +3. Chunk the sorted entries into fixed buckets of `bucket_size` (floored at 1 via + `bucket_size.max(1)`, `index.rs:173`); record each bucket's first (== minimum) + m/z in `bucket_min`; within each bucket sort by `candidate_id` + (`index.rs:173-179`). +4. Split into the three parallel arrays `idx_mz`/`idx_cid`/`idx_int`. + +`page_search` (`index.rs:247`) probes this index for an observed neutral m/z `q`. +It early-returns on a degenerate window or empty index (`cand_hi <= cand_lo || +idx_mz.is_empty()`, `index.rs:255-257`). Otherwise `ppm_bounds(q, tol_ppm)` gives +the query window `[lo, hi]` (cast to f32); the first bucket is +`partition_point(|m| m <= lo32).saturating_sub(1)` and the last is +`partition_point(|m| m <= hi32)` over `bucket_min` (`index.rs:258-268`); within +each bucket, because `idx_cid` is ascending, two `partition_point` calls narrow to +the `[cand_lo, cand_hi)` slice (`index.rs:275-276`); a linear tail applies the +exact f32 m/z bound (`index.rs:277-283`). `candidate_range` (`index.rs:238`) turns +an isolation window into `[lo, hi)` over `prec_mz` by two `partition_point`s (`m < +win_lo` for `lo`, `m <= win_hi` for `hi`, so `win_hi` is inclusive and `win_lo` +exclusive). + +### The fragindex CSR matcher (`matchers/fragindex.rs`, fragindex_spec Section 2-3) + +`FragIndex::build` (`fragindex.rs:46`) is a two-pass counting sort into a CSR +layout keyed by log-space bin. It first derives the m/z range by scanning +`lib.frag_mz` for the min and max (`fragindex.rs:61-70`); if the library is empty +(no finite bound) it falls back to `[1.0, 2.0]` (`fragindex.rs:71-74`), and it +clamps the arguments to `LogBins::new` so `mz_min >= 1.0` and `mz_max > mz_min` +(`fragindex.rs:75`). `LogBins::new` (`binning.rs:27`) asserts `mz_min > 0 && +mz_max >= mz_min` and precomputes the geometry: `delta = tol_ppm * 1e-6`, bin +width `w = ln(1 + delta)`, `inv_w`, `ln_min`, and `n_bins = floor(span * inv_w) ++ 2` (the `+2` pads the top so `bin+1` never overflows). `LogBins::bin` +(`binning.rs:49`) maps m/z to `floor((ln(mz) - ln_min) * inv_w)`, clamped to +`[0, n_bins-1]` (m/z <= 0 or <= `mz_min` maps to bin 0). Pass 1 counts +per-bin occupancy with the `+1` counting-sort offset (`fragindex.rs:83-86`); a +prefix sum turns counts into CSR start offsets (`fragindex.rs:88-90`); pass 2 +scatters postings in candidate-id order (`fragindex.rs:98-110`) so `post_cand` is +ascending within every bin. Postings are Structure-of-Arrays +(`post_cand`/`post_mz`/`post_int`/`post_frag`) so the verify hot loop streams only +`post_mz`. + +`FragIndex` keeps its own copy of `prec_mz` and exposes `n_cand()` +(`fragindex.rs:125`), `tol_ppm()` (`fragindex.rs:129`), and a `candidate_range` +(`fragindex.rs:137-141`) with the same `[lo, hi)` semantics as +`Library::candidate_range`, so a caller holding only the index can still narrow to +the isolation window. `probe_peak` (`fragindex.rs:152`) early-returns on a +degenerate window (`cand_hi <= cand_lo`, `fragindex.rs:159`), then probes bins +`bin(peak)-1 ..= bin(peak)+1` (clamped, `fragindex.rs:162-165`), narrows each bin +to `[cand_lo, cand_hi)` by binary search over the ascending `post_cand` +(`fragindex.rs:172-174`), and verifies each posting with the exact `within_ppm` +predicate in f64 (`fragindex.rs:176-179`). Its callback receives `(cid, +post_mz_f64, post_int, post_frag)`, where `post_frag` is the candidate-local +fragment ordinal that extract carries through directly (the bucketed path has to +recover it via `local_frag_index`). `SeedScratch` (`fragindex.rs:191`) is the epoch-stamped +dense accumulator: `stamp[cc]` records the last epoch a candidate was touched; +`epoch` is incremented before each scan (`fragindex.rs:221`) and starts at 0 so 0 +is never a live epoch; on first touch the score is zeroed and the candidate pushed +to `touched` (`fragindex.rs:227-232`); after a scan only touched candidates are +read. The seed accumulates a fused `(count, obs_sum)` semiring where `obs_sum` +sums the observed peak intensity per matched posting (predicted intensity +deliberately discarded, `fragindex.rs:234`), and exposes the touched set plus +per-candidate `count(cid)` / `obs_sum(cid)` getters (`fragindex.rs:241-253`). + +The free function `score_scan_count_dot` (`fragindex.rs:260`) is a separate, +non-`SeedScratch` scorer used only by the equivalence gate: it accumulates +`(count, dot)` per candidate over one scan, where `dot` is the sum over matched +postings of `predicted_intensity * peak_intensity` (both widened to f64, distinct +from `SeedScratch`'s observed-only `obs_sum`), collects into a `HashMap`, and +returns the result sorted by candidate id (`fragindex.rs:275-277`). The `naive` +band-join scorer (`naive.rs:16`) computes the identical `(count, dot)` under the +same f32-rounded `within_ppm` predicate (`naive.rs:30-33`), which is what lets the +gate assert exact Count equality and near-exact Dot equality. + +**The +/-1 probe exactness.** The correctness of probing only three bins rests on: +two m/z values within tolerance differ by at most one bin. Proof sketch: within +tolerance means `|ln(a) - ln(b)| <= w`, one bin width, so the two points span at +most two adjacent bins (`binning.rs` test `within_tol_pairs_are_at_most_one_bin_apart`, +`fragindex_spec.md` Section 2.2). One subtlety makes this exact in the +implementation: posting m/z is stored f32, so build bins each posting by the +**same f32-rounded value** the verify uses (`fragindex.rs:85` and +`fragindex.rs:102`), not the raw f64. Binning by raw f64 while verifying the f32 +value could place a posting two bins from the peak and silently drop a +boundary-straddling within-tolerance pair. The test +`probe_finds_within_tol_across_bin_boundaries` (`fragindex.rs:429`) sweeps the m/z +range to guard this. + +**The bucketed fallback vs fragindex predicate.** The two backends do not use the +same tolerance predicate. `page_search` uses `ppm_bounds` (a window symmetric +around the query `q`, `constants.rs:78`). `fragindex` uses `within_ppm` (a +min-relative predicate, `hi - lo <= tol_ppm*1e-6*lo`, `constants.rs:92`). These +differ at the tolerance edge, so a handful of edge pairs are accepted by one and +not the other. On AIF full-range-window data the predicate difference shifts +identifications enough to matter, which is why the bucketed path is retained for +A/B (`config.rs:30-35`). The fragindex `probe_peak` and its `naive` band-join both +verify under `within_ppm` on f32-rounded m/z (`naive.rs:30`), so the equivalence +gate is not contaminated by storage precision. + +### Wiring into stages + +Both stages build the fragindex backend only when `MatcherKind::Fragindex` is +selected and otherwise fall through to the bucketed `Library` path. +search-seed: `FragIndex::build` at `search_seed.rs:56-57`; fragindex path +`seed_fragindex_windows` parallelizes across isolation-window groups with a +per-thread `SeedScratch` and a deterministic total-order merge +(`search_seed.rs:313-409`); bucketed path uses `page_search` +(`search_seed.rs:77`). extract selects via the `probe_matched` helper +(`extract.rs:42-59`): fragindex carries the true generating fragment ordinal in +`post_frag`, whereas the bucketed path recovers the ordinal by nearest stored m/z +(`Library::local_frag_index`, `index.rs:222`). + +## Key types and functions + +| name | file:line | what it does | +|---|---|---| +| `PredictFragParams` | `predict_frag.rs:24` | Stage C entry args: in/out paths, `cfg`, `work_dir`, `config_hash` | +| `predict_frag::run` | `predict_frag.rs:50` | Stage C entry: parse, fragment, assign intensity/iRT, top-N, sort, write; returns `(n_prec, n_frag)` | +| `Raw` | `predict_frag.rs:34` | one candidate pre-assignment; caches the `ParsedPeptidoform` so RT/intensity reuse the parse | +| `assign_rt` | `predict_frag.rs:265` | native or DeepLC iRT; emits the DeepLC-miss warning; DeepLC id `"deeplc-4.0-mt"` | +| `assign_intensities` | `predict_frag.rs:315` | native or MS2PIP intensity with per-charge-group normalization + native charge-2 fallback; MS2PIP id `"ms2pip-{model}"` | +| `RtPredictor` / `FragmentPredictor` | `predict.rs:13` / `predict.rs:19` | predictor traits (predict + `identity`); implemented only by the native structs | +| `NativeRt` | `predict.rs:25` | additive retention-coefficient model + `sqrt(len)` + `0.01*mod` term, `identity` `native-rt-v1` | +| `NativeFrag` | `predict.rs:73` | heuristic b/y intensity model (y=1.0, b=0.75, mid-seq positional, charge-2 x0.5), max-normalized, `identity` `native-frag-v1` | +| `resolve_script` | `sidecar.rs:20` | locate a worker script (CWD, exe dir/dir, exe dir/scripts, else CWD-relative) | +| `run_ms2pip` | `sidecar.rs:42` | MS2PIP client; in `id`/`peptidoform`/`charge`, out `id`/`ion_type`/`ordinal`/`intensity`; returns `cid -> (ion_byte, ordinal) -> intensity` | +| `run_deeplc` | `sidecar.rs:81` | DeepLC client; in `id`/`peptidoform`, out `id`/`predicted_rt`; returns `id -> predicted_rt` | +| `run_deeplc_finetune` | `sidecar.rs:111` | DeepLC multitask fine-tune; `deeplc_finetune.py ` + epoch/patience/q-train/batch flags (called by `run`, not predict-frag) | +| `run_mbr` | `sidecar.rs:162` | MBR transfer (Stage D3); `mbr_worker.py ` + flags | +| `run_worker` | `sidecar.rs:217` | shared launcher; bails on non-zero exit; `utf8` sets `PYTHONUTF8`/`PYTHONIOENCODING` | +| `Candidate` | `index.rs:23` | one library row in SoA; `is_decoy` derived from the `label` string | +| `Library` | `index.rs:37` | SoA candidate + fragment model plus the bucketed inverted index | +| `Library::load` | `index.rs:54` | read Parquet, validate labels, group fragments, build index, enforce preconditions | +| `Library::n_candidates` / `cand_frags` | `index.rs:204` / `index.rs:209` | candidate count; per-candidate (m/z, intensity, name) slices | +| `Library::page_search` | `index.rs:247` | bucketed probe: bucket select -> candidate slice -> f32 ppm verify | +| `Library::candidate_range` | `index.rs:238` | isolation window -> `[lo, hi)` over `prec_mz` | +| `Library::local_frag_index` | `index.rs:222` | nearest-stored-m/z fragment ordinal (bucketed path only) | +| `deconvolve` | `index.rs:291` | z-charged peak m/z -> neutral m/z, in f64 | +| `LogBins` / `LogBins::bin` | `binning.rs:11` / `binning.rs:49` | log-space bin geometry and mapping | +| `FragIndex::build` | `fragindex.rs:46` | two-pass counting-sort CSR build at a fixed tolerance; derives m/z range from the library | +| `FragIndex::probe_peak` | `fragindex.rs:152` | +/-1 bin probe + `within_ppm` verify, candidate-window narrowed; callback `(cid, mz, int, frag)` | +| `FragIndex::candidate_range` / `n_cand` / `tol_ppm` | `fragindex.rs:137` / `:125` / `:129` | index-side isolation-window narrowing + accessors | +| `SeedScratch` | `fragindex.rs:191` | epoch-stamped dense `(count, obs_sum)` accumulator; `touched`/`count`/`obs_sum` getters | +| `score_scan_count_dot` (fragindex / naive) | `fragindex.rs:260` / `naive.rs:16` | equivalence-gate scorers, `dot = predicted*observed`, under an identical predicate | +| `within_ppm` / `ppm_bounds` | `constants.rs:92` / `constants.rs:78` | min-relative vs query-relative tolerance predicates | + +## Configuration + +`PredictFragConfig` (`config.rs:252-282`, `#[serde(default, deny_unknown_fields)]`, +so the struct was pruned to exactly these fields and unknown keys are rejected): + +| field | default | effect | +|---|---|---| +| `predictor` | `Native` (`FragPredictorKind`) | native heuristic vs `Ms2pip` sidecar for fragment intensities | +| `rt_predictor` | `Native` (`RtPredictorKind`) | native additive model vs `Deeplc` sidecar for iRT | +| `charge2_from_precursor_charge` | `2` | precursor charge at/above which charge-2 fragments are added (was 3; lowered to keep the ~16% of charge-2 precursors' doubly-charged transitions) | +| `top_n_fragments` | `6` | fragments kept per candidate after intensity ranking (top-6 is standard DIA) | +| `ms2pip_model` | `"HCD"` | MS2PIP model name passed as argv | +| `ms2pip_python` | `None` | interpreter for the MS2PIP sidecar; required when `predictor=ms2pip`, else the stage errors | +| `deeplc_python` | `None` | interpreter for the DeepLC sidecar; required when `rt_predictor=deeplc`, else the stage errors | +| `sidecar_script_dir` | `"scripts"` | directory searched by `resolve_script` for the worker scripts | + +Matcher selection (both stages default to `Fragindex`): + +| field | default | effect | +|---|---|---| +| `MatcherKind` | `Fragindex` (`config.rs:36-42`) | `Fragindex` (log-bin CSR) or `Bucketed` (`Library::page_search`) | +| `search_seed.matcher` | `Fragindex` (`config.rs:300,316`) | backend for the seed search | +| `search_seed.fragment_tol_ppm` | `20.0` (`config.rs:288,312`) | tolerance the seed index is built at | +| `extract.matcher` | `Fragindex` (`config.rs:460,534`) | backend for extraction | +| `extract.bucket_size` | `8192` (`config.rs:446,530`) | fixed bucket size of the bucketed `Library` index (fragindex ignores it) | + +## Invariants, determinism, gotchas + +- **candidate_id contiguity.** `candidate_id` must be the dense range `0..N` in + precursor-m/z-ascending row order. predict-frag guarantees it + (`predict_frag.rs:142,174`); `Library::load` re-checks it and bails with a clear + message (`index.rs:78-87`); `FragIndex::build` asserts it (`fragindex.rs:50-56`). + An external library fed in unsorted or unindexed (for example + `import_diann_lib.py` output not passed through `make_reverse_decoys.py`) fails + here rather than silently misgrouping fragments. +- **precursors ascending by m/z.** `Library::load` verifies `prec_mz` is + non-decreasing and bails otherwise (`index.rs:135-146`); the `candidate_range` + binary search assumes it. +- **fragment foreign key.** A fragment row referencing `candidate_id >= N` is a + hard error (`index.rs:92-96`). +- **both label classes must be present.** A library with zero targets or zero + decoys makes target-decoy q-values meaningless, so `load` now bails at library + load with a clear message rather than completing a long search on an invalid + null (`index.rs:147-158`). The label column is also validated: any value other + than `"target"`/`"decoy"` is rejected up front by `fdr::validate_labels` + (`index.rs:65`, `fdr.rs:127`). +- **total_frags must fit u32.** `FragIndex::build` asserts this + (`fragindex.rs:58`) because posting indices are u32. +- **f32 posting m/z, f64 math.** Index/posting m/z is stored f32; the ppm verify + widens to f64 (`fragindex.rs:176-177`, `index.rs` module docs). f32 ULP is + ~0.12 ppm, 200-400x below the 20-50 ppm regime, so storage precision is not a + gate disagreement source; build and verify use the same f32-rounded value. +- **per-posting accumulation.** A candidate with two fragments within tolerance of + one peak counts twice; a peak within tolerance of two candidate fragments counts + twice (fragindex_spec Section 1.4). Do not deduplicate. Tests + `two_frags_one_peak_counts_both` (`fragindex.rs:371`) and the count assertion in + `equivalence_gate_vs_naive` (`fragindex.rs:356`) guard this. +- **epoch, not value, for first touch.** First touch is `stamp[cc] != epoch`, never + `acc[cc] == 0`; a legitimate zero score would otherwise be misclassified + (`fragindex.rs:227`, spec Section 3.5). +- **determinism.** predict-frag maps rows in parallel but `collect` + a sequential + fold reproduce the serial order; the precursor-m/z sort is stable and top-N is a + deterministic ranking; `Library::load` uses a parallel stable sort equal to the + serial one; `FragIndex::build` is fully serial (candidate-order scatter, no + hashing); `seed_fragindex_windows` merges partial results in a total order + independent of thread/group order (`search_seed.rs:392-408`). `SeedScratch` + sums `obs_sum` in the caller's fixed peak order and `touched()` is in + first-touch order, so callers sort before any float reduction + (`fragindex.rs:239-243`). +- **DeepLC nondeterminism and the iRT-0 anchor.** The DeepLC sidecar and fine-tune + are not seeded, so iRT values vary run to run. Any peptidoform DeepLC does not + return lands at iRT 0.0; the warning at `predict_frag.rs:303-308` reports the + count so a large miss is visible rather than silent. +- **predicate mismatch between backends.** `page_search` (query-relative + `ppm_bounds`) and `fragindex` (`within_ppm`) accept slightly different edge + pairs; switching `MatcherKind` can move a small number of identifications. This + is expected, not a bug (`config.rs:30-35`). +- **`naive.rs` is not production.** It is O(C x frags x peaks), used only as the + equivalence oracle. + +## How to extend / modify + +- **Add a fragment or RT predictor.** Implement `FragmentPredictor` / + `RtPredictor` (`predict.rs:13,19`) for a native model, or add a variant to + `FragPredictorKind` / `RtPredictorKind` (`config.rs:90,79`) plus a branch in + `assign_intensities` / `assign_rt`. Match the existing `identity()` convention; + the id flows into `model_identity` and the artifact report. Keep the native + fallback path intact so the engine still runs with no Python. +- **Add a sidecar.** Follow the positional-CLI file contract in `sidecar.rs`: + write an input Parquet, invoke `run_worker(python, script, &[argv...], utf8)`, + read an output Parquet keyed by id. Use `resolve_script` so a deployed binary + finds the worker regardless of CWD. Set `utf8 = true` for Keras/torch workers + (they crash on the Windows cp1252 console). Never hardcode the interpreter path; + it comes from config (`ms2pip_python`, `deeplc_python`, or the rescore/finetune + equivalents). +- **Change the matcher.** New backends go under `matchers/` behind a `MatcherKind` + variant. Any new backend must pass the equivalence gate against `naive.rs` at + `K = C` under the same `within_ppm` predicate (test pattern in + `fragindex.rs:331`) before its speed is trusted. Before attempting a fragindex + optimization, read `fragindex_spec.md` Section 5: cache-blocking, accumulator + prefetch, radix partitioning, bin-major inversion, and distinct-m/z dedup were + all measured null or negative in the realistic DIA regime. The real levers are + top-N reduction (already applied, `top_n_fragments`) and per-window parallelism + (already applied in `seed_fragindex_windows`). +- **Change fragment charges or top-N.** `charge2_from_precursor_charge` and + `top_n_fragments` are the two knobs; both are sensitivity/speed tradeoffs. + Lowering top-N removes collisions roughly proportionally (spec Section 5.5) and + shrinks the library; it does not change matcher correctness. +- **Feeding an external library.** It must satisfy the `load()` preconditions + (dense `candidate_id` in precursor-m/z order, ascending `prec_mz`, decoys + present). The decoy-builder scripts (`make_reverse_decoys.py`) sort and reindex + to satisfy them; do not bypass that step. diff --git a/docs/07_search_seed.md b/docs/07_search_seed.md new file mode 100644 index 0000000..58eef74 --- /dev/null +++ b/docs/07_search_seed.md @@ -0,0 +1,387 @@ +# search-seed (Stage S): broad search + mass recalibration + +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +Stage S is a native broad, DIA-aware search whose product is **calibration +anchors, not final identifications**. It scores library candidates against every +MS2 scan with a Sage-lite hyperscore, keeps the single best-scoring spectrum per +candidate, assigns target-decoy q-values, and from the confident subset derives a +per-run fragment mass recalibration (systematic ppm offset plus a learned +tolerance). Downstream stages consume its two outputs: + +- `rt-im-train` reads `seed_psms.parquet` (the confident target PSMs give the + observed-RT-vs-predicted-iRT anchors for LOESS/linear calibration and the RT + window widths). +- `extract` reads `.masscal.json` (the offset is applied to the observed + peak m/z to align it with the predicted-fragment frame, the learned tolerance + replaces `extract.frag_tol_ppm`). + +The stage sits behind a file contract (`search_seed.rs:2-4`), so a real Sage / +sage-core adapter can replace the native scorer later without touching the +consumers, as long as it writes the same `seed_psms` schema and `masscal.json`. +Library-level decoys are the single source of truth: there is no engine-side decoy +generation here, so target-decoy counting is never mixed-method +(`search_seed.rs:6-7`). Because the seed is broad and best-per-candidate rather +than best-per-spectrum, it is deliberately high-recall and low-precision at this +point; precision is recovered downstream by RT/mass calibration, feature +rescoring, and FDR control. + +The stage is iRT-independent: `predicted_irt` is only copied through to the +output, never used in scoring. That is why the `run` orchestrator computes the +seed once on the base library and reuses it both before and after the optional +DeepLC fine-tune (`run.rs:237-280`). + +## Files + +| path | role | +|---|---| +| `rust/mumdia/crates/mumdia/src/stages/search_seed.rs` | the stage: scoring loop, best-per-candidate merge, mass recalibration, output writers | +| `rust/mumdia/crates/mumdia/src/fdr.rs` | `target_decoy_q`, `count_targets_at_q`, `ln_factorial` (shared with `rescore`) | +| `rust/mumdia/crates/mumdia/src/matchers/fragindex.rs` | `FragIndex` (CSR inverted fragment index) + `SeedScratch` epoch accumulator | +| `rust/mumdia/crates/mumdia/src/index.rs` | `Library::load`, `candidate_range`, `page_search` (bucketed backend), `cand_frags` | +| `rust/mumdia/crates/mumdia/src/calibrate.rs` | `percentile` used by the tolerance fit | +| `rust/mumdia/crates/mumdia/src/spectra.rs` | `load_ms2` (reads the converted MS2 Parquet into `Ms2Scan`) | +| `rust/mumdia/crates/mumdia-core/src/constants.rs` | `ppm_bounds`, `ppm_diff`, `within_ppm` (the shared ppm math) | +| `rust/mumdia/crates/mumdia-core/src/config.rs` | `SearchSeedConfig` (`:286`), `MatcherKind` (`:38`) | +| `rust/mumdia/crates/mumdia-core/src/schema.rs` | `artifact::SEED_PSMS = ("seed_psms", 1)` (`:15`) | +| `rust/mumdia/crates/mumdia/src/main.rs` | CLI `Cmd::SearchSeed` (`:75`, dispatch `:458`) | +| `rust/mumdia/crates/mumdia/src/stages/run.rs` | orchestrator wiring (`:219`), masscal handoff to extract (`:309`) | + +## Inputs and outputs + +### Consumed + +- **MS2 spectra** (`--ms2`, a `convert` output Parquet), loaded by `load_ms2` + into `Vec`. Each `Ms2Scan` carries `scan_index`, `id`, `rt_seconds`, + an `IsolationWindow { target_mz, lower_mz, upper_mz, im_lower, im_upper }`, and + `peaks: Vec`. +- **Library** (`--library-precursors` + `--library-fragments`), loaded by + `Library::load` (`index.rs:54`). Provides `cands: Vec` + (`candidate_id`, `peptidoform`, `charge`, `precursor_mz`, `base_peptide_id`, + `protein`, `is_decoy`, `predicted_irt: f32`, `frag_start`, `n_frag`), the flat + fragment arrays (`frag_mz`, `frag_int`, `frag_name`), and the bucketed inverted + index for `page_search`. +- **Config**: `cfg.search_seed` plus `cfg.extract.bucket_size` (the bucketed + Library index bucket width; `run.rs:225`, `main.rs:473`). + +### Produced + +**`seed_psms.parquet`** (schema id `seed_psms` v1, `schema.rs:15`), one row per +candidate that, in at least one scan, both cleared `min_matched_peaks` and ranked +within that scan's top `report_psms` (the per-scan hyperscore sort is truncated to +`report_psms` *before* the best-per-candidate fold, `search_seed.rs:89`/`:369`, so +a candidate that always ranks below `report_psms` gets no row even if it cleared +`min_matched_peaks`). Rows are sorted by `candidate_id`. Written at +`search_seed.rs:233-250`. The `ArtifactReport` records `logical_name` = +`schema_name` = `"seed_psms"`, `schema_version = 1`, and `stage = "search-seed"` +(`search_seed.rs:257-260`): + +| column | type | meaning | +|---|---|---| +| `candidate_id` | u32 | dense library candidate id (== library row index) | +| `peptidoform` | str | ProForma-lite peptidoform string | +| `charge` | i32 | precursor charge | +| `precursor_mz` | f64 | library precursor m/z | +| `base_peptide_id` | u32 | stripped-peptide id (for peptide-level rollup) | +| `protein` | str | protein accession | +| `label` | str | `"target"` or `"decoy"` (from `Candidate::is_decoy`) | +| `score` | f64 | best hyperscore over all scans | +| `spectrum_q` | f64 | best-per-candidate target-decoy q-value | +| `observed_rt` | f64 | RT (seconds) of the best-scoring scan | +| `predicted_irt` | f32 | library iRT, copied through (unused in scoring) | +| `matched_peaks` | i32 | matched-fragment count at the best scan | +| `scan_index` | u32 | `scan_index` of the best-scoring scan | + +**`.masscal.json`** (written at `search_seed.rs:217-227`): + +| key | type | meaning | +|---|---|---| +| `frag_ppm_offset` | f64 | median signed ppm of `observed_peak` relative to `predicted_fragment` (`ppm_diff(peak_mz, fmz)`), i.e. the systematic offset | +| `frag_tol_ppm` | f64 | learned tolerance: `max(5.0, 1.5 * P95(|dev - offset|))` | +| `frag_ppm_sigma` | f64 | duplicate of `frag_tol_ppm` (the learned tolerance is the local mass-uncertainty estimate) | +| `n_dev` | usize | number of fragment-to-nearest-peak deviations collected (`devs.len()`) | +| `cal_passes` | int | 0 (fallback, too few devs), 1 (single pass), or 2 (robust second pass) | + +Only `frag_ppm_offset` and `frag_tol_ppm` are consumed downstream (by `extract`, +`extract.rs:678-698`). `frag_ppm_sigma`, `n_dev`, and `cal_passes` are written but +read by no consumer (grep-confirmed); they are diagnostic / audit fields. The +deviations are not the postings matched during scoring: for every confident target +PSM a fresh nearest-peak search over the candidate's full library fragment list is +run at its best scan (see step 6), so `n_dev` counts fragment-to-peak pairs, not +matched hyperscore postings. + +**`.report.json`** (`ArtifactReport`, `search_seed.rs:256-274`): rows, +`content_hash` (blake3 of the output Parquet, `search_seed.rs:262`), `params` +(`fragment_tol_ppm`, `report_psms`, `min_matched_peaks`, `top_n_peaks`, +`fdr_seed`; note `matcher` and `two_pass_mass_cal` are *not* recorded in `params`), +a `stats` map (`psms`, and `targets_at_q` whose key is the float- +formatted threshold, e.g. `targets_at_q0.01`), `model_identity = +"native-seed-hyperscore-v1"`, and `elapsed_ms`. + +## How it works + +Entry point: `search_seed::run(SearchSeedParams)` (`search_seed.rs:45-283`). + +**1. Load** the library (`Library::load`, `index.rs:54`) and MS2 scans +(`load_ms2`, `spectra.rs:20`), then log candidate and scan counts +(`search_seed.rs:47-53`). `load_ms2` sorts the returned `Vec` by +`rt_seconds` ascending (`spectra.rs:95`); this RT ordering is what makes the +within-group strictly-greater update deterministic (earliest-RT wins a tie). It +does **not** re-sort each scan's peaks: peak m/z order is inherited from `convert` +(see the mass-cal invariant below). + +**2. Build the matcher.** When `cfg.matcher == Fragindex` (the default), a +`FragIndex` is built once over the whole library at `cfg.fragment_tol_ppm` +(`search_seed.rs:56-57`; `FragIndex::build`, `fragindex.rs:46`). This is a +log-space-binned CSR inverted fragment index: postings are scattered in +candidate-id order so `post_cand` is ascending within every bin, which lets +`probe_peak` narrow to the precursor-window candidate sub-range by binary search +(`fragindex.rs:152-182`). The bucketed backend uses the Library's own inverted +index via `page_search` and needs no separate build. + +**3. Best-per-candidate accumulation.** Two paths produce the same +`HashMap` where `Best { score, rt, matched, scan_index }` +(`search_seed.rs:37-43`): + +- *Fragindex path* (`seed_fragindex_windows`, `search_seed.rs:313-410`): scans + are grouped by isolation window, keyed on + `(lower_mz.to_bits(), upper_mz.to_bits())` in a `BTreeMap` for deterministic + group order (`:321-330`). Each scan belongs to exactly one window, so groups + are independent parallel units; `rayon` `par_iter().map_init(SeedScratch::new)` + processes them (`:334-390`). Per group, `candidate_range` is computed once + (`:343`). Per scan, `select_peaks` picks the probe set, `SeedScratch::accumulate` + probes each peak and fuses `(count, obs_sum)` per touched candidate + (`fragindex.rs:214-237`), candidates with `count >= min_matched_peaks` are + scored by `hyperscore`, sorted by score desc then candidate-id asc, truncated to + `report_psms`, and folded into a group-local best with a strictly-greater update + (`:357-385`). +- *Bucketed path* (serial, `search_seed.rs:66-107`): per scan, `candidate_range` on + the Library, `select_peaks`, then `page_search` for each probed peak accumulates + `(count, obs_sum)` into a `HashMap`. Same `min_matched_peaks` filter, hyperscore, + sort, `report_psms` truncation, and best-per-candidate update. + +Both paths skip a scan whose candidate range is empty (`hi <= lo`, +`search_seed.rs:69-71` / `:344-346`), so out-of-library-range windows cost nothing. +Both accumulate `obs_sum` as the summed **observed** peak intensity of matched +postings; the predicted fragment intensity is deliberately discarded in the seed +(`fragindex.rs:185-190`, and the `_pi`/`_mz` discards at `search_seed.rs:77`). + +The two backends do **not** use the same tolerance-edge predicate, so their matched +sets (and therefore ID counts) can differ slightly on the same data. The fragindex +`probe_peak` verifies with `within_ppm` (min-relative, symmetric about the smaller +mass, f64; `constants.rs:92`), while the bucketed `page_search` matches inside +`ppm_bounds(peak, tol)` (query/observed-relative, f32-truncated bounds; +`index.rs:258-282`). The mass-cal deviation collection uses a third convention, +`ppm_diff` (theoretical/predicted-relative; `constants.rs:66`). Do not assume the +three are interchangeable at the edge; `config.rs:30-35` records that the +predicate difference shifts IDs more on the AIF full-range-window case. + +**Hyperscore** (`search_seed.rs:413-415`): + +``` +hyperscore = ln(matched!) + ln(1 + sum_obs) +``` + +`ln(matched!)` is `ln_factorial(matched)` = the summed logs of `2..=matched` +(`fdr.rs:137-143`), rewarding fragment breadth; `ln(1 + sum_obs)` adds a bounded +intensity term. This is the Sage-style form and is intentionally simple because +the seed is a calibration pass, not the scored identification. + +**4. Cross-group merge** (fragindex path, `search_seed.rs:392-409`): a total-order +merge over the per-group partials keeps, per candidate, `max score`, ties broken by +`earliest rt`, then `min scan_index`. This is order-independent, so it is +bit-identical to the serial global best regardless of thread or group scheduling. + +**5. q-values.** Rows are collected and sorted by `candidate_id` +(`search_seed.rs:111-112`), a `(score, is_decoy)` vector is built, and +`target_decoy_q` (`fdr.rs:7-52`) computes per-candidate q. That routine sorts by +score descending, walks tied-score blocks together so every PSM in a block gets +the same q, uses the conservative numerator `q = (n_decoys + 1) / max(1, n_targets)`, +and monotonizes from worst to best score so q is non-increasing. The output column +is `spectrum_q`. `count_targets_at_q(q, is_decoy, fdr_seed)` (`fdr.rs:115`) reports +the confident target count into `stats` (`search_seed.rs:148`). + +**6. Fragment mass recalibration** (`search_seed.rs:150-231`). A +`scan_index -> &Ms2Scan` map is built (`:155-158`). For every **confident target** +PSM (`!is_decoy && q <= fdr_seed`, `:161`), each library fragment m/z of that +candidate (`lib.cand_frags(cid)`, `index.rs:209`) is searched against the best +scan's peaks inside a **hardcoded 50 ppm window** (`ppm_bounds(fmz, 50.0)`, +`:167`; `partition_point` finds the low edge, then a linear scan to the high edge). +This `partition_point` + linear scan **assumes `scan.peaks` is m/z-ascending**; +`load_ms2` does not re-sort peaks by m/z (`spectra.rs:20-97`), so the assumption +rests on `convert` writing peaks in m/z order. The nearest peak by absolute m/z +distance contributes one signed ppm deviation (`ppm_diff(peak_mz, fmz)`, +`constants.rs:66`) to `devs` (`:160-184`). Every library fragment of the candidate +is probed, not only those that matched during scoring, so a fragment that found no +scoring posting can still supply a calibrant. The 50 ppm net is deliberately wider +than `fragment_tol_ppm` so a systematic offset larger than the tolerance can still +be measured. + +The fit closure (`:187-194`) takes deviations, sorts them, takes the median as the +offset (`sorted[len/2]`, the upper-middle element for even-length inputs, not the +two-element average), then sets `tol = max(5.0, 1.5 * P95(|dev - offset|))` using +`calibrate::percentile(.., 0.95)` (nearest-rank on `round(p*(n-1))`, +`calibrate.rs:156-164`). Control flow (`:195-216`): + +- `devs.len() < 20`: fallback, `(0.0, fragment_tol_ppm, cal_passes = 0)` (no + offset, tolerance unchanged). +- otherwise single pass: `(o1, t1, 1)`. +- if `two_pass_mass_cal`: keep only deviations inside `|dev - o1| <= t1` and, when + `>= 20` survive, re-fit to `(o2, t2, 2)`; otherwise keep the single-pass result. + The second pass rejects random-match outliers so they cannot bias the median. + +The result is written to `.masscal.json` (`:217-227`). + +**7. Write** `seed_psms.parquet` (`write_table`, `:233`) and the `ArtifactReport` +(`:256-274`), then log `psms`, `confident`, `elapsed_ms`. + +### How the outputs are consumed + +`run` writes the seed to `seed_psms.parquet` (`run.rs:218-235`) and passes +`mass_cal: Some(".masscal.json")` to `extract` (`run.rs:309`). In +`extract` (`extract.rs:678-698`), the JSON is read: `frag_ppm_offset` becomes +`offset_factor = 1.0 + frag_ppm_offset * 1e-6` (`extract.rs:698`), which divides +each observed peak m/z before matching (`q_mz = peak.mz / offset_factor`, e.g. +`extract.rs:303`/`:740`) to bring observed peaks into the predicted-fragment +frame, and `frag_tol_ppm` replaces `extract.frag_tol_ppm` as the matching +tolerance (falling back to the config value if the file is absent, +`extract.rs:696`). + +## Key types and functions + +| name | file:line | what it does | +|---|---|---| +| `run` | `search_seed.rs:45` | stage entry point; orchestrates load, score, q, masscal, write | +| `SearchSeedParams` | `search_seed.rs:27` | input struct (`ms2`, `library_precursors`, `library_fragments`, `out` output-path prefix, `cfg`, `bucket_size`, `config_hash`) | +| `Best` | `search_seed.rs:37` | per-candidate best `{ score, rt, matched, scan_index }` | +| `seed_fragindex_windows` | `search_seed.rs:313` | parallel per-window fragindex scoring + deterministic merge | +| `select_peaks` | `search_seed.rs:288` | top-N-by-intensity peak selection, re-sorted to index order | +| `hyperscore` | `search_seed.rs:413` | `ln(matched!) + ln(1 + sum_obs)` | +| `FragIndex::build` | `fragindex.rs:46` | build CSR inverted index at a fixed tolerance | +| `FragIndex::probe_peak` | `fragindex.rs:152` | matched postings for one peak in a candidate range | +| `SeedScratch::accumulate` | `fragindex.rs:214` | epoch-stamped fused `(count, obs_sum)` accumulation | +| `Library::candidate_range` | `index.rs:238` | half-open candidate id range for an isolation window | +| `Library::page_search` | `index.rs:247` | bucketed inverted-index probe (bucketed backend) | +| `Library::cand_frags` | `index.rs:209` | `(m/z, intensity, name)` slices for a candidate | +| `target_decoy_q` | `fdr.rs:7` | tied-block, monotonized `(n_decoys+1)/max(1,n_targets)` q | +| `count_targets_at_q` | `fdr.rs:115` | target count at or below a q threshold | +| `ln_factorial` | `fdr.rs:137` | `ln(n!)` via summed logs | +| `ppm_diff` / `ppm_bounds` | `constants.rs:66` / `:78` | signed ppm (theoretical-relative) and ppm window bounds (query-relative) | +| `within_ppm` | `constants.rs:92` | min-relative tolerance predicate used by the fragindex match (differs at the edge from the two above) | +| `load_ms2` | `spectra.rs:20` | reads `spectra_ms2.parquet` into `Vec`, RT-sorted | +| `percentile` | `calibrate.rs:156` | nearest-rank percentile (used for the tolerance) | + +## Configuration + +`SearchSeedConfig` (`config.rs:286-320`, `#[serde(default, deny_unknown_fields)]`). +The struct was pruned to the fields actually read here; unknown keys are rejected +on load. + +| field | default | effect | +|---|---|---| +| `fdr_seed` | `0.01` | q threshold defining "confident" for the calibration subset and the reported `targets_at_q` stat | +| `fragment_tol_ppm` | `20.0` | matching tolerance for scoring; also the masscal fallback tolerance when too few deviations | +| `report_psms` | `5` | max candidates kept per spectrum before the best-per-candidate fold (wide-window DIA) | +| `min_matched_peaks` | `4` | minimum matched fragments for a candidate to score in a scan | +| `top_n_peaks` | `300` | probe only the N most intense peaks per scan (`0` = all); seed-only, does not shrink the extract artifact | +| `matcher` | `Fragindex` | backend; `MatcherKind::Bucketed` (`config.rs:38`) uses the serial `page_search` path | +| `two_pass_mass_cal` | `false` | robust second-pass mass fit on the in-window inliers (sensitivity_plan P3.1) | + +`bucket_size` for the bucketed Library index is taken from `cfg.extract.bucket_size`, +not from `SearchSeedConfig` (`run.rs:225`). The 50 ppm mass-calibration search +window, the `min 5.0 ppm` tolerance floor, the `1.5 * P95` scale, and the `>= 20` +deviation threshold are hardcoded in `search_seed.rs` (`:167`, `:192`, `:195`), +not config fields. + +The standalone subcommand `Cmd::SearchSeed` (`main.rs:75-86`, dispatch `:458-476`) +takes `--ms2`, `--library-precursors`, `--library-fragments`, `--out`, and +`--config` (all `String`; `--config` optional). There is no `--bucket-size` flag; +`bucket_size` is read from the resolved config's `extract.bucket_size` +(`main.rs:473`). + +## Invariants, determinism, gotchas + +- **Determinism** (PLAN.md Section 7): the fragindex parallel path is bit-identical + to the *serial fragindex* best-per-candidate (this is a parallel-vs-serial claim + about the same backend, not a fragindex-vs-bucketed claim; the two backends use + different edge predicates, see the accumulation section). Groups run in `BTreeMap` + key order, within a group scans run in RT-ascending order (guaranteed by + `load_ms2`'s `sort_by rt_seconds`, `spectra.rs:95`) with a strictly-greater + update, and the cross-group merge is a total order (`max score`, tie earliest RT, + tie min scan_index; `search_seed.rs:392-409`). `select_peaks` re-sorts the top-N + back to index-ascending (`:299`) so the `obs_sum` float reduction is summed in a + fixed order. The `HashMap` is only ever reduced through this total order, never + iterated for a float sum. +- **Best-per-candidate, not best-per-spectrum.** One output row per candidate that, + in at least one scan, cleared `min_matched_peaks` **and** ranked within that + scan's top `report_psms` (the per-scan sort is truncated before the fold). A + candidate below `report_psms` in every scan it appears in is dropped. Ties in the + update use strictly-greater (`score > entry.score`, `:97` and `:377`), so the + earliest-RT scan wins a tie. +- **Library decoys only.** `label` comes straight from `Candidate::is_decoy` + (`:138`); the stage never mints decoys. The target-decoy null therefore depends on + the library carrying paired decoys (see the DIA-NN library recipe / `digest`). +- **obs_sum is observed intensity.** The seed drops predicted fragment intensity on + purpose (`fragindex.rs:185-190`); do not "fix" the discarded `_pi`/`_mz`. +- **Mass-cal fallback is an identity.** With `< 20` deviations the offset is `0.0` + and the tolerance is left at `fragment_tol_ppm` (`cal_passes = 0`); extract then + applies no shift. `frag_ppm_sigma` is an exact copy of `frag_tol_ppm`. +- **The 50 ppm mass-cal window is fixed** and independent of `fragment_tol_ppm`; it + is intentionally wide so a systematic error larger than the scoring tolerance is + still observable. Do not tie it to `fragment_tol_ppm`. +- **Mass-cal assumes m/z-sorted peaks.** The deviation search uses + `scan.peaks.partition_point(|pk| pk.mz < lo)` + a linear scan to the high edge + (`search_seed.rs:168-178`), which is only correct if `scan.peaks` is m/z-ascending. + `load_ms2` sorts scans by RT but never re-sorts peaks (`spectra.rs:20-97`), so the + invariant is inherited from `convert`. The fragindex scoring path does not need + it (it bins each peak independently); only mass-cal relies on it. +- **`config_hash` is carried but unused** inside `run` (part of `SearchSeedParams` + for call-site uniformity, `search_seed.rs:34`). In the standalone CLI it is + `blake3_str(cfg.canonical_json())` (`main.rs:466`); in the `run` orchestrator it + is the shared chain hash `ch` (`run.rs:226`). Either way the report's + `content_hash` is the blake3 of the output Parquet, not this value. +- **`top_n_peaks` is seed-only.** It caps the probe set to reduce index probing on + the dominant cost, but abundant peptides supply the calibration anchors anyway; + the downstream `extract` stage still sees all converted peaks. +- **`MatcherKind::Bucketed` stays serial** (`search_seed.rs:66-107`); only the + fragindex path is parallelized. +- **iRT is inert here.** `predicted_irt` is only copied to the output, so the seed + can be computed once and reused across the DeepLC fine-tune boundary + (`run.rs:237-280`). +- **Test coverage.** Only `select_peaks` has a unit test here + (`zero_selects_all_and_seed_cap_keeps_only_top_intensity_peaks`, + `search_seed.rs:417-450`), asserting `top_n = 0` returns all indices and + `top_n = 300` over 305 peaks keeps only the top-intensity `5..305` re-sorted to + index order. `target_decoy_q` (tied-block, `+1` conservatism) and `entrapment_q` + are tested in `fdr.rs` (`:145-207`); `ln_factorial` has no direct test. The + fragindex match, epoch reset, precursor gate, and the naive-equivalence gate are + tested in `fragindex.rs` (`:280-451`). + There is no stage-level test for `search_seed::run`, the masscal output, or + bucketed-vs-fragindex equivalence at the stage level (see CLAUDE.md "test gaps"). + +## How to extend / modify + +- **Swap in a real Sage / sage-core scorer.** Replace the accumulation in `run` + behind the existing file contract. Keep the `seed_psms` schema and + `masscal.json` keys unchanged so `rt-im-train` and `extract` need no edits. Bump + `model_identity` (`search_seed.rs:271`) so provenance in `report.json` is honest. +- **Add a `seed_psms` column.** Bump the schema version at `schema.rs:15` + (`("seed_psms", 1)` -> `2`), push the new `Col` in the `write_table` call + (`search_seed.rs:233-250`), and update any consumer that reads by column name. +- **Make the mass-cal window / thresholds configurable.** The 50 ppm search + window, the `>= 20` minimum, the `1.5x` scale, and the `5.0 ppm` floor are + literals in `run`; promote them to `SearchSeedConfig` fields (with conservative + defaults) if you need to tune them, per the "every algorithmic choice is a typed + config field" convention. +- **Add a matcher backend.** Add a `MatcherKind` variant (`config.rs:38`) and a + branch in the `if let Some(idx) = fidx` dispatch (`search_seed.rs:63-108`); mirror + the deterministic merge if the new path is parallel. +- **Charge-2 / m/z-binned tolerance.** The current fit produces one global offset + and tolerance. To make them charge- or m/z-dependent, partition `devs` before the + `fit` closure and emit per-bin entries in `masscal.json`, then teach `extract` to + pick the matching bin. +- **Change the score.** `hyperscore` (`search_seed.rs:413`) is a free function; keep + it monotone in matched-fragment count and observed intensity so the target-decoy q + ordering stays meaningful, and keep the summation order fixed for determinism. diff --git a/docs/08_rt_im_train.md b/docs/08_rt_im_train.md new file mode 100644 index 0000000..b19c074 --- /dev/null +++ b/docs/08_rt_im_train.md @@ -0,0 +1,477 @@ +# rt-im-train (Stage B): RT calibration + DeepLC fine-tune + +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +Stage B (`mumdia rt-im-train`, PLAN.md Stage B) turns the run-independent +predicted iRT carried on each library candidate into a per-run predicted +retention time in seconds, and derives a per-candidate RT acceptance window that +the extractor uses to bound its scan search. It is the bridge between the +library (built once, run-independent) and this run's chromatography. + +The stage does two things: + +1. Fit a calibration map `predicted_irt -> observed_rt` from the confident seed + PSMs of this run (a linear least-squares fit whenever at least two anchors + exist; a LOESS local-linear smoother when configured and enough anchors are + present). With fewer than two target anchors no trustworthy mapping exists, + so the calibrated RT is marked unavailable and the windows are left unbounded. +2. Set an RT half-window from the residual distribution of that fit + (residual percentile times a multiplier), and emit `rt_lo`/`rt_hi` around the + calibrated RT of every library candidate. + +An optional pre-step, wired into the `run` orchestrator rather than into this +stage, fine-tunes the DeepLC multitask model on the same confident seed PSMs and +writes a new precursor-library table with updated `predicted_irt` before Stage B +reads it. The input library is unchanged. The fine-tune is a Python sidecar and +is nondeterministic. It is default-off. + +Ion mobility (IM) is stubbed. The MVP is 3D, so the IM columns are always +written null; there is no IM calibration or IM window. + +## Files + +| path | role | +|---|---| +| `rust/mumdia/crates/mumdia/src/stages/rt_im_train.rs` | The stage. Joins iRT to seed PSMs, fits calibration, computes windows, writes `run_windows.parquet` + `cal.json`. | +| `rust/mumdia/crates/mumdia/src/calibrate.rs` | Calibration math: `linear_fit`, the `Loess` local-linear smoother, and `percentile`. Shared, no external deps. | +| `rust/mumdia/crates/mumdia/src/stages/run.rs` | Orchestrator. Runs the optional DeepLC fine-tune between `search-seed` and this stage, then calls `rt_im_train::run` (see `run.rs:242-291`). | +| `rust/mumdia/crates/mumdia/src/sidecar.rs` | `run_deeplc_finetune` (sidecar.rs:110-155): the file-contract client that invokes the fine-tune worker. | +| `scripts/deeplc_finetune.py` | The fine-tune worker: transfer-learns DeepLC 4.0 on the seed and writes a new library parquet with replaced `predicted_irt`. | +| `rust/mumdia/crates/mumdia-core/src/config.rs` | `RtImTrainConfig` (config.rs:324-367), `CalibrationMethod` enum (config.rs:56-61), and the load-time validation that rejects `calibration_method=none` (config.rs:1030-1036). | +| `rust/mumdia/crates/mumdia-core/src/schema.rs` | `artifact::RUN_WINDOWS = ("run_windows", 1)` (schema.rs:16). | + +## Inputs and outputs + +### Consumed + +**`fragment_library_precursors.parquet`** (the run-independent library; in +library-input mode this is the imported speclib, and when `finetune_deeplc` is +set it is the `_ft` rewrite). Columns read (rt_im_train.rs:78-79): + +| column | type | use | +|---|---|---| +| `candidate_id` | u32 | join key; also the output row identity | +| `predicted_irt` | f32 | the value calibrated to observed RT | + +The library is the single source of truth for iRT (rt_im_train.rs:75-83): both the +training anchors and the applied calibration read `predicted_irt` from this same +table, so a patched or fine-tuned library iRT is used consistently for fit and +apply. + +**`seed_psms.parquet`** (from `search-seed`). Columns read (rt_im_train.rs:88-93): + +| column | type | use | +|---|---|---| +| `candidate_id` | u32 | join to library `predicted_irt` | +| `base_peptide_id` | u32 | best-per-peptide grouping key | +| `spectrum_q` | f64 | confidence gate (`< q_train`) | +| `score` | f64 | picks the best PSM within a `base_peptide_id` | +| `observed_rt` | f64 | the calibration target (seconds) | +| `label` | str | only rows equal to `"target"` anchor the fit | + +### Produced + +**`run_windows.parquet`** (`artifact::RUN_WINDOWS`, version 1). One row per +library candidate. Columns (rt_im_train.rs:266-277): + +| column | type | meaning | +|---|---|---| +| `candidate_id` | u32 | library candidate identity | +| `rt_pred_cal` | f64 | calibrated predicted RT (seconds); `NaN` when calibration is unavailable (fewer than two anchors) | +| `rt_lo` | f64 | `rt_pred_cal - width` (window lower bound, seconds); negative infinity when calibration is unavailable | +| `rt_hi` | f64 | `rt_pred_cal + width` (window upper bound, seconds); positive infinity when calibration is unavailable | +| `im_pred_cal` | f64, nullable | always `None` (3D MVP) | +| `im_lo` | f64, nullable | always `None` | +| `im_hi` | f64, nullable | always `None` | + +The unbounded row `(NaN, -inf, +inf)` is materialized by `candidate_window` +(rt_im_train.rs:65-70) whenever no calibrated RT or window width is available; the +infinite bounds make the downstream extractor scan the full isolation-window RT +range (recall-safe) rather than a numeric window. + +**`cal.json`** (side artifact, written with `mumdia_io::json::write_json`, +rt_im_train.rs:290-302). Fields: + +| field | meaning | +|---|---| +| `method` | `"loess"` if the LOESS path was used, `"linear"` if the linear map was used, or `"unavailable"` when fewer than two anchors exist (rt_im_train.rs:279-285) | +| `slope`, `intercept` | the linear fit coefficients, computed only when at least two anchors exist and therefore serialized as `null` when calibration is unavailable (rt_im_train.rs:286-287) | +| `w_rt` | the global RT half-window (seconds), `null` when the windows are unbounded | +| `p_rt` | the residual percentile used | +| `multiplier` | `rt_window_multiplier` | +| `n_train` | number of calibration anchors | +| `calibration_status` | `"loess"`, `"linear"`, `"fallback_fixed"`, or `"insufficient_anchors_unbounded"` | + +**`.report.json`** (`ArtifactReport`, rt_im_train.rs:309-321) records +`logical_name`/`schema_name`/`schema_version` (all from `artifact::RUN_WINDOWS`), +`stage = "rt-im-train"`, rows, the blake3 content hash of `run_windows.parquet`, +params (`q_train`, `p_rt`, `method` as the `Debug` form of `calibration_method`), +stats (`n_train`, `w_rt`, `calibration_status`), and `elapsed_ms`. `model_identity` +is `None` (this stage applies no serialized model, rt_im_train.rs:318). Params +`stats` is a `BTreeMap`, so its key order is stable. + +## How it works + +The stage entry point is `run(p: RtImTrainParams)` (rt_im_train.rs:72). Standalone +it is invoked as `mumdia rt-im-train --seed-psms

--library-precursors

+--out-windows

--out-cal

[--config

]` (the `RtImTrain` subcommand, +main.rs:88-99, dispatched at main.rs:477-494); the `run` orchestrator calls the +same `run` function directly (run.rs:284-291). Both call sites build the config +hash and pass it as `config_hash`, but the stage never reads it (see gotchas). + +### 1. Join iRT to candidates + +Read the library precursors and build `irt_by_cid: HashMap` mapping +`candidate_id -> predicted_irt` (rt_im_train.rs:80-83). This is the only source of +iRT; both training and application use it. + +### 2. Select calibration anchors + +Read the seed PSMs and build `best_per_pep: HashMap`, +keyed by `base_peptide_id` (rt_im_train.rs:95-120). A seed row enters the pool only +if: + +- `spectrum_q`, `score`, and `observed_rt` are all finite and + `spectrum_q < q_train` (rt_im_train.rs:97-103); a non-finite field or a row at + or above the threshold is skipped, and +- `label == "target"` (rt_im_train.rs:106); a decoy anchor would inject a random + iRT/RT pair into the fit, and +- its `candidate_id` resolves to a finite library `predicted_irt` + (rt_im_train.rs:109-113); a missing or non-finite library iRT is skipped. + +Within a `base_peptide_id`, the highest-`score` PSM wins (rt_im_train.rs:117-118), +so each confident peptide contributes exactly one `(predicted_irt, observed_rt)` +anchor. `sorted_anchor_vectors` (rt_im_train.rs:54-60) then sorts the anchors by +`base_peptide_id` and splits them into the parallel vectors `train_irt`/`train_rt` +(rt_im_train.rs:121), with `n_train = train_irt.len()` (rt_im_train.rs:122). Sorting +before any float reduction makes the fit order-stable (see the determinism note in +the gotchas). + +### 3. Fit the calibration map + +Calibration is attempted only when `calibration_available = n_train >= 2` +(rt_im_train.rs:127): zero or one point cannot define a useful mapping across the +gradient. When available, the linear fit is computed: `let (slope, intercept) = +linear_fit(&train_irt, &train_rt)` (rt_im_train.rs:128-132); otherwise `slope` and +`intercept` are both `NaN`. LOESS is used only when calibration is available, the +method is `CalibrationMethod::Loess`, and there are at least +`min_seed_for_calibration` anchors (rt_im_train.rs:133-135); it is fit with +`Loess::fit(&train_irt, &train_rt, loess_span, 200)` (200 grid points, +rt_im_train.rs:136-140). + +`predict` is a closure (rt_im_train.rs:142-150): it returns `NaN` when calibration +is unavailable, otherwise `loess.predict(irt)` when a LOESS model exists, else +`slope * irt + intercept`. The linear coefficients are therefore both the primary +map (Linear method) and the extrapolation fallback (LOESS outside its training +range; see below). + +The math: + +- **`linear_fit`** (calibrate.rs:6-28) is ordinary least squares. With + `n = xs.len()`: `slope = (n*Sxy - Sx*Sy) / (n*Sxx - Sx*Sx)` and + `intercept = (Sy - slope*Sx) / n`. Degenerate guards: fewer than 2 points + returns `(0, mean(ys))` (a constant map, calibrate.rs:8-16); a near-zero + denominator (all `x` equal) returns `(0, Sy/n)` (calibrate.rs:22-24). +- **`Loess::fit`** (calibrate.rs:42-83) first computes the global linear fit as + the extrapolation fallback (calibrate.rs:43), sorts the points by `x` + (calibrate.rs:45-48), and if fewer than 4 points are present just fills the grid + from the linear line (calibrate.rs:50-66). Otherwise the local window size is + `k = clamp(ceil(span * n), 3, n)` (calibrate.rs:67) and it evaluates a + local-linear regression on a uniform grid of `grid_n` points spanning + `[min(x), max(x)]` (calibrate.rs:69-76). +- **`local_linear`** (calibrate.rs:107-153) selects the `k` nearest anchors by + walking outward from the insertion point (calibrate.rs:110-127), assigns each a + tricubic weight `w = (1 - d^3)^3` with `d = |x_i - x0| / dmax` (calibrate.rs: + 133-139), and solves a weighted least-squares line, returning the fitted value + at `x0`. `dmax` is the larger of the two window-edge distances, floored at + `1e-12` to avoid a zero divide (calibrate.rs:128-130), and the weight is exactly 0 + when `d >= 1.0` (calibrate.rs:134-139), so anchors past the window edge do not + contribute. If the weighted system is degenerate, meaning `sw < 1e-12` (all + weights vanished) or `sw*swxx - swx^2` near zero (calibrate.rs:146-149), it + falls back to the global line. +- **`Loess::predict`** (calibrate.rs:87-102) uses the linear fallback for `x` at or + outside the grid ends, and also when the grid has fewer than 2 nodes + (calibrate.rs:89-94), and otherwise linearly interpolates between the two + bracketing grid nodes found by `partition_point` (calibrate.rs:95-101). When the + two bracketing nodes are within `1e-12` in `x` it returns the lower node's `y` + rather than interpolating, guarding against a zero divide (calibrate.rs:98-99). + Interpolating a precomputed grid makes bulk application over the whole library + cheap. + +### 4. Derive the RT window + +`min_anchors = min_seed_for_calibration.max(2)` (rt_im_train.rs:157). The half-window +is then chosen by `window_plan(n_train, min_anchors, fallback_rt_window_s)` +(rt_im_train.rs:42-50, called at 158-159), which returns one of three `WindowPlan` +variants (rt_im_train.rs:30-40): + +- **`Unbounded`** (`n_train < 2`): no trustworthy iRT to RT mapping exists. The + stage warns, sets `w_rt = None`, and uses status + `"insufficient_anchors_unbounded"` (rt_im_train.rs:160-167). Every candidate then + gets the unbounded window `(NaN, -inf, +inf)` so extraction scans the full + isolation-window RT range. +- **`Fixed(fallback_rt_window_s)`** (`2 <= n_train < min_anchors`): a linear map + exists but there are too few anchors to estimate its residual distribution. The + stage warns and retains the configured broad fixed half-window with status + `"fallback_fixed"` (rt_im_train.rs:168-175). +- **`Calibrated`** (`n_train >= min_anchors`): the absolute residuals + `|observed_rt - predict(irt)|` are formed (rt_im_train.rs:177-181), and the global + half-window is `w_rt = percentile(resid, p_rt) * rt_window_multiplier`, floored at + 1.0 second (rt_im_train.rs:182). The status is `"loess"` or `"linear"` + (rt_im_train.rs:183). + +The anchor-count gate exists for a concrete failure mode documented in the code +(rt_im_train.rs:152-156): with only a handful of anchors a linear fit passes almost +exactly through them, so residuals are near zero, the percentile window collapses +to the 1-second floor, and that floor then discards nearly every true co-elution +downstream. The fixed fallback avoids that collapse; the unbounded case below two +anchors avoids fitting a mapping from a single point at all. + +### 5. Optional adaptive per-region window (default off) + +When `adaptive_rt_window` is set (and `n_train >= min_anchors`, which guarantees the +`Calibrated` plan and therefore a non-null `w_rt`), the global `w_rt` is replaced by +a per-bin half-width (rt_im_train.rs:192-229). Anchors are binned into +`nb = adaptive_rt_bins.max(1)` equal-width bins over the calibrated-RT range +(rt_im_train.rs:202), with the bin index computed from +`frac = ((cal - rt_min)/span).clamp(0.0, 0.999_999)` so the maximum RT lands in +the last bin rather than out of range (rt_im_train.rs:207). Each bin's +half-width is its local residual percentile times the multiplier, clamped to +`[lo_clamp, hi_clamp]` where `lo_clamp = rt_window_min_s.max(0.0)` and +`hi_clamp = fallback_rt_window_s.max(lo_clamp)` (rt_im_train.rs:210-211). Empty +bins fall back to the global `w_rt` (rt_im_train.rs:215-216). Each candidate then +takes the width of the bin its calibrated RT lands in (rt_im_train.rs:248-253). +Degenerate case: when all calibrated anchor RTs are equal (`rt_max <= rt_min`) the +adaptive block yields `None` and the stage silently uses the global `w_rt` +(rt_im_train.rs:203, 224-226). The rationale (config.rs:353-361): a single fixed +window is simultaneously too wide for well-calibrated regions and too narrow for +poorly-calibrated ones. This knob is part of the sensitivity program and has not +passed the entrapment gate, so it stays default-off. + +### 6. Apply to every candidate and write + +For each library candidate (rt_im_train.rs:246-264): `calibrated_rt = +calibration_available.then(|| predict(irt))` (rt_im_train.rs:247), the width is +either the adaptive per-bin value or the global `w_rt` (rt_im_train.rs:248-255), and +`candidate_window(calibrated_rt, width)` (rt_im_train.rs:256) produces the row +`(cal, cal - width, cal + width)`, or the unbounded `(NaN, -inf, +inf)` when either +value is absent. The three IM columns are pushed as `None` (rt_im_train.rs:261-263). +The table is written (rt_im_train.rs:266-277), `cal.json` is written +(rt_im_train.rs:290-302), and the artifact report is emitted (rt_im_train.rs:309-321). + +### DeepLC multitask fine-tune (orchestrator pre-step, default off) + +This is not part of `rt_im_train::run`; it runs in the `run` orchestrator between +`search-seed` and Stage B, guarded by `cfg.rt_im_train.finetune_deeplc` +(run.rs:242-280). Preflight (`run.rs:62-67`) rejects `finetune_deeplc` unless +`predict_frag.deeplc_python` names a Python interpreter with DeepLC 4.0 multitask. +The seed PSMs are searched once on the base library before the fine-tune and are +reused as-is (the search-seed hyperscore does not depend on iRT), so the fine-tune +and Stage B both consume that same seed table; only the library's `predicted_irt` +changes in the newly written `_ft` table between them. + +When enabled, the orchestrator resolves `deeplc_finetune.py` +(`sidecar::resolve_script`, run.rs:248-251), writes a fine-tuned library +`fragment_library_precursors_ft.parquet` (run.rs:252), and rebinds `lib_p` to that +file so both Stage B and `extract` read the fine-tuned iRT (run.rs:242-280). The +fine-tune is driven through `run_deeplc_finetune` (sidecar.rs:110-155), whose +positional CLI contract is `deeplc_finetune.py --epochs E +--patience P --q-train Q --batch B`. The worker runs with both `PYTHONUTF8=1` and +`PYTHONIOENCODING=utf-8` set (the `utf8` flag on `run_worker`, sidecar.rs:217-225) +because DeepLC/Keras crash on the Windows cp1252 console. `run_worker` spawns +`python script arg...` and returns an error if the process exits non-zero +(sidecar.rs:226-232); there is no JSON request file, only argv plus the parquet +column contract. + +Inside the worker (`scripts/deeplc_finetune.py`): the reference set is the +confident target seed PSMs (`label == "target"`, `spectrum_q <= q_train`, standard +residues only via `is_std`, deeplc_finetune.py:100-104). Note the gate is `<=` +q_train here, whereas the Rust anchor selection uses strict `<` q_train +(rt_im_train.rs:100), so the two reference sets can differ by the boundary rows. +The worker keys the reference dict by `peptidoform` string (`ref[pf] = observed_rt`, +deeplc_finetune.py:99-104), so it joins seed to library by peptidoform, not by +`candidate_id` as the Rust stage does, and it keeps one RT per peptidoform by +last-write-wins in iteration order rather than best-score. The batch size +auto-scales when `--batch 0`: `min(512, max(16, n_ref // 30))`, so each epoch runs +at least about 30 gradient steps; a fixed large batch underfits small references +(deeplc_finetune.py:111-117). `deeplc.finetune(ref_psms, train_kwargs=...)` +transfer-learns the model (deeplc_finetune.py:119-129); predictions come from +`deeplc.predict(batch, model=ft_model)` over the unique standard peptidoforms in +chunks of 100_000 (deeplc_finetune.py:138-154). DeepLC 4.0 multitask returns a 2D +array (one column per task head); `agg` reduces it to one iRT by averaging across +heads (`a.mean(axis=1)`, deeplc_finetune.py:48-50, 151). Prediction is on the +DECOY_-stripped underlying sequence so decoys land on the same iRT scale as targets +(deeplc_finetune.py:44, 135-158). The rewritten column overwrites `predicted_irt` +in place in the output parquet (deeplc_finetune.py:156-159); peptidoforms with no +prediction, including non-standard ones, keep their original iRT +(`preds.get(base_pf(pf), orig[i])`, deeplc_finetune.py:156). + +The worker also carries a documented OpenMP crash fix: numpy's OpenBLAS (GNU +OpenMP) and torch's Intel OpenMP coexist only under `KMP_DUPLICATE_LIB_OK=TRUE`, +and each spawns a full thread pool that oversubscribes the CPU during the +sustained backward pass and crashes the machine, so the worker pins OMP/BLAS to 1 +thread and bounds torch to `DEEPLC_FT_THREADS` (default 8) before importing numpy +and torch (deeplc_finetune.py:6-13, 22-28, 82-91). `deeplc` is imported before +numpy for OpenMP load order (deeplc_finetune.py:32). Beyond the four flags the +engine passes, the worker accepts standalone-only flags the engine never sets, so +they take their defaults: `--device {cpu,cuda}` (cuda sidesteps the CPU OpenMP +crash entirely), `--threads`, `--max-ref`, `--predict-limit`, and `--skip-predict` +(deeplc_finetune.py:58-76). + +The fine-tune sets no torch/numpy seed, so it is nondeterministic across runs +(CLAUDE.md notes this). Its main use is library-input mode, where the base iRT is +the imported DIA-NN library value rather than a native/DeepLC prediction, and a +per-run fine-tune materially tightens the RT window. + +## Key types and functions + +| name | file:line | what it does | +|---|---|---| +| `RtImTrainParams` | rt_im_train.rs:19-26 | Input struct: seed PSMs path, library precursors path, output windows + cal paths, config, config hash. `config_hash` is a field but `run` never reads it (dead param; see gotchas). | +| `WindowPlan` | rt_im_train.rs:30-40 | Enum `{ Unbounded, Fixed(f64), Calibrated }`: the three window regimes by anchor count. | +| `window_plan` | rt_im_train.rs:42-50 | Maps `(n_train, min_anchors, fallback_width)` to a `WindowPlan`. | +| `sorted_anchor_vectors` | rt_im_train.rs:54-60 | Sorts the best-per-peptide map by `base_peptide_id` into `(train_irt, train_rt)` so the fit is order-stable. | +| `candidate_window` | rt_im_train.rs:65-70 | Builds `(cal, lo, hi)`; returns `(NaN, -inf, +inf)` when calibrated RT or width is absent. | +| `rt_im_train::run` | rt_im_train.rs:72-331 | The stage: join iRT, select anchors, fit, window, apply, write. | +| `linear_fit` | calibrate.rs:6-28 | OLS `y = slope*x + intercept` with degenerate-case guards. | +| `Loess` | calibrate.rs:31-37 | Grid-based local-linear smoother; carries a linear fallback for extrapolation. | +| `Loess::fit` | calibrate.rs:42-83 | Sorts anchors, builds a `grid_n`-point local-linear grid, `k = clamp(ceil(span*n),3,n)`. | +| `Loess::predict` | calibrate.rs:87-102 | Grid interpolation inside range, linear extrapolation outside. | +| `local_linear` | calibrate.rs:107-153 | Tricubic-weighted local least squares at one point. | +| `percentile` | calibrate.rs:156-164 | Nearest-rank percentile: sorts a copy, `rank = round(p.clamp(0,1)*(len-1))`. Not interpolated. Empty input returns 0.0. | +| `CalibrationMethod` | config.rs:56-61 | Enum `{ Loess, Linear, None }`; default `Loess`. `None` is rejected at load. | +| `RtImTrainConfig` | config.rs:324-367 | All Stage B config fields (below). | +| `run_deeplc_finetune` | sidecar.rs:110-155 | Sidecar client: `deeplc_finetune.py [flags]`. | + +## Configuration + +All fields live under `rt_im_train` in the config (`RtImTrainConfig`, +config.rs:324-367). The struct is `#[serde(default, deny_unknown_fields)]`, so any +unknown key is a hard load error and every field has the default below. The config +surface was pruned: there are no IM calibration fields (IM is a stub), and +`CalibrationMethod::None` is a foot-gun rejected at load (config.rs:1030-1036) even +though the enum variant still exists. + +| field | default | effect | +|---|---|---| +| `calibration_method` | `loess` | `loess`, `linear`, or `none`. `loess` uses the smoother when anchors suffice, else falls back to linear. `none` is rejected by `Config::validate` (config.rs:1030-1036) because the code path silently degrades to linear. | +| `q_train` | `0.01` | Max `spectrum_q` for a seed PSM to become a calibration anchor (rt_im_train.rs:100) and to enter the DeepLC fine-tune reference (`--q-train`). | +| `p_rt` | `0.95` | Residual percentile for the RT half-window (rt_im_train.rs:182). 0.95 keeps ~95% of anchor residuals inside the window. | +| `rt_window_multiplier` | `1.0` | Scales the residual-percentile window (rt_im_train.rs:182). Larger widens the window: higher recall, more interference. | +| `min_seed_for_calibration` | `50` | Minimum anchors to (a) use LOESS instead of linear (rt_im_train.rs:135) and (b) trust the residual window instead of the fixed fallback (via `min_anchors = max(this, 2)`, rt_im_train.rs:157). | +| `loess_span` | `0.3` | LOESS local-fit fraction; passed as `span` to `Loess::fit` (rt_im_train.rs:137). Fraction of anchors in each local window. | +| `fallback_rt_window_s` | `120.0` | Fixed half-window (seconds) when anchors are too few (`WindowPlan::Fixed`, rt_im_train.rs:168-175); also the upper clamp for adaptive bin widths (rt_im_train.rs:211). | +| `finetune_deeplc` | `false` | Enable the DeepLC multitask fine-tune pre-step in `run` (run.rs:242). Requires `predict_frag.deeplc_python` (preflight, run.rs:62-67). | +| `finetune_epochs` | `25` | Fine-tune epoch cap (`--epochs`); early stopping usually halts sooner. Used only when `finetune_deeplc`. | +| `finetune_patience` | `10` | Early-stopping patience (`--patience`): epochs without val-loss improvement before stopping. | +| `finetune_batch` | `0` | Fine-tune batch size (`--batch`). `0` auto-scales to the seed size so each epoch has ~30+ steps (config.rs:348-351); a fixed large batch underfits small seeds. | +| `adaptive_rt_window` | `false` | Per-region window instead of one global width (rt_im_train.rs:192-229). Sensitivity-program knob, not yet default-on. | +| `adaptive_rt_bins` | `12` | Number of equal-width calibrated-RT bins for the adaptive window (rt_im_train.rs:202). | +| `rt_window_min_s` | `1.0` | Lower clamp (seconds) for any adaptive half-window (rt_im_train.rs:210); mirrors the 1s floor on the global window. | + +## Invariants, determinism, gotchas + +- **IM is a stub.** `im_pred_cal`, `im_lo`, `im_hi` are always `None` + (rt_im_train.rs:261-263). There is no IM calibration, no IM window, and no IM + config field. Any 4D/diaPASEF work must add both the model and the columns. +- **Only targets anchor the fit.** Decoy seed PSMs are excluded + (rt_im_train.rs:106); admitting them would inject random iRT/RT pairs. Keep this + filter if you refactor anchor selection. +- **Library is the single iRT source.** Training and application both read + `predicted_irt` from the same library table (rt_im_train.rs:75-83, 232-233). When + the fine-tune writes its new table, the orchestrator rebinds `lib_p` to the + `_ft` file so both this stage and `extract` see the updated iRT. Do not + reintroduce a second iRT source or imply that the original file was mutated. +- **Fewer than two anchors leaves the windows unbounded.** With `n_train < 2` no + linear map is fit (`slope`/`intercept` are `NaN`), `w_rt` is `None`, and every + candidate gets `(rt_pred_cal = NaN, rt_lo = -inf, rt_hi = +inf)` with status + `"insufficient_anchors_unbounded"` (rt_im_train.rs:42-50, 160-167). Downstream this + is recall-safe: the extractor scans the full isolation-window RT range rather than + a numeric window. Do not treat `NaN`/infinite rows as a bug. The feature stage + neutralizes the `NaN` `rt_pred_cal` sentinel explicitly: `calibrated_rt_error` + (features.rs:271-277) returns a 0 RT error when either `apex_rt` or `rt_pred_cal` + is non-finite, so the sentinel never contaminates the feature matrix or the + preliminary competition score. +- **The 1-second floor.** The `Calibrated` global window is floored at 1.0s + (rt_im_train.rs:182). With too few anchors the fit is near-exact, residuals + collapse, and this floor would discard true co-elutions, which is exactly why the + `Fixed` fallback exists for `2 <= n_train < min_anchors` (rt_im_train.rs:152-186). + Do not remove the fallback branch. +- **`percentile` is nearest-rank, not interpolated** (calibrate.rs:156-164). It + sorts a copy each call, so it is order-independent, but repeated calls on the + same data re-sort. This is fine at Stage B sizes. +- **Calibration is order-deterministic (fixed).** `best_per_pep` is still a + `HashMap`, but `sorted_anchor_vectors` (rt_im_train.rs:54-60, called at 121) sorts + the anchors by `base_peptide_id` before building `train_irt`/`train_rt`, so + `linear_fit`'s summation order (calibrate.rs:17-20) is fixed across processes and + `slope`/`intercept`/`w_rt`/the calibrated RTs are reproducible. `Loess::fit` + additionally re-sorts by `x` internally (calibrate.rs:45-48). This satisfies + CLAUDE.md's determinism rule (ordered iteration where floats are summed); do not + reintroduce a `.values()`-order reduction. The test + `anchor_vectors_are_sorted_by_base_peptide_id` (rt_im_train.rs:337-352) guards it. +- **The fine-tune is explicitly nondeterministic.** `deeplc_finetune.py` sets no + torch/numpy seed (CLAUDE.md), so `finetune_deeplc = true` makes `predicted_irt`, + and therefore the whole run, non-reproducible. Treat it as an accuracy lever, not + a deterministic default. +- **`slope`/`intercept` are emitted in `cal.json` whenever calibration is available, + including under LOESS** (rt_im_train.rs:128, 286-287, 290-302). They are the LOESS + extrapolation fallback, not dead values; do not assume they were unused when + `method == "loess"`. They are serialized as `null` only when calibration is + unavailable (`n_train < 2`), since the fit is not computed in that case. +- **`CalibrationMethod::None` still exists but is rejected** at config load + (config.rs:1030-1036). The stage would otherwise fall through to the linear path + because `use_loess` matches only `Loess` (rt_im_train.rs:133-135). This fallthrough + is a known correctness wart (see CLAUDE.md "Correctness"). +- **Report schema version is 1** (`artifact::RUN_WINDOWS`, schema.rs:16). Bump it if + the column set changes. +- **`config_hash` is a dead param in this stage.** Both call sites build the blake3 + config hash and pass it as `RtImTrainParams.config_hash` (run.rs:290, + main.rs:492), but `run` never reads it (unlike stages that stamp it into their + report). Do not rely on the report to carry the config hash for Stage B. +- **Anchor gate is `<`, fine-tune gate is `<=`.** The Rust anchor selection keeps + seed rows with `spectrum_q < q_train` (rt_im_train.rs:100), while the DeepLC + fine-tune worker keeps `spectrum_q <= q_train` (deeplc_finetune.py:102). The two + confident-seed reference sets can therefore differ by the boundary rows. Keep this + in mind when reasoning about why the fine-tune reference count and the calibration + anchor count are not identical. +- **Fine-tune joins by peptidoform, Stage B joins by `candidate_id`.** The stage + maps seed to library iRT through `candidate_id` (rt_im_train.rs:109-113); the worker + maps seed observed RT to library peptidoforms through the `peptidoform` string + (deeplc_finetune.py:99-104, 156). A library whose peptidoform strings do not match + the seed's would silently fine-tune on nothing. +- **Test coverage.** `calibrate.rs` has three unit tests: `linear_recovers_line` + (calibrate.rs:170-176), `loess_tracks_nonlinear` (calibrate.rs:178-185), and + `percentile_basic` (calibrate.rs:187-191). The stage `rt_im_train.rs` now has three + unit tests covering the helper functions: `anchor_vectors_are_sorted_by_base_peptide_id` + (rt_im_train.rs:337-352), `sparse_anchor_policy_is_unbounded_only_below_two` + (rt_im_train.rs:354-365), and `unavailable_calibration_emits_unbounded_window_and_nan_prediction` + (rt_im_train.rs:367-378). The full `run` body (anchor selection over a real seed + table, the LOESS fit path, and the adaptive window) is still exercised only in full + runs. + +## How to extend / modify + +- **Add IM (4D).** Populate `im_pred_cal`/`im_lo`/`im_hi` (currently `None` at + rt_im_train.rs:261-263) from an IM calibration analogous to the RT path (an + IM2Deep-style model), and add IM config fields to `RtImTrainConfig`. The output + columns already exist and are nullable, so downstream reads survive the + transition. `extract` and the IM feature families must then consume them. +- **Change the calibration model.** Extend `CalibrationMethod` (config.rs:56-61) and + branch in the `predict` closure setup (rt_im_train.rs:133-150). Keep `linear_fit` as + the universal fallback so a degenerate anchor set never panics. New models belong + in `calibrate.rs` next to `Loess`, with unit tests like `loess_tracks_nonlinear` + (calibrate.rs:178-185). +- **Tune the window policy.** The window-derivation logic is localized at + rt_im_train.rs:152-229 (the `window_plan` match plus the optional adaptive block). + New window strategies (for example a symmetric-vs-asymmetric window, or a learned + per-charge width) should be config-gated and default-off, then validated against the + entrapment holdout before becoming a default, per the sensitivity program + (`sensitivity_plan/NEXT_STEPS.md`). +- **Swap the fine-tune worker.** The contract is purely the CLI and the parquet + columns (`sidecar.rs:107-155`, `scripts/deeplc_finetune.py`). A replacement worker + need only accept ` ` plus the four flags and rewrite the + `predicted_irt` column. Preserve the DECOY_-stripping behavior + (deeplc_finetune.py:44, 135-156) so decoys stay on the target iRT scale, and set a + seed if you want the fine-tune to be reproducible. diff --git a/docs/09_extract.md b/docs/09_extract.md new file mode 100644 index 0000000..15d1774 --- /dev/null +++ b/docs/09_extract.md @@ -0,0 +1,559 @@ +# extract (Stage D): the core targeted extraction +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +`extract` is the central stage of the pipeline. It takes the run-independent +spectral library (precursor and b/y fragment m/z, predicted intensities, iRT), +the per-candidate RT windows learned in `rt-im-train`, and the converted MS2 +(optionally MS1) scans, and produces one apex-level PSM per surviving candidate +plus per-fragment chromatograms. Exact intensity-based scores are not computed +here; they are computed downstream in `features` from the chromatograms this +stage emits. `extract` is therefore an evidence-gathering and coarse-acceptance +stage, not a scoring stage. + +The design is **data-driven and peak-major** (see the module doc comment, +`extract.rs:1`). Observed peaks probe an inverted fragment index, and a candidate +hypothesis is materialized only where fragment evidence actually collides with an +observed peak. Work scales with peak-candidate collisions, not with library size. +In wide-window DIA roughly 98% of fragment m/z collide within tolerance +(`extract.rs:743`), so one observed peak typically matches many co-isolated +candidates; the peak-claim strategies decide how that shared intensity is +apportioned. + +RT is applied as a per-candidate window post-filter (the documented Stage D part +2 fallback), and the MVP is 3D so the ion-mobility (IM) dimension is absent +(`extract.rs:8`, `apex_im` is always written `None` at `extract.rs:1435`). + +## Files + +| Path | Role | +|---|---| +| `rust/mumdia/crates/mumdia/src/stages/extract.rs` | The stage: accumulation, cascade, apex, chromatogram/MS1 emission, schema writing | +| `rust/mumdia/crates/mumdia/src/peaks.rs` | Pure top-K chromatographic peak enumerator (`enumerate_peaks`) for the `retain_top_peaks` path | +| `rust/mumdia/crates/mumdia/src/index.rs` | `Library` (SoA library + bucketed `page_search`, `candidate_range`, `cand_frags`) | +| `rust/mumdia/crates/mumdia/src/matchers/fragindex.rs` | `FragIndex` backend (`build`, `probe_peak`, `candidate_range`), the default matcher | +| `rust/mumdia/crates/mumdia/src/spectra.rs` | `load_ms2` / `load_ms1` / `Ms1Scan` scan loaders | +| `rust/mumdia/crates/mumdia-core/src/config.rs` | `ExtractConfig`, `GateMode`, `PeakClaim`, `MatcherKind` | +| `rust/mumdia/crates/mumdia-core/src/constants.rs` | `ISOTOPE_SPACING`, `ppm_bounds` | +| `rust/mumdia/crates/mumdia-core/src/schema.rs` | `artifact::PSMS_EXTRACTED`, `artifact::CHROMATOGRAMS` schema ids | + +Entry point: `stages::extract::run(ExtractParams)` (`extract.rs:604`), wired from +the CLI in `main.rs:495` (`Cmd::Extract`) and from the orchestrator in +`stages/run.rs:303`. + +## Inputs and outputs + +### Inputs (`ExtractParams`, `extract.rs:61`) + +- `ms2` (Parquet): converted MS2 scans, loaded via `load_ms2` (`extract.rs:643`). + Each `Ms2Scan` carries `rt_seconds`, an isolation `window` (`lower_mz`, + `upper_mz`), and centroided `peaks` (`mz`, `intensity`). +- `library_precursors` + `library_fragments` (Parquet): loaded once into + `Library::load` (`extract.rs:606`) at `cfg.bucket_size`. +- `run_windows` (Parquet): per-candidate RT windows, columns `candidate_id`, + `rt_pred_cal`, `rt_lo`, `rt_hi` (read at `extract.rs:625`, scattered into the + dense `rt_lo`/`rt_hi`/`rt_cal` arrays indexed by `candidate_id`, + `extract.rs:634`). Candidates with no window row keep `[-inf, +inf]` and + `rt_cal = 0.0` (`extract.rs:631`); the `0.0` disables the Gaussian RT prior for + those candidates (the prior requires `rt_cal > 0`, `extract.rs:1049`). +- `ms1` (optional Parquet): MS1 scans via `load_ms1` (`extract.rs:645`). When + absent, all MS1 columns are null. +- `mass_cal` (optional JSON, the seed's `.masscal.json`): reads + `frag_ppm_offset` and `frag_tol_ppm` (`extract.rs:678`). Missing file = + offset 0 and `cfg.frag_tol_ppm`. +- `restrict_candidates` (optional prior `psms.parquet`): a `candidate_id` + allowlist (`extract.rs:611`) for the "gate first, then compete" workflow. + +### Output: `psms_extracted` (`out_psms`, schema `psms_extracted` v1) + +Column order and types from `extract.rs:1474`: + +| Column | Type | Meaning | +|---|---|---| +| `candidate_id` | U32 | Library candidate id (sorted ascending in the file) | +| `apex_rt` | F64 | Selected apex RT (seconds) | +| `apex_im` | OptF64 | Always null (3D MVP) | +| `apex_intensity` | F32 | Full summed intensity of the apex scan group | +| `n_matched_fragments` | I32 | Distinct matched predicted fragments | +| `n_predicted_fragments` | I32 | Predicted fragment count for the candidate | +| `coelution_run` | I32 | Longest consecutive-scan co-elution run | +| `rt_pred_cal` | F64 | Calibrated predicted RT for this candidate | +| `precursor_mz` | F64 | Candidate precursor m/z | +| `charge` | I32 | Precursor charge | +| `label` | Str | `"target"` or `"decoy"` | +| `base_peptide_id` | U32 | Stripped-peptide id (for competition grouping) | +| `peptidoform` | Str | ProForma-lite string | +| `protein` | Str | Protein accession | +| `predicted_irt` | F32 | Library iRT | +| `contested_frac` | F64 | Intensity fraction lost to a better co-eluter (0 when two-pass off) | +| `ms1_isom1` | OptF64 | MS1 intensity at (mono - spacing/z) | +| `ms1_mono` | OptF64 | MS1 monoisotopic intensity | +| `ms1_iso1` | OptF64 | MS1 +1 isotope intensity | +| `ms1_iso2` | OptF64 | MS1 +2 isotope intensity | + +Conditional columns (default-off, added only when the knob is set so the +production schema stays byte-identical): +- `emit_contested_features` -> `contested_count_frac` F64, `apportioned_frac` F64 (`extract.rs:1499`). +- `emit_gate_diagnostics` -> `gate_apex`, `gate_peak_spectral`, `gate_coelution`, `gate_spectral_entropy` (all F32, `extract.rs:1505`). + +The three soft-competition columns are all derived from the per-candidate +`Contested` accumulator (`extract.rs:104`), whose fields are `won`/`lost` +(summed observed intensity of shared peaks this candidate won or lost as +most-eluting claimant), `n_won`/`n_lost` (the corresponding peak-instance +counts), and `apportioned` (the co-elution-weighted proportional share the +candidate would keep under `CoelutionProportional`). The columns are (all 0 when +the two-pass path did not run): +- `contested_frac` = `lost / (won + lost)` (`extract.rs:1238`): fraction of + contested *intensity* lost to a better co-eluter. +- `contested_count_frac` = `n_lost / (n_won + n_lost)` (`extract.rs:1246`): + fraction of contested fragment-*peaks* lost. +- `apportioned_frac` = `apportioned / (won + lost)` (`extract.rs:1254`): fraction + of contested intensity the candidate retains under proportional apportionment + (1 = keeps all; ~0 = a peak-borrower stripped by its co-eluting competitors). + +### Output: `chromatograms` (`out_chrom`, schema `chromatograms` v1) + +Column order from `extract.rs:1512`: + +| Column | Type | Meaning | +|---|---|---| +| `candidate_id` | U32 | Candidate id | +| `frag_name` | Str | Fragment name (`b3`, `y7`, or `ms1_mono`/`ms1_iso1`/`ms1_iso2`) | +| `frag_mz` | F64 | Theoretical fragment m/z (or precursor isotope m/z for MS1 rows) | +| `frag_obs_mz` | F64 | Intensity-weighted observed m/z (falls back to theoretical) | +| `predicted_intensity` | F32 | Library predicted intensity (0.0 for MS1 rows) | +| `rt` | LargeListF32 | Per-scan RT axis of the trace | +| `intensity` | LargeListF32 | Per-scan intensity, 0-filled on the window grid | + +One row is emitted for **every** predicted transition (`extract.rs:1295`), so +`features` sees the full predicted set and can penalize a missing strong ion. A +never-observed fragment carries an **empty** trace (not a grid-length zero +vector) to avoid bloating the list column (`extract.rs:1320`). `rt`/`intensity` +are `LargeList` (64-bit offsets) because the total list-value count can exceed +the ~2.1B 32-bit `ListArray` offset ceiling when gates are opened wide +(`extract.rs:1520`). + +### Output: top-K peaks sidecar (`.peaks.parquet`) + +Written only when `retain_top_peaks > 1` (`extract.rs:1530`), one row per +(candidate, peak). Columns: `candidate_id` U32, `peak_rank` I32, `apex_rt` F64, +`start_rt` F64, `end_rt` F64, `evidence_count` F64, `area` F64. **These peaks are +not scored** and do not affect FDR; they are candidate peaks for an offline +peak-selection model (see below). + +Each artifact (`psms_extracted` and `chromatograms`, but **not** the top-K peaks +sidecar) also gets an `.report.json` (`extract.rs:1551`) recording the +row count, blake3 content hash, stage name, schema name+version, and +`elapsed_ms`. The `params` object (`extract.rs:1562`) carries `frag_tol_ppm` +(nominal), `effective_frag_tol_ppm` (post mass-cal), `frag_ppm_offset`, +`presence_min_fragments`, `presence_min_coelution`, `min_frag_corr`, `gate_mode`, +`gate_coelution_min`, and `scan_window`. The `stats` map (`extract.rs:1548`) +carries `accepted` (the accepted-candidate count) and `scan_window`. +`model_identity` is `None` (no model in this stage). + +## How it works + +The control flow of `run` (`extract.rs:604`) is: load library and windows, load +scans, build the matcher, accumulate peak-candidate hits (peak-major), then per +candidate run the acceptance cascade and apex selection, emit chromatograms and +MS1 XICs, and write the tables. + +### 1. Matcher and mass recalibration + +The fragment tolerance and a systematic ppm offset come from `mass_cal` +(`extract.rs:678`). The offset is applied as a divisor `offset_factor = 1 + +offset*1e-6` (`extract.rs:698`); every observed peak m/z is corrected by +`q_mz = peak.mz / offset_factor` before probing (`extract.rs:740`). + +Two matcher backends dispatch through `probe_matched` (`extract.rs:43`): +- `MatcherKind::Fragindex` (default): builds a `FragIndex` once at the learned + tolerance (`extract.rs:704`). `FragIndex::probe_peak` (`fragindex.rs:152`) + probes bins `bin-1 ..= bin+1`, verifies each posting with the exact f64 ppm + predicate, and carries the **true generating fragment ordinal** in `post_frag`. +- Bucketed fallback: `Library::page_search` (`index.rs:247`) resolves the + fragment ordinal by nearest stored m/z via `Library::local_frag_index` + (`index.rs:222`). This is a semantic difference for fragments at + sub-f32-identical m/z (`extract.rs:40`). + +`Library::candidate_range` / `FragIndex::candidate_range` (`index.rs:238`, +`fragindex.rs:137`) give the half-open candidate-id range `[lo, hi)` whose +precursor m/z falls in an isolation window, exploiting that the library is sorted +by precursor m/z. This is what makes a per-window probe cheap. + +### 2. Peak-major accumulation + +The accumulator `acc: HashMap>` (`extract.rs:707`) maps each +candidate to the observed hits it collected. A `Hit` (`extract.rs:86`) is +`{rt, frag, inten, obs_mz}`. Entries are created lazily on the first collision. + +There are three accumulation paths: + +- **Parallel per-window, single-pass** (`extract_accumulate_windows`, + `extract.rs:264`): used when `fidx` is present and there is no `restrict` list + and the path is not two-pass. Scans are grouped by isolation window (each scan + belongs to exactly one window), and the ~150 windows are processed in parallel + with rayon. It is bit-identical to the serial loop because the per-candidate + cascade rt-sorts hits before summing, and same-rt hits for a candidate all come + from one window (`extract.rs:260`). +- **Serial single-pass** (`extract.rs:731`): the fallback when there is a + `restrict` allowlist or no fragindex. It honors every non-co-elution + `peak_claim` strategy. +- **Two-pass co-elution** (`extract_twopass_windows`, `extract.rs:385`): used + when `peak_claim` is one of the `Coelution*` variants **or** + `emit_contested_features` is set (`extract.rs:715`). Like the single-pass + parallel path it partitions scans by isolation window and fans out over the + ~150 windows with rayon (`extract.rs:413`), so it stays parallel even with a + `restrict` allowlist. Pass 1 builds each candidate's per-scan elution profile + (summed matched intensity, `extract.rs:459`). Pass 2 arbitrates each shared + peak to the claimant most eluting at that scan (highest profile height at `rt`, + `extract.rs:503`; ties break by higher predicted intensity then lower + `candidate_id`, `extract.rs:508`), tracks won/lost/apportioned contested + intensity in `Contested` (`extract.rs:104`), and reassigns intensity per the + co-elution claim variant. The `reassign` flag (`extract.rs:817`) is true **only** + for the three `Coelution*` variants; when the two-pass path is triggered by + `emit_contested_features` alone (a non-co-elution `peak_claim`), pass 2 still + computes the contested statistics but the returned accumulation is the base + full-intensity `acc1`, not the reassigned `acc2` (`extract.rs:578`). So + `emit_contested_features` adds the soft-competition features **without** altering + any extracted intensity. `restrict` is honored inside both passes via the push + closure (`extract.rs:440`, `extract.rs:484`). + +Per-peak claim strategies (`PeakClaim`, applied in the single-pass loop at +`extract.rs:764` and mirrored in the parallel path): +- `None`: every matching candidate gets the full peak intensity (legacy default). +- `WinnerPredictedIntensity`: only the claimant with the highest predicted + intensity keeps the peak; ties break by lowest `candidate_id` for determinism + (`extract.rs:770`). +- `Proportional`: split the peak by predicted-intensity share (`extract.rs:782`). +- `CoelutionWinner` / `CoelutionProportional` / `CoelutionWinnerMargin`: two-pass + variants keyed on elution-profile height, arbitrated in `extract.rs:540`. + `CoelutionWinnerMargin` only strips a peak from the runner-up when the top + eluter dominates by `peak_claim_margin` (`extract.rs:521`); otherwise the peak + stays shared, avoiding stripping real peptides at ambiguous peaks. + +### 3. Per-candidate cascade (parallel over candidates) + +`acc` is drained into `(cid, hits)` in sorted `candidate_id` order +(`extract.rs:880`, `cand_hits` at `extract.rs:892`) for determinism, then each +candidate is processed in parallel via `into_par_iter().map(...)` returning +`Option` (`extract.rs:939`). Each candidate's work depends only on its +own hits plus read-only library/window/MS1 data, so `collect()` in the sorted +order reproduces the serial push order byte-for-byte (`extract.rs:883`). + +The cheap-to-expensive acceptance cascade, in order: + +1. **Distinct-fragment presence (tier b)**: distinct matched fragments must be at + least `presence_min_matched` (`extract.rs:946`), else the candidate is dropped + before any grouping work. `presence_min_matched`, `presence_min_fragments`, and + `presence_min_coelution` are each floored at 1 via `.max(1)` (`extract.rs:946`, + `extract.rs:1120`, `extract.rs:1109`), so a configured 0 still requires at + least one fragment. +2. **Scan grouping**: hits are rt-sorted and grouped into scan groups + `Vec<(rt, BTreeMap)>`, deduping the same fragment within one + scan by max (`extract.rs:951`). The `BTreeMap` fixes per-scan fragment order so + the f32 apex sum is deterministic. +3. **Acquisition-scan grid projection** (`extract.rs:978`): when + `emit_window_grid` is on, the sparse groups are projected onto the full set of + covering-window scans inside the RT window, so missing acquisition scans count + as 0 and break a co-elution run rather than being invisible. +4. **Apex selection** (see below). +5. **Co-elution run**: the longest run of consecutive scan groups with at least + `presence_min_coelution` fragments present (`extract.rs:1106`). +6. **Acceptance (tier c)** (`extract.rs:1120`): reject unless distinct fragments + >= `presence_min_fragments`, `best_run >= scan_window` (the + `fixed_scan_window` floor), `best_run >= min_coelution_run`, and + `matched_fraction >= min_matched_fraction`. `matched_fraction` is + `distinct / n_predicted` (`extract.rs:1119`); it is the primary symmetric + discriminator (real peptides match a large fraction, chimeric and decoy matches + a small fraction alike, keeping the target-decoy null valid). +7. **MS1 isotope evidence** (`extract.rs:1133`) computed *before* the Pearson gate + so it can rescue a candidate. `ms1_support` requires a present mono and a +1/mono + ratio in `[0.1, 1.5]` (`extract.rs:1150`). +8. **Pearson gate (tier d, optional)** (`extract.rs:1194`): only when + `min_frag_corr > 0.0`. It thresholds the score of the **active** `GateMode` + and only the active score is computed (the closures at `extract.rs:1179` are + lazy). If `ms1_rescue` is set, a gate failure is overridden when the candidate + has MS1 support and enough matched fragments (`extract.rs:1210`). + +### 4. Apex selection (`extract.rs:1011`) + +The apex is chosen among scan groups whose smoothed distinct-fragment count +qualifies, then by a per-scan score: + +- **Rolling-window count** (`apex_count_window`): a centered rolling **sum** of + the per-scan distinct-fragment count (`extract.rs:1030`). It is deliberately a + sum, not a mean: edge truncation makes interior positions accumulate more, + center-weighting the apex toward the RT-window centre (a mild RT prior). Window + 1 reproduces exact per-scan counts. Only scans with smoothed count `>= maxc - + apex_count_tol` qualify (`extract.rs:1042`). +- **Signature ions** (`apex_top_fragments`, default 3 via `k_sig`, + `extract.rs:1054`): the top-K predicted fragments, so a bright interferent on a + non-signature ion cannot define the apex. +- **RT prior** (`apex_rt_prior_s`): when > 0 and `rt_cal > 0`, each qualifying + scan's score is multiplied by `exp(-0.5*((rt - rt_cal)/sigma)^2)` + (`extract.rs:1079`). +- **Scoring mode** (`apex_evidence_rank`, `extract.rs:1084`): when true, the score + is `n_frag + sig_sum/(sig_sum+1)` times the prior, so the number of distinct + co-eluting predicted fragments dominates and observed signature intensity only + breaks sub-integer ties. This is interference-resistant in wide-window DIA + because intensity is chimeric. When false (default), the legacy + `sig_sum * prior` is used, bit-identical to the pre-feature behaviour. + +`apex_intensity` reported is the full summed intensity of the winning scan group +(`extract.rs:1101`), not the signature-only score. + +### 5. Gate scoring internals + +Spectral-agreement scores share a helper `peak_window` (`extract.rs:158`) that +finds the contiguous elution-peak scan range `[lo, hi]` around the signature-ion +apex (scans above 10% of the reference apex height), returning `None` when there +are fewer than 3 scans or no reference signal. This restriction matters: over the +full wide extraction window the traces are mostly zeros and any correlation is +noise, so the co-elution and spectral gates are only meaningful across the +elution peak itself (`extract.rs:155`). `GateMode` scores: +- `ApexPearson`: Pearson of observed-vs-predicted intensities at the single apex + scan (`extract.rs:1179`); one chimeric scan can dominate it. +- `PeakSpectral`: `peak_spectral_score` (`extract.rs:199`) correlates the + peak-summed observed spectrum (each fragment integrated over the peak scans) + with predicted intensities, averaging out a single interfered scan. +- `SpectralEntropy`: Li spectral-entropy similarity of the sqrt-transformed apex + spectrum via the shared `features::entropy` kernel (`extract.rs:1185`). +- `Coelution`: `coelution_gate_score` (`extract.rs:225`), the + predicted-intensity-weighted mean Pearson of each matched fragment's XIC to the + signature-ion reference profile, restricted to the elution peak. Orthogonal to + intensity agreement (temporal, not shape). +- `Combined`: requires `peak_spectral >= min_frag_corr` **and** `coelution >= + gate_coelution_min` (`extract.rs:1205`). + +All four diagnostic scores are computed for every accepted candidate only when +`emit_gate_diagnostics` is set (`extract.rs:1224`); otherwise they are zeroed and +not written, so the default chain pays no extra cost. + +### 6. Chromatograms and MS1 XICs + +Per-fragment observed m/z is an intensity-weighted mean per fragment +(`extract.rs:1264`) for mass-accuracy features. Every predicted transition emits a +chromatogram row (`extract.rs:1295`); observed fragments carry the grid-sampled +(or rt-sorted) trace, absent ones an empty trace. When MS1 is present and grid +mode is on, three MS1 isotope XICs (`ms1_mono`, `ms1_iso1`, `ms1_iso2`) are +sampled on the same scan grid via nearest-MS1-scan lookup (`extract.rs:1337`), +using `ISOTOPE_SPACING / charge` (`constants.rs:22`, value `1.003354835`) for the +isotope offsets and `sum_near` (`extract.rs:130`) to integrate within +`prec_tol_ppm`. The apex MS1 isotope intensities (`ms1_isom1`/mono/`iso1`/`iso2`) +are the per-PSM columns, taken from the nearest MS1 scan to the apex RT +(`extract.rs:1133`). + +### 7. Top-K peak enumeration (`retain_top_peaks`) + +When `retain_top_peaks > 1` and the candidate has groups (`extract.rs:1360`), the +per-scan distinct-fragment count profile (`count_prof`, `extract.rs:1363`) is +passed to `crate::peaks::enumerate_peaks` (`peaks.rs:52`) with `k = +retain_top_peaks`, `bound_fraction = 1/3` and `min_prominence_frac = 0.1` +(`extract.rs:1364`). `enumerate_peaks` is a pure, side-effect-free function +(`peaks.rs:1`): +1. An empty profile, `k == 0`, or an all-non-positive profile returns an empty + vector (`peaks.rs:58`, `peaks.rs:63`). +2. **Local maxima** (`peaks.rs:71`): index `i` qualifies when its value is `> + 0`, is `>= prom_floor` (the prominence floor `min_prominence_frac * + global_max`, `peaks.rs:66`), is **strictly** greater than its left neighbour + (or is the left edge), and is `>=` its right neighbour (or is the right edge). + The strict-left / non-strict-right rule makes a flat plateau register once, at + its left edge; edges count as maxima against their single neighbour. +3. **Boundary walk** (`peaks.rs:88`): from each apex, walk left and right, + stopping when the profile drops below `bound_fraction * apex` **or** turns back + upward (a valley); `area` is the integrated profile over `[start_idx, + end_idx]`. +4. **Dedup + rank** (`peaks.rs:125`): peaks are sorted strongest-first by `area`, + then by `apex_intensity`, then by earliest `apex_idx`; a maximum whose apex + falls inside an already-kept peak's `[start_idx, end_idx]` envelope is dropped + (`peaks.rs:140`); the first `k` survivors are kept and assigned `rank` + 0..k-1 (`peaks.rs:148`). + +`enumerate_peaks(.., k=1, ..)` returns the single global-argmax peak, so callers +can adopt it incrementally. The retained peaks rank by **co-eluting fragment +breadth (count), not intensity**, per finding A6 in +docs/18_findings_and_decisions.md (intensity is chimeric in DIA). Back in `extract`, each returned `PeakGroup` is mapped to a sidecar row +(`extract.rs:1365`): `apex_rt`/`start_rt`/`end_rt` are the RTs of the +apex/start/end scan groups, `evidence_count` is the distinct-fragment count at +the apex group (`groups[pk.apex_idx].1.len()`), and `area` is `pk.area`. The main +PSM still reports the single selected apex, so FDR is unaffected; the sidecar +peaks are unscored candidate peaks for an offline peak-selection model +(`extract.rs:1354`). + +## Key types and functions + +| Name | file:line | What it does | +|---|---|---| +| `run` | `extract.rs:604` | Stage entry point; orchestrates load, accumulate, cascade, write | +| `ExtractParams` | `extract.rs:61` | Input path bundle + config + config hash | +| `Hit` | `extract.rs:86` | One observed hit: `rt`, `frag`, `inten`, `obs_mz` | +| `Contested` | `extract.rs:104` | Per-candidate two-pass contested-peak stats: `won`/`lost` intensity, `n_won`/`n_lost` peak counts, `apportioned` share | +| `CandOut` | `extract.rs:897` | Per-candidate parallel-map result (PSM row + chrom rows + peaks) | +| `probe_matched` | `extract.rs:43` | Dispatch a peak probe to fragindex or bucketed backend | +| `extract_accumulate_windows` | `extract.rs:264` | Parallel single-pass per-window accumulation | +| `extract_twopass_windows` | `extract.rs:385` | Parallel two-pass co-elution arbitration + contested stats | +| `peak_window` | `extract.rs:158` | Elution-peak scan range around the signature apex (10% height) | +| `peak_spectral_score` | `extract.rs:199` | Peak-integrated observed-vs-predicted Pearson | +| `coelution_gate_score` | `extract.rs:225` | Weighted mean XIC-to-reference co-elution Pearson | +| `nearest_index` | `extract.rs:113` | Binary search for nearest RT in a sorted array | +| `sum_near` | `extract.rs:130` | Sum intensities within a ppm window (m/z-sorted) | +| `enumerate_peaks` | `peaks.rs:52` | Pure top-K peak-group enumerator | +| `PeakGroup` | `peaks.rs:22` | One enumerated peak (apex/start/end idx, apex intensity, area, rank) | +| `Library::candidate_range` | `index.rs:238` | Candidate-id range for an isolation window | +| `Library::cand_frags` | `index.rs:209` | Fragment m/z, predicted intensity, name slices | +| `FragIndex::probe_peak` | `fragindex.rs:152` | Verified postings for one observed peak in a candidate range | + +## Configuration + +All fields live in `ExtractConfig` (`config.rs:391`); defaults in +`config.rs:508`. The config was recently pruned of dead fields in the +"clean-slate declutter" commit: `extract.scan_window_mode` (and the +`ScanWindowMode` enum), `extract.scan_scale`, `extract.k_select`, +`extract.max_fragment_charge`, and `extract.tolerance_regime` were removed. Do +not reintroduce them. + +| Field | Default | Effect | +|---|---|---| +| `fixed_scan_window` | 3 | Minimum co-elution run length (`scan_window` floor); `.max(1)` at `extract.rs:845` | +| `frag_tol_ppm` | 20.0 | Fragment match tolerance (overridden by `mass_cal`) | +| `prec_tol_ppm` | 20.0 | MS1 isotope integration tolerance | +| `presence_min_matched` | 3 | Tier-b: minimum distinct matched fragments (`extract.rs:946`) | +| `presence_min_fragments` | 3 | Acceptance: minimum distinct fragments (`extract.rs:1120`) | +| `presence_min_coelution` | 2 | Min simultaneously-present fragments to extend a run (`extract.rs:1109`) | +| `min_frag_corr` | **0.2** | Pearson/gate threshold; 0 disables the gate. Relaxed from a historical 0.5 to recover low-abundance candidates (`config.rs:522`) | +| `min_matched_fraction` | 0.0 | Acceptance: min matched/predicted fraction (default off) | +| `apex_top_fragments` | 0 | Signature-ion count for apex; 0 -> default 3 (`extract.rs:1054`). Config marks it superseded by `apex_count_tol`, kept for compat (`config.rs:524`) | +| `apex_rt_prior_s` | 0.0 | Gaussian RT-prior sigma on apex tiebreak; 0 = off | +| `apex_count_tol` | 1 | Count slack for qualifying apex scans | +| `apex_count_window` | 1 | Rolling-sum width for the count profile; 1 = no smoothing. Window 5 cut AIF apex misassignment (median \|dRT\| 131s -> 9s) | +| `apex_evidence_rank` | false | Breadth-of-evidence apex vs legacy signature-intensity apex | +| `emit_window_grid` | true | Zero-filled full-window-grid chromatograms | +| `bucket_size` | 8192 | m/z bucket size (power of two) | +| `peak_claim` | `None` | Shared-peak apportionment strategy (`PeakClaim`) | +| `peak_claim_margin` | 2.0 | Dominance factor for `CoelutionWinnerMargin` | +| `emit_contested_features` | false | Adds `contested_count_frac`/`apportioned_frac`; forces the two-pass path | +| `matcher` | `Fragindex` | Fragment-matcher backend | +| `min_coelution_run` | 0 | Extra co-elution-run floor (0 = off; `scan_window` still applies) | +| `ms1_rescue` | false | Rescue Pearson-gate failures with MS1 isotope support | +| `retain_top_peaks` | 1 | K>1 writes the `.peaks.parquet` sidecar (unscored) | +| `emit_candidate_audit` | false | Candidate-audit sidecar (diagnostic) | +| `emit_gate_diagnostics` | false | Adds the four `gate_*` diagnostic columns | +| `gate_mode` | `ApexPearson` | Which score `min_frag_corr` thresholds (`GateMode`) | +| `gate_coelution_min` | 0.5 | Second threshold for `GateMode::Combined` | + +Note: `emit_candidate_audit` is a declared knob but the candidate-audit write is +not present in this `extract.rs`; the audit ladder is produced by the separate +`mumdia audit` command. Treat the in-stage audit as unwired here. + +### Default-off sensitivity knobs (index) + +Every knob below is default-off, so the production PSM schema and per-candidate +compute stay byte-identical unless the knob is set. None has an end-to-end +identification-gain measurement yet; each must pass the entrapment-holdout gate +on >=2 datasets before being enabled by default +(`sensitivity_plan/NEXT_STEPS.md`). Two of the knobs live in other stages' +configs (`search_seed`, `rt_im_train`) but are listed here for one index. + +| knob | extra columns / artifact | validation status | +|---|---|---| +| `extract.retain_top_peaks` (`config.rs:478`, default 1) | `.peaks.parquet` sidecar, unscored, written only when K>1 (`extract.rs:1530`); no PSM columns | ID loop not closed (sidecar peaks are unscored); gate pending | +| `extract.emit_candidate_audit` (`config.rs:483`) | none in `extract.rs` (unused there); in `run` it gates the `audit` stage -> `candidate_audit.parquet` (`run.rs:405`) | diagnostic; no ID effect | +| `extract.emit_gate_diagnostics` (`config.rs:498`) | 4 F32 cols `gate_apex`, `gate_peak_spectral`, `gate_coelution`, `gate_spectral_entropy` (`extract.rs:1505`) | diagnostic; no ID effect | +| `extract.apex_evidence_rank` (`config.rs:492`) | none; changes the apex-selection score (`extract.rs:1084`) | diagnostic support (finding A6, docs/18); no end-to-end gain measured | +| `search_seed.two_pass_mass_cal` (`config.rs:306`) | none; refits the `.masscal.json` offset + tolerance | not measured; gate pending | +| `rt_im_train.adaptive_rt_window` (`config.rs:362`) | none; per-region RT half-window widths in `run_windows` | not measured; gate pending | +| `extract.emit_contested_features` (`config.rs:454`) | 2 F64 cols `contested_count_frac`, `apportioned_frac` (`extract.rs:1499`); forces the two-pass path (`extract.rs:715`) | not measured; gate pending | +| `compete.mode` (`CompetitionMode`, `config.rs:641`, default `winner_take_all`) | retains more rows in `competed`; optional `.compete_audit.parquet` via `emit_competition_audit` (`config.rs:651`) | not measured; gate pending | + +## Invariants, determinism, gotchas + +- **Determinism**: output is emitted in ascending `candidate_id` order + (`extract.rs:880`); the parallel per-candidate map preserves that order via + `collect()`. Per-scan fragment maps are `BTreeMap` so f32 apex sums have a fixed + addition order (`extract.rs:952`). The parallel window accumulation is documented + as bit-identical to the serial loop (`extract.rs:260`, `extract.rs:725`). A + HashMap f32 sum shifting the apex once broke reproducibility; keep ordered maps + and sorted iteration wherever floats are summed. +- **Default-off contract**: `retain_top_peaks=1`, `apex_evidence_rank=false`, + `emit_contested_features=false`, `emit_gate_diagnostics=false`, `peak_claim=None` + make the schema and per-candidate compute byte-identical to the production chain. + Every sensitivity knob added here must keep that property. +- **The gate optimum depends on the rescorer**, not the gate in isolation. The + full-feature search found `spectral_entropy_similarity_sqrt` the single best + target/decoy discriminator (AUC 0.826), yet gating on it *regressed* + end-to-end identifications versus the apex gate, because gating on the + rescorer's own best feature enriches hard decoys. "Best discriminator" is the + wrong criterion for a gate; the lever is the rescorer. This is why every gate + change must pass an entrapment-holdout gate before being enabled by default. +- **Mass recal is a divisor, not a subtraction**: observed m/z is corrected by + `q_mz = peak.mz / (1 + offset*1e-6)` (`extract.rs:740`); a missing/absent + `masscal.json` yields offset 0 and the config tolerance. +- **Empty vs zero traces**: a never-observed predicted fragment carries an empty + trace, not a grid-length zero vector (`extract.rs:1320`). Downstream code must + treat an empty trace as `obs_apex = 0`. +- **`apex_im` is always null** (3D MVP); do not assume an IM value. +- **`restrict_candidates` only forces the serial path in the non-two-pass case.** + In the non-two-pass branch the parallel window accumulation is used only when a + fragindex is present **and** there is no `restrict` list (`extract.rs:724`); a + `restrict` list therefore routes to the slower serial single-pass loop, which + honors the allowlist and every non-co-elution `peak_claim` strategy. In the + two-pass branch (`Coelution*` or `emit_contested_features`) extraction stays on + the parallel `extract_twopass_windows` path regardless of `restrict`, which + applies the allowlist inside each pass's push closure (`extract.rs:440`, + `extract.rs:484`, wired at `extract.rs:832`). +- **`peaks.parquet` is not scored** and never enters FDR; it is a research sidecar. +- Chromatogram list columns are `LargeListF32` on purpose (`extract.rs:1520`); do + not downgrade to 32-bit `ListF32` or wide-open gates overflow the offset buffer. + +## Tests + +Only the pure gate-scoring helpers and the peak enumerator are unit-tested; there +is no stage-level test of `run` itself (consistent with the "no stage tests for +extract" gap in CLAUDE.md). The tests encode the behavioral invariants that gate +tuning must preserve. + +- `coelution_tests` (`extract.rs:1590`): co-eluting fragments score `> 0.95` + (`extract.rs:1611`); a strong non-co-eluting interferent drops the co-elution + score `< 0.8` (`extract.rs:1625`); fewer than 3 scan groups returns `1.0` + (do-not-reject) rather than a low score (`extract.rs:1634`); `peak_spectral` + scores `> 0.99` when the peak-integrated pattern matches predicted + (`extract.rs:1648`) and still recovers a fragment that is momentarily unsampled + at the apex scan by integrating over the peak (`extract.rs:1665`, the DIA + scan-gap case the single-scan apex Pearson fails). +- `peaks::tests` (`peaks.rs:154`): empty/all-zero profiles yield no peaks; a + clean triangular peak resolves apex and 1/3-height boundaries; `k=1` keeps only + the strongest-area peak; a dominant interference peak with `k=1` discards a + weaker true peak but `k>=2` retains it (the core sensitivity behavior, + `peaks.rs:190`); two maxima rank by area; a left-edge-truncated peak apexes at + index 0; the prominence filter suppresses a noise bump; a shoulder inside a + stronger envelope collapses into it; and repeated calls are bit-identical with + ties broken by earliest apex (`peaks.rs:259`). + +## How to extend / modify + +- **A new gate metric**: add a variant to `GateMode` (`config.rs:551`), add a lazy + score closure next to `apex_pearson`/`peak_spec`/`coel` (`extract.rs:1179`), and + a match arm in the acceptance gate (`extract.rs:1200`). If it is worth + diagnosing, also wire it into the `emit_gate_diagnostics` tuple + (`extract.rs:1224`) and the conditional column block (`extract.rs:1505`). Default + the gate off and validate against entrapment before enabling. +- **A new peak-claim strategy**: add a `PeakClaim` variant (`config.rs:132`); if it + needs elution profiles, extend the two-pass trigger (`extract.rs:715`) and the + reassignment match in `extract_twopass_windows` (`extract.rs:540`); otherwise add + it to the single-pass match (`extract.rs:764`) and the parallel accumulation + (`extract.rs:316`). Keep tie-breaks deterministic (lowest `candidate_id`). +- **Scoring the top-K peaks**: the sidecar peaks are currently unscored. To close + the loop, per-peak feature computation must be added (each retained peak needs + the full per-peak feature vector `features` computes for the selected apex), + then an out-of-fold peak-selection model chooses. This is the open + `retain_top_peaks` work item in the sensitivity plan. +- **New per-PSM columns**: append to the `CandOut` struct (`extract.rs:897`), set + it in the `Some(CandOut { .. })` block (`extract.rs:1381`), push it in the serial + append loop (`extract.rs:1433`), and add the `Col` in the `psms_cols` vector + (`extract.rs:1474`). Gate any non-production column behind an `emit_*` flag to + preserve the byte-identical default schema, and bump the schema version in + `schema.rs` if the default schema changes. +- **IM / 4D**: `apex_im` and the IM data-model hooks exist but are unfilled; a + diaPASEF extension adds an IM window post-filter alongside the RT window and IM + apex/feature families. It cannot be validated without diaPASEF data. diff --git a/docs/10_features.md b/docs/10_features.md new file mode 100644 index 0000000..58b3e70 --- /dev/null +++ b/docs/10_features.md @@ -0,0 +1,678 @@ +# features (Stage E): the feature battery + PIN + +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +Stage E (`mumdia features`) turns one extracted PSM (apex-level identification +plus its per-fragment chromatograms) into a fixed, named, versioned feature +vector for the semi-supervised rescorer. It computes a config-selected feature +set, a scalar `prelim_score` used by `compete`, a Percolator PIN, and a +`blake3` schema id that pins the ordered column list so the classifier is never +trained or applied under a mismatched feature layout. + +The stage is pure per-PSM: each row's features are a function of that row's own +inputs only, with two cross-row exceptions computed up front (charge-state +corroboration grouped by peptidoform, and a global elution half-width learned +from the confident seed set). This makes the heavy per-PSM work embarrassingly +parallel (`rayon`) while keeping byte-identical output to a serial run. + +Everything is driven by `FeaturesConfig` (`mumdia-core/src/config.rs:578`). The +three feature sets are `Minimal` (14), `Rich` (44), and `Extended` (381). The +default is `Minimal`; the tuned `--profile dia` preset and the DIA-NN-library +recipe use `Extended`. + +## Files + +| path | role | +|---|---| +| `rust/mumdia/crates/mumdia/src/stages/features.rs` | stage entry point, `run`, `Evidence`, `build_evidence`, `fragment_features`, boundary detection, `prelim_score`, PIN, schema hash, the Minimal/Rich column lists, and the Extended family registry | +| `rust/mumdia/crates/mumdia/src/stages/features/similarity.rs` | Extended family: observed-vs-library intensity agreement kernels (64 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/entropy.rs` | Extended family: spectral-entropy / information-divergence (18 names); also exports the gate kernel `spectral_entropy_similarity_sqrt` | +| `rust/mumdia/crates/mumdia/src/stages/features/coelution.rs` | Extended family: fragment-vs-reference and pairwise co-elution + cross-correlation (38 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/interference.rs` | Extended family: co-isolation / chimera detection, interference removal, rank decomposition (26 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/chromatographic.rs` | Extended family: peak-shape descriptors of the reference profile and fragments (43 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/mass_accuracy.rs` | Extended family: fragment ppm distribution + positive mass evidence (17 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/ion_series.rs` | Extended family: b/y series coverage, runs, complementarity, per-series similarity (34 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/ms1.rs` | Extended family: precursor isotope-envelope agreement + MS1/MS2 XIC co-elution (25 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/rt.rs` | Extended family: RT-agreement variants (13 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/novel.rs` | Extended family: seed corroboration + precursor/charge metadata (12 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/nonzero.rs` | Extended family: zero-ignoring apex/co-elution variants (12 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/order_consistency.rs` | Extended family: prediction-free MS2-XIC rank stability (8 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/peak_scans.rs` | Extended family: peak-scan count / window-degeneracy indicator (2 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/apex_dispersion.rs` | Extended family: fragment apex dispersion + consensus peak shape (13 names) | +| `rust/mumdia/crates/mumdia/src/stages/features/mass_uncertainty.rs` | Extended family: fragment mass-error dispersion + evidence breadth (10 names) | +| `rust/mumdia/crates/mumdia/src/stats.rs` | shared `pearson`/`cosine`/`spectral_angle` kernels used by every family | +| `rust/mumdia/crates/mumdia-core/src/config.rs:578` | `FeaturesConfig` | + +## Inputs and outputs + +`FeaturesParams` (`features.rs:537`) names the paths: `psms` (extracted PSMs), +`chromatograms`, optional `seed` (seed PSMs), `out` (features Parquet), +`out_pin` (PIN), `cfg`, `config_hash`. + +### Consumed: psms_extracted (`features.rs:550`-`588`) + +Columns read (getter -> column): `candidate_id` (u32), `apex_rt` (f64), +`apex_intensity` (f32), `n_matched_fragments` (i32), `n_predicted_fragments` +(i32, optional; defaults to 6 when absent), `coelution_run` (i32), +`rt_pred_cal` (f64), `charge` (i32), `label` (str), `base_peptide_id` (u32), +`peptidoform` (str), `protein` (str), `precursor_mz` (f64). Optional soft- +competition columns default to 0.0 when absent: `contested_frac`, +`contested_count_frac`, `apportioned_frac`. Optional MS1 apex isotope columns +(`opt_f64`, default `None`): `ms1_isom1`, `ms1_mono`, `ms1_iso1`, `ms1_iso2`. + +### Consumed: chromatograms (`features.rs:591`-`618`) + +`candidate_id` (u32), `frag_name` (str), `frag_mz` (f64), `frag_obs_mz` (f64, +optional; falls back to `frag_mz`), `predicted_intensity` (f32), `rt` +(list), `intensity` (list). Rows whose `frag_name` starts with +`ms1_` are routed to a separate `ms1x` map and fed to the MS1 XIC evidence; +all others are the fragment chromatograms. + +### Consumed: seed PSMs (optional, `features.rs:620`-`652`) + +`candidate_id` (u32), `score` (f64), `spectrum_q` (f64), `label` (str). Builds +the `candidate_id -> seed score` and `-> identified flag` maps, and the +confident-target set (`spectrum_q <= 0.01` and `label == "target"`) used for +`bound_from_confident`. + +### Produced: features Parquet (`features.rs:912`-`931`) + +Bookkeeping columns, in order: `candidate_id` (u32), `label` (str), +`base_peptide_id` (u32), `peptidoform` (str), `protein` (str), `apex_rt` (f64), +`elution_lo` (f64), `elution_hi` (f64), `precursor_mz` (f64), `prelim_score` +(f64). Then one `F64` column per name in `active_features(cfg.set)`, in order. +`elution_lo`/`elution_hi` are the RT bounds the stage actually used (emitted so +downstream and plotting read them rather than re-derive). + +### Produced: companion + PIN + report + +- `.schema.json` (`features.rs:936`): `FeatureSchema { feature_columns, + schema_id }`, read back by `FeatureSchema::read` (`features.rs:245`). +- ``: Percolator PIN, header + `SpecId Label ScanNr ExpMass CalcMass Peptide Proteins` + (`write_pin`, `features.rs:1537`). +- `.report.json` (`ArtifactReport`, `features.rs:969`): logical name + `features`, schema version 1, `stats` carrying `feature_schema_id`, + `n_features`, and `set`; `params` records `set` and + `coelution_corr_threshold` (`features.rs:976`); `content_hash` is the blake3 of + the features Parquet; `model_identity` is `None`. + +## How it works + +Control flow of `run` (`features.rs:548`): + +1. Read the PSM table and pull the scalar columns (`features.rs:550`-`588`). +2. Read chromatograms and group `ChromRow`s by `candidate_id` into `chrom`, + splitting off `ms1_*` rows into `ms1x` (`features.rs:591`-`618`). A + `ChromRow` (`features.rs:279`) holds `frag_name`, `frag_mz`, `frag_obs_mz`, + `pred_int`, and the `rt`/`inten` vectors. +3. Build the seed maps and confident-target set from the optional seed table + (`features.rs:620`-`652`). +4. If `bound_from_confident` is set, learn a global elution half-width + (see "bound_from_confident" below), producing `Option<(L, R)>` in seconds + (`features.rs:658`-`702`). +5. Compute the gradient as `max(apex_rt).max(1.0)` (`features.rs:704`); this is + the run length used to normalize RT errors. +6. Compute the three cross-charge corroboration columns by grouping rows by + `peptidoform` (`features.rs:714`-`734`). +7. In parallel over rows (`features.rs:757`-`808`), compute the two expensive + per-PSM pieces: `fragment_features` (the Minimal/Rich fragment battery) and, + when Extended, `build_evidence` + `extended_values`. Results collect into a + `Vec` indexed by row, preserving order. +8. Serially assemble `fmap` (name -> per-row value vector), `prelim`, and the + elution bounds (`features.rs:810`-`905`). The serial loop reads `per[i]` and + pushes each named value with the `push` closure (`features.rs:738`). +9. Insert the three cross-charge columns (`features.rs:908`-`910`). +10. Build the output columns (bookkeeping + `active_features`), write Parquet, + write the schema JSON, build the feature matrix, write the PIN, and emit the + report (`features.rs:912`-`983`). + +### Fragment features (`fragment_features`, `features.rs:1193`) + +For each fragment it takes the observed intensity at the scan nearest the apex +RT (`obs`), the predicted intensity (`pred`), and `|ppm|`. It then computes: +`frag_corr`=`pearson(obs,pred)`, `frag_cosine`=`cosine(obs,pred)`, +`spectral_angle`; sum-normalized L1 (`norm_manhattan`) and `rmsd`; intensity- +weighted and unweighted mean `|ppm|`; b/y intensity sums and counts. + +It then aligns all fragment traces on the union RT axis (`axis_full`), restricts +to the elution peak (`lo_i..=hi_i`) so co-elution/profile features are not +diluted over the whole `+/- w_rt` window, and computes pairwise Pearson +statistics (`coelution_mean`, `coelution_best`, `n_coelution_above`), lag- +optimized cross-correlation (`xcorr_coelution` mean absolute lag, +`xcorr_shape`), the DIA-NN-style profile block (`profile_cos` = elution^2- +weighted spectral cosine, `ref_corr` = mean fragment-vs-reference Pearson, +`best_ref_corr`, `low_frag_coel`), and the interference-correction block +(`evidence` = summed fragment-vs-reference correlations, `contrast_min`, +`resid_corr`, `coel_clean`, `shadow_frac` from the `1.5*r*ref` cap). Finally +`log_sn` from apex vs median trace point. `elution_lo`/`elution_hi`/ +`base_width_rt`/`n_observations` come from the peak-bounded axis. + +### Elution-boundary detection + +`peak_bounds` (`features.rs:1049`) descends from the apex-nearest scan while the +smoothed profile stays `>= frac * apex_height`, bridging up to `grace` +consecutive sub-threshold scans. If the supplied apex sits at zero height it is +relocated to the global maximum first (`features.rs:1057`), so a zero-height apex +does not collapse the window. The reference profile for boundary finding is the +`smooth3` (`features.rs:1028`) of the summed top-3-predicted-intensity fragment +XICs. + +### prelim_score (`features.rs:901`) + +``` +prelim = n_matched * (0.5 + max(0, frag_corr)) + + max(0, coelution_mean) + + 0.1 * ln(1 + apex_intensity) + - rt_err / gradient +``` + +A cheap heuristic (not a trained score) that rewards matched-fragment count +scaled by spectral correlation, co-elution, and log intensity, penalized by +gradient-normalized RT error. `compete` uses it as the within-group ranking key. + +### Directly-computed Minimal/Rich scalars (serial loop, `features.rs:810`-`905`) + +Several Minimal/Rich columns are assembled inline in the serial loop rather than +inside `fragment_features`: + +- `rt_error_abs` = `calibrated_rt_error(apex_rt, rt_pred_cal)` + (`features.rs:271`), i.e. `|apex_rt - rt_pred_cal|` when both are finite and + `0.0` when either is non-finite; `rt_error_rel` = that over `gradient` + (`features.rs:820`-`821`). Stage B marks an unavailable RT calibration (fewer + than two anchors) as NaN, so this guard keeps the sentinel from leaking NaN + into the feature matrix or `prelim` (tested at `features.rs:1608`). +- `log_apex_intensity` = `ln(1 + apex_intensity)` (`features.rs:824`). +- `n_matched_fragments`, `coelution_run`, `charge` pass through from the PSM + columns; `peptide_length` calls `peptide_length(peptidoform)`. +- `n_proteins` = `protein.matches(';').count() + 1` (`features.rs:842`): the + `protein` string is semicolon-delimited, so this counts group membership. +- `diff_by_intensity` = `sum_b_intensity - sum_y_intensity` (`features.rs:852`). +- `matched_fraction` = `n_matched / max(1, n_predicted)` (`features.rs:874`- + `878`). + +### MS1 isotope features (Rich set, `isotope_features`, `features.rs:1515`) + +The four Rich MS1 columns (`isotope_corr`, `ms1_isom1_ratio`, `log_mono_ms1`, +`has_ms1`) come from `isotope_features` (`features.rs:1515`) over the apex +isotope intensities carried on the PSM rows. It fits a Poisson-averagine envelope +`[1, lambda, lambda^2/2]` with `lambda = 0.00052 * neutral_mass` +(`features.rs:1525`), the neutral mass being +`precursor_mz * charge - charge * PROTON` (`features.rs:816`). This `0.00052` +averagine differs from the `0.000594` used by the Extended `ms1` family +(`ms1.rs:85`); the two MS1 code paths are independent and both are kept. +`isotope_corr` = `pearson([mono, +1, +2], theo)`, `ms1_isom1_ratio` = +`isom1 / (mono + 1)`, `log_mono_ms1` = `ln(1 + mono)`, `has_ms1` = 1.0 when +`mono`/`+1`/`+2` are all present, else all four return 0.0. + +### The Evidence struct (`features.rs:293`) + +`build_evidence` (`features.rs:360`) constructs the per-PSM `Evidence` handed to +every Extended family. It mirrors the alignment and peak-bounding of +`fragment_features` so families see the same elution peak. Fields: + +- Time series: `axis` (RT seconds over the detected elution peak), `traces` + (per-fragment intensity over `axis`, zero-filled, fragment order), `axis_full` + / `traces_full` (whole extracted window), `apex_idx` (index of apex in + `axis`), `ref_profile` (predicted-intensity-weighted sum of the peak traces). +- Fragment-indexed arrays (shared order): `pred` (library intensity), + `obs_apex` (intensity at the apex scan; `> 0` defines "matched"), `is_b`, + `ordinal`, `frag_charge`, `frag_mz` (theoretical), `frag_obs_mz` (intensity- + weighted observed), `mass_err_ppm` (signed ppm). +- `apex_rt` is set inside `build_evidence` itself (`features.rs:517`) from its + `apex_rt` argument, not by the caller. +- Scalars filled by the caller after build (`features.rs:784`-`798`): + `rt_pred_cal`, `rt_err` (via `calibrated_rt_error`), `gradient`, + `precursor_mz`, `charge`, `seq_len`, `n_matched`, `n_predicted`, `seed_score`, + `seed_identified`, `apex_intensity` (plus the MS1 apex isotopes below, + `features.rs:795`-`798`). All start at a zero/`None` default set by + `build_evidence` (`features.rs:517`-`533`). +- MS1: `ms1_mono`/`ms1_iso1`/`ms1_iso2`/`ms1_isom1` (apex isotope intensities, + `None` when no MS1) and `ms1_xic` (the `[mono,+1,+2]` XICs resampled onto + `axis`; empty unless the extract stage persisted `ms1_*` chromatogram rows). + +`parse_ion` (`features.rs:346`) parses `b3`, `y7`, `b3^2` into +`(is_b, ordinal, charge)`. + +### The Extended battery + +`FAMILIES` (`features.rs:52`) is an ordered array of `(NAMES, values)` pairs; +the order is part of the frozen schema and is append-only. Each family exposes +`NAMES: &[&str]` and `values(&Evidence) -> Vec` of identical length and +matching order. `extended_values` (`features.rs:138`) calls each family and +applies the precomputed dedup plan (`extended_value_plan`, `features.rs:112`), +keeping names and values in lockstep, and coerces any non-finite value to 0.0. + +Deduplication (`extended_name_refs`, `features.rs:84`): a name that already +appears in `MINIMAL_FEATURES` or `RICH_EXTRA` (`reserved_names`, +`features.rs:72`), or that repeats across families, is kept only on first +appearance. In the current tree four Extended names are dropped as reserved +collisions: `spectral_angle` (similarity vs Minimal), `rt_error_abs` (rt vs +Minimal), `peptide_length` and `seed_identified` (novel vs Minimal/Rich). So the +335 raw family names reduce to 331 unique Extended names. + +## The three feature sets + +`active_features(set)` (`features.rs:210`) returns the ordered active column +list: + +- `Minimal` (14, `MINIMAL_FEATURES` `features.rs:158`): `rt_error_abs`, + `rt_error_rel`, `n_matched_fragments`, `coelution_run`, `log_apex_intensity`, + `frag_corr`, `frag_cosine`, `spectral_angle`, `coelution_mean`, + `coelution_best`, `n_coelution_above`, `charge`, `peptide_length`, + `n_proteins`. +- `Rich` (44 = Minimal + 30, `RICH_EXTRA` `features.rs:176`): adds + `library_norm_manhattan`, `library_rmsd`, `xcorr_coelution`, `xcorr_shape`, + `sum_b_intensity`, `sum_y_intensity`, `diff_by_intensity`, `n_b_ions`, + `n_y_ions`, `weighted_mass_error`, `mean_mass_error`, `isotope_corr`, + `ms1_isom1_ratio`, `log_mono_ms1`, `has_ms1`, `log_sn`, `n_observations`, + `base_width_rt`, `seed_score`, `seed_identified`, `matched_fraction`, + `profile_cos`, `ref_corr`, `best_ref_corr`, `low_frag_coel`, `evidence`, + `contrast_min`, `resid_corr`, `coel_clean`, `shadow_frac`. +- `Extended` (381 = Minimal + Rich + 331 family names + 6 psms-derived): appends + `extended_names()` then six columns computed outside the family registry. + Three co-elution peak-contest metrics (`peak_contested_frac`, + `peak_contested_count_frac`, `peak_apportioned_frac`) come from the PSM + columns (default 0.0 when absent) and separate peak-borrowing decoys from + genuine IDs. Three charge-corroboration columns (`n_charge_states`, + `charge_multi_flag`, `cross_charge_intensity_log`) aggregate across the charge + states of one peptidoform, an axis invisible to the per-PSM Evidence families. + +The size invariant is asserted in `feature_sets_sized` (`features.rs:1590`): +`Extended.len() == 14 + 30 + extended_names().len() + 6`, and all Extended names +are unique. `FeatureSet` (`config.rs:65`) has exactly `Minimal`, `Rich`, +`Extended`; the earlier `Custom` variant is gone (do not document it). + +## Extended family reference + +Each family below lists its name count and what it measures. All values are +finite (NaN/Inf coerced to 0.0), and every family returns a stable-length vector +even on degenerate evidence. + +### similarity (64, `similarity.rs`) + +Observed-vs-library fragment-intensity agreement under many kernels. `o` is the +per-fragment apex intensity, `l` the predicted intensity; `_matched` restricts +to `o_i > 0`, `_area` replaces `o` with the per-fragment peak-XIC trapezoid +area, `on`/`ln` are sum-normalized. Names: `spectrum_cosine_matched`, +`spectrum_cosine_sqrt`, `spectrum_cosine_log`, `spectral_angle` (dropped as +reserved), `spectral_angle_sqrt`, `spectral_angle_matched`, +`pearson_intensity_matched`, `pearson_intensity_log`, `spearman_intensity`, +`spearman_intensity_matched`, `kendall_tau_intensity`, `dot_product_raw`, +`dot_product_norm`, `library_recall_intensity`, `manhattan_sim`, +`manhattan_sqrt`, `rmsd_norm`, `mae_norm`, `mse_log`, `mae_weighted_pred`, +`abs_diff_q3`, `max_positive_residual`, `chebyshev_dist`, `minkowski_p3`, +`bray_curtis`, `bray_curtis_sqrt`, `canberra`, `canberra_matched`, +`wave_hedges`, `chi_square_pearson`, `chi_square_symmetric`, +`divergence_distance`, `bhattacharyya_coef`, `hellinger`, `squared_chord`, +`harmonic_mean_sim`, `jaccard_presence`, `dice_presence`, +`intensity_weighted_pearson`, `regression_slope`, `gini_diff`, `wasserstein_mz`, +`footrule_norm`, `rank_overlap_top3`, `top1_frag_match`, +`top1_predicted_observed`, `frac_top3_predicted_observed`, +`count_strong_predicted_absent`, `frac_predicted_absent`, `cosine_area`, +`pearson_area`, `spectral_angle_area`, `cosine_fullwindow`, +`stein_scott_weighted_dot`, and the unbounded/granular scores that address +cosine saturation (`log_dot_product`, `spectral_log_evidence`, `scribe_score`, +plus their `_area` twins), `cosine_high_ordinal` (drop b1/b2/y1/y2), and the +robust trimmed-cosine trajectory (`cosine_robust_trim1/2/3`). Local helpers +include tie-corrected `ranks`, `kendall_tau_b`, interpolated `quantile`, `gini`, +`weighted_pearson`, and `trapz`. + +### entropy (18, `entropy.rs`) + +Li spectral-entropy similarity and information divergences between sum- +normalized `o` and `l`. `entropy_sim` (`entropy.rs:107`) is +`1 - (2 H(m) - H(o) - H(l)) / ln 4` with `m = (o+l)/2`, clamped to [0,1]. +Names: `spectral_entropy_similarity`, `weighted_spectral_entropy_similarity` +(Li per-spectrum weighting), `spectral_entropy_similarity_sqrt`, +`spectral_entropy_similarity_topk` (top-6 by predicted intensity), +`spectral_entropy_similarity_area`, `jensen_shannon_divergence`, +`jeffreys_divergence`, `kl_obs_pred`, `kl_pred_obs`, `cross_entropy_obs_pred`, +`obs_spectrum_entropy`, `pred_spectrum_entropy`, `entropy_diff`, +`entropy_ratio`, `obs_normalized_entropy` (Pielou evenness), +`normalized_entropy_diff`, `residual_spectrum_entropy`, `entropy_weight_obs`. +The public `spectral_entropy_similarity_sqrt(obs, pred)` (`entropy.rs:92`) is +reused by the extraction gate `GateMode::SpectralEntropy` so the kernel is not +duplicated. + +### coelution (38, `coelution.rs`) + +Fragment co-elution against the predicted-weighted reference profile R and +pairwise between fragments. Three groups: per-fragment vs R (peak/full windows, +leave-one-out), pairwise Pearson statistics and lag cross-correlation +(`best_xcorr`, `MAXLAG=5`), and structured cross-correlations (b vs y ions, +charge-1 vs multiply-charged). Names include `frag_ref_corr_mean`, +`frag_ref_corr_obsweighted`, `frag_ref_corr_min`, `frag_ref_corr_std`, +`frag_ref_corr_sq_mean`, `frag_ref_corr_topk_weighted`, +`n_frag_ref_corr_above_0_9`, `frac_frag_ref_corr_above_0_8`, +`frag_ref_corr_mean_full`, `full_vs_peak_corr_gain`, +`pairwise_coelution_weighted`, `pairwise_coelution_min/median/std/frac_negative`, +`pairwise_coelution_hi/lo`, `coelution_hi_lo_contrast`, +`coelution_corr_entropy` (Shannon entropy of a 10-bin `NBINS_CORR` +pairwise-Pearson histogram over `[-1, 1]`), the `xcorr_shape_*` and +`xcorr_lag_*` statistics +(`_mean/_min/_std/_mean_abs/_iqr/_frac_zero/_max_abs/_entropy`), +`ref_xcorr_lag_mean`, `ref_xcorr_shape_mean`, `observed_sum_vs_template_corr`, +`frag_loo_ref_corr_mean/_min`, `frac_frags_apex_aligned`, `top3_frag_ref_corr`, +`by_cross_coelution`, `by_cross_lag_mean`, `charge_cross_coelution`. Note the +JSON `coelution_weighted_mean` is an exact alias of `pairwise_coelution_weighted` +and is emitted once under the latter. + +### interference (26, `interference.rs`) + +Co-isolation/chimera detection via the least-squares scale +`r_f = / ` projecting each fragment onto R. Two gating constants: +`IFS_MIN_CORR = 0.6` (`interference.rs:48`) prunes the least-coherent matched +fragment in the iterative `remove_ifs` loop until its leave-one-out ref-corr +stays above it or only 3 fragments remain, and `COHERENT_THR = 0.7` +(`interference.rs:47`) gates `explained_apex_intensity_frac` and `apex_purity` +(only fragments with ref-corr at or above it count as explained); +`n_interfered_fragments` flags a fragment whose apex intensity exceeds +`2 * r_f * R_apex`. Names: `explained_variance_ref`, +`profile_residual_fraction`, `n_interfered_fragments`, +`corrected_vs_raw_cos`, `corrected_vs_raw_ratio`, the iterative interference- +removal block (`ifs_removed_count`, `ifs_removed_intensity_frac`, +`ifs_corr_gain`, `ifs_retained_frac`, `matched_frac_after_ifs`), the peak-vs- +full area ratios (`peak_to_full_area_ratio_profile/_frag_mean/_weighted`, +`out_of_peak_intensity_frac`), `profile_corr_full_vs_peak_delta`, +`frac_frag_ref_corr_below_0_5`, `explained_apex_intensity_frac`, `apex_purity`, +`interference_apex_residual_fraction`, `dominant_frag_ref_corr`, the rank +decomposition of the matched fragment x time Gram matrix by power iteration +(`explained_variance_ratio`, `second_component_fraction`), competing-peak +descriptors on the full-window profile (`profile_second_peak_ratio`, +`n_competing_peaks_in_window`), `matched_pred_intensity_fraction`, and +`top_pred_frag_matched`. + +### chromatographic (43, `chromatographic.rs`) + +Peak-shape quality of R (Gaussian moment-match fit, EMG grid-fit comparison via +`erfc`) and of individual fragments. Names: `gaussian_fit_r2`, +`gaussian_cosine`, `emg_fit_improvement`, `apex_prominence`, `profile_peak_snr` +(MAD-based over out-of-peak scans), width descriptors (`fwhm_seconds`, +`fwhm_to_window_ratio`, `width_at_10pct`, `width_ratio_10_50`), +asymmetry/tailing (`hwhm_asymmetry`, `tailing_factor_usp`, +`asymmetry_factor_10pct`), apex shape (`apex_sharpness`, `apex_curvature`, +`apex_to_boundary_ratio`, `apex_dominance`), roughness (`zigzag_index`, +`jaggedness`, `roughness_2nd_deriv`), multimodality (`n_local_maxima`, +`modality`), RT moments (`rt_skewness`, `rt_excess_kurtosis`, `rt_std_seconds`, +`mean_mode_offset`), area descriptors (`fraction_area_within_fwhm`, +`triangle_area_similarity`), `baseline_fraction`, `peak_completeness`, +`apex_centering_offset`, `intensity_score`, `total_xic_log`, per-fragment +descriptors (`frag_fwhm_cv/_mean`, `frag_apex_rt_dispersion/_weighted`, +`frag_apex_offset_from_profile_mean`, `frag_gaussianity_mean/_weighted`, +`frag_zigzag_mean`), `sumtrace_unweighted_gaussian_r2`, and +`reference_profile_rt_entropy_peak/_ratio`. Has unit tests +(`chromatographic.rs:979`). + +### mass_accuracy (17, `mass_accuracy.rs`) + +Fragment ppm-error distribution over matched fragments (uses a fixed +`FRAG_TOL_PPM = 20.0`; the config tolerance is not carried in Evidence). Names: +`median_abs_frag_ppm`, `signed_mean_frag_ppm`, `ppm_std`, `ppm_iqr`, +`ppm_range`, `max_abs_frag_ppm`, `intensity_weighted_abs_ppm`, +`intensity_weighted_signed_ppm`, `intensity_weighted_ppm_std`, +`lib_weighted_abs_ppm`, `frac_frag_within_half_tol`, `high_ppm_intensity_frac`, +`ppm_intensity_anticorr`, `mass_error_mz_trend`, `mean_abs_mz_error_da`, and the +positive DIA-NN-style evidence `mass_evidence_gauss` (predicted-weighted +Gaussian concentration, sigma 10 ppm) and `mass_log_evidence`. +`precursor_mass_error_ppm` (in the plan.md feature spec, local only) is +deliberately skipped (no theoretical precursor m/z in Evidence). + +### ion_series (34, `ion_series.rs`) + +b/y series coverage, ladder contiguity, complementarity, per-series similarity. +Names: `n_matched_b/_y`, `frac_matched_b/_y`, `by_count_balance`, +`by_intensity_ratio`, `by_ratio_agreement` (tanh-squashed log-odds), +`by_ratio_consistency`, `longest_b_run/_y_run`, `longest_run_max`, +`longest_run_frac_length`, `series_coverage_b/_y`, `sequence_coverage` +(cleavage-site union), `series_gap_fraction`, `by_complement_count`, +`by_complement_mz_consistency` (b+y obs m/z vs M + 2 proton), +`by_complement_coelution`, `ordinal_intensity_concordance_y/_b`, +`series_coelution_y/_b`, `spectral_angle_b/_y`, `pearson_b/_y`, +`cosine_charge1/_charge2`, `charge_corr_balance`, `mean_matched_ordinal_norm`, +`by_ion_contiguous_intensity`, `by_ion_contiguous_lib_frac`, +`both_series_present`. Uses `PROTON` from `mumdia-core::constants`. + +### ms1 (25, `ms1.rs`) + +Precursor isotope-envelope agreement against a Poisson-averagine model +(`lambda = 0.000594 * M`) plus MS1/MS2 XIC co-elution. Apex names: +`ms1_isotope_cosine_apex`, `ms1_isotope_spectral_angle_apex`, +`ms1_isotope_chi2_apex`, `ms1_isotope_manhattan_apex`, `iso_ratio_1_0`, +`iso_ratio_2_0`, `iso_plus1_ratio_dev`, `iso_plus2_ratio_dev`, +`iso_minus_one_fraction`, `iso_overlap_flag`, `log_ms1_mono`, +`ms1_total_isotope_log`, `has_ms1_signal`, `ms1_isotope_apex_entropy_3`, +`ms1_m1_entropy_contribution`. The XIC block (`ms1_ms2_time_corr`, +`ms1_ms2_envelope_time_corr`, `ms1_iso_coelution`, `ms1_ms2_apex_rt_delta`, +`ms1_iso_ratio_stability`, `ms1_mono_gaussianity`, `ms1_ms2_fwhm_ratio`, +`ms1_isotope_corr_xic`, `ms1_envelope_over_time_corr`, +`ms1_isotope_xic_shape_consistency`) reads `Evidence.ms1_xic`. Extract now +persists `ms1_mono`, `ms1_iso1`, and `ms1_iso2` chromatogram rows when MS1 input +and a scan grid are available, so these features are populated in normal +orchestrated runs. They remain 0.0 for older artifacts, standalone extraction +without `--ms1`, or candidates without usable MS1/grid evidence. + +### rt (13, `rt.rs`) + +RT agreement between the observed apex and the calibrated predicted RT. Names: +`rt_error_signed`, `rt_error_abs` (dropped as reserved), `rt_error_squared`, +`rt_error_signed_norm_gradient`, `rt_error_abs_norm_gradient`, `observed_rt_raw`, +`predicted_rt_raw`, `observed_rt_fraction`, `predicted_rt_fraction`, +`rt_error_over_peak_width` (base width at 10%), `rt_error_over_fwhm`, +`rt_diff_profile_apex` (vs full-window profile argmax RT), +`predicted_rt_in_gradient`. + +### novel (12, `novel.rs`) + +Seed corroboration and precursor/charge metadata. Names: +`log_seed_hyperscore`, `seed_hyperscore_per_matched`, `seed_identified` (dropped +as reserved), `peptide_length` (dropped as reserved), `precursor_charge`, +`charge_is_2/_is_3/_is_4plus`, `precursor_mass`, `log_total_matched_intensity`, +`n_matched_frags`, `n_predicted_frags`. Sequence-dependent features (missed +cleavages, C-terminal residue, modification count) are skipped because Evidence +carries only `seq_len`. + +### nonzero (12, `nonzero.rs`) + +Zero-ignoring variants of the apex spectral and co-elution features. The apex +scan samples one grid point, so a fragment peaking one scan off reads 0 there +(~8% of fragments); these recompute over the per-fragment peak-max and over +present-only scans. Names: `frag_corr_peakmax`, `frag_cosine_peakmax`, +`spectral_angle_peakmax`, `frag_corr_matched_nz`, `frag_cosine_matched_nz`, +`peakmax_apex_gain`, `n_frag_present_inpeak`, `frac_frag_present_inpeak`, +`coelution_mean_bothpos`, `coelution_mean_summpos`, `ref_corr_nz`, +`profile_cos_nz`. + +### order_consistency (8, `order_consistency.rs`) + +Prediction-free MS2-XIC rank stability: at each scan the fragments are ranked by +observed intensity, and the features measure whether that ranking persists +across the peak (orthogonal to library-agreement families). Names: +`rank_corr_vs_apex_mean`, `rank_corr_vs_apex_std`, `rank_corr_adjacent_mean`, +`kendall_vs_apex_mean`, `top1_frag_persistence`, `top2_order_persistence`, +`argmax_frag_entropy`, `self_cosine_vs_apex_mean`. Degenerate below 3 fragments +or 3 non-empty scans. Has unit tests (`order_consistency.rs:300`). + +### peak_scans (2, `peak_scans.rs`) + +Label-blind window-degeneracy indicator, emitted for every PSM so the rescorer +can tell an undefined zero from a measured zero when the window-based families +collapse. Names: `n_peak_scans`, `peak_window_degenerate` (1 when fewer than 3 +non-empty scans, mirroring `order_consistency::MIN_SCANS`). + +### apex_dispersion (13, `apex_dispersion.rs`) + +Fragment apex dispersion and consensus peak shape, intensity-independent +(breadth of co-elution rather than height). Names: `frag_apex_rt_std`, +`frag_apex_rt_mad`, `frag_apex_max_dev`, `frag_apex_mean_dev`, +`frag_apex_agree_frac`, `precursor_frag_apex_delta`, `peak_symmetry`, +`peak_tailing`, `peak_n_local_maxima`, `peak_shoulder_score`, `peak_fwhm_scans`, +`peak_truncation`, `apex_frac_of_window`. `precursor_frag_apex_delta` reads the +mono MS1 XIC (`ms1_xic[0]`) and is populated under the same MS1/grid conditions +as the `ms1` XIC block; otherwise it is 0.0. Has unit tests +(`apex_dispersion.rs:240`). + +### mass_uncertainty (10, `mass_uncertainty.rs`) + +Fragment mass-error distribution over matched fragments plus evidence-breadth. +Names: `frag_mass_err_median`, `frag_mass_err_abs_median`, `frag_mass_err_std`, +`frag_mass_err_iqr`, `frag_mass_err_max_abs`, `frag_mass_err_range`, +`effective_frag_count` (inverse participation ratio), `evidence_concentration` +(fraction in the strongest fragment), `frac_top3_pred_observed`, +`frac_top5_pred_observed`. Has unit tests (`mass_uncertainty.rs:142`). + +## The Percolator PIN (`write_pin`, `features.rs:1537`) + +Streamed row-by-row through a `BufWriter` (not materialized as one String). +Header: `SpecId\tLabel\tScanNr\tExpMass\tCalcMass\t\t +Peptide\tProteins`. Per row: `SpecId = cand_`, +`Label = -1` when `label == "decoy"` else `1`, `ScanNr = candidate_id`, +`ExpMass = CalcMass = precursor_mz` (`{:.5}`), each feature at `{:.6}`, then +`Peptide = -..-` and `Proteins = `. The feature matrix +passed to the PIN is built column-parallel from `fmap` in `active_features` +order (`features.rs:945`), so the PIN and the Parquet share the same ordered +feature list. + +## The feature-schema hash (`feature_schema_id`, `features.rs:233`) + +`blake3_str(cols.join(","))` of the ordered active column list. Written to +`.schema.json` and recorded in the report `stats`. It is a content hash of +the exact ordered names, so any addition, removal, or reordering changes the id +and a classifier trained under one schema is never silently applied under +another. Because the family registry order is frozen and append-only, appending +a new family or feature at the end changes the id predictably while leaving all +prior positions stable. + +## bound_from_confident (elution-boundary calibration, `features.rs:658`) + +When `bound_from_confident` is true (the default), the stage learns one pair of +elution half-widths `(L, R)` in seconds from the confident-target seed set +(`spectrum_q <= 0.01`, `label == "target"`; the same anchor set used for RT +calibration and DeepLC fine-tune). For each confident candidate it detects the +per-candidate peak with `elution_peak_rt_bounds` (`features.rs:1141`, which +returns `None` for a candidate with fewer than 3 distinct scans, so it does not +contribute an anchor) and records `apex - lo` and `hi - apex`. If at least 20 +anchors resolve, it takes the `bound_confident_pct` percentile of the left and +right half-widths (median by default) and returns `Some((L, R))`; every +candidate is then bounded on `[apex - L, apex + R]` via `global_bound_indices` +(`features.rs:1114`), which falls back to the single apex-nearest scan +`(ai, ai)` when the mapped window collapses between grid points (sparse grid or +a half-width below one cycle). This +removes per-candidate boundary manipulation, so a chimeric decoy is scored over +a real-peptide-width window centred on its apex rather than one it can widen or +narrow. With fewer than 20 anchors the stage logs a warning and falls back to +per-candidate boundary detection for that run. When the flag is false, every +candidate detects its own peak boundary from its top-3-predicted-fragment +profile (the legacy path). The same `global_bounds` argument threads into both +`fragment_features` and `build_evidence`, so Minimal/Rich and Extended see the +identical window. + +## The shared stats kernel (`stats.rs`) + +One implementation of `pearson`, `cosine`, `spectral_angle`, used by +`fragment_features` and every family (do not reimplement). `pearson` +(`stats.rs:6`) is population Pearson with a zero-variance guard returning 0.0 +for `n < 2` or zero variance. `cosine` (`stats.rs:29`) returns 0.0 if either +vector is all-zero. `spectral_angle` (`stats.rs:44`) is +`1 - 2 * acos(clamp(cosine, -1, 1)) / pi`, in [0,1] with 1 = identical. Families +add local specializations that do not belong in the shared kernel (weighted +Pearson, Spearman via `pearson` on average ranks, Kendall tau, windowed cross- +correlation via `super::best_xcorr`, `features.rs:1484`, which returns the best +normalized correlation and its integer lag over `[-maxlag, maxlag]` as +`(lag_of_max, value.max(0.0))`), but the base Pearson/cosine call the shared +functions. + +## Configuration + +`FeaturesConfig` (`config.rs:578`) is `#[serde(default, deny_unknown_fields)]`, +so every field defaults independently and an unknown config key is a hard load +error. The `set` field's default is `t()` (`config.rs:614`, helper at +`config.rs:163`), a generic +`Default`-forwarding helper, so it resolves to `FeatureSet::default()` = +`Minimal`. The config was pruned of dead fields (the `FeatureSet::Custom` +variant no longer exists). + +| field | default | effect | +|---|---|---| +| `set` | `Minimal` | which set `active_features` returns (Minimal 14 / Rich 44 / Extended 381) | +| `coelution_corr_threshold` | 0.9 | threshold for `n_coelution_above` (count of pairwise fragment correlations at or above it) | +| `prec_tol_ppm` | 20.0 | precursor tolerance carried for feature bookkeeping | +| `bound_features` | true | restrict trace-based features to the elution peak instead of the whole extracted window; **gates only the Minimal/Rich `fragment_features` path** (Extended `build_evidence` always peak-bounds, see gotchas) | +| `bound_peak_fraction` | 1/3 | peak-boundary threshold as a fraction of apex height (DIA-NN-style; matched DIA-NN RT bounds best) | +| `bound_peak_grace` | 0 | consecutive sub-threshold scans to bridge before stopping (0 = stop at first miss; 1 bridges a single-scan dip) | +| `bound_from_confident` | true | learn one global left/right half-width from the confident seed set and apply it to every candidate; false = per-candidate detection | +| `bound_confident_pct` | 50.0 | percentile of the confident-set half-widths taken as the global half-width (50 = median) | + +Note that the fragment tolerance used inside `mass_accuracy` is a hardcoded +`FRAG_TOL_PPM = 20.0` (`mass_accuracy.rs:44`), not `prec_tol_ppm`; Evidence does +not carry the configured fragment tolerance. + +## Invariants, determinism, gotchas + +- Every family must return exactly `NAMES.len()` values in `NAMES` order; a + `debug_assert_eq!` in `extended_values` (`features.rs:143`) and in most + families catches a mismatch in debug builds. Non-finite values are coerced to + 0.0 at family boundaries and again in `extended_values`. +- The `FAMILIES` registry order and each family's `NAMES` order are the frozen + schema; they are append-only. Reordering or renaming changes `schema_id` and + invalidates any trained classifier. +- Deduplication is stable and precomputed once (`extended_value_plan`, + `features.rs:112`), reproducing the same survivors and order as + `extended_name_refs`, so names and values stay in lockstep across runs. +- Determinism: the parallel per-PSM pass collects into a `Vec` indexed by row, + so the serial assembly is byte-identical to a serial run regardless of thread + count. RT-axis alignment maps intensities keyed by `f32::to_bits` + (`features.rs:407`), which is exact-equality safe because the same `rt` + values are reused, not recomputed. +- "Matched" throughout the families means `obs_apex[i] > 0.0` (observed at the + apex scan), which differs subtly from "present in the peak" used by the + `nonzero` family (per-fragment peak-max `> 0`). +- The apex scan samples a single grid point; a fragment peaking one scan off + reads 0.0 at the apex. The `nonzero` family exists specifically to give the + classifier zero-tolerant variants alongside the originals. +- `peptide_length` (`features.rs:250`) strips a leading `DECOY_` prefix before + counting residues, and ignores bracketed modifications, so the decoy marker is + not a length-based target/decoy label leak (tested at `features.rs:1581`). +- MS1 XIC features require `ms1_*` chromatogram rows. Normal `run` supplies + converted MS1 spectra to extract, which writes those rows when a usable grid is + present; older artifacts or standalone extraction without `--ms1` legitimately + leave the features at 0.0. +- `bound_features` gates only the Minimal/Rich `fragment_features` path + (`features.rs:1286`): when false, that path scores over the whole extracted + window. The Extended `build_evidence` (`features.rs:360`) takes no such flag + and always peak-bounds `axis`/`traces` while still retaining + `axis_full`/`traces_full`, so Extended families read whichever window they + name regardless of `bound_features`, and `global_bounds` from + `bound_from_confident` always applies to them. +- The peptidoform grouping for cross-charge features uses the ProForma string, + which is charge-independent and keeps `DECOY_` peptidoforms grouped among + themselves, so it is not a target/decoy label leak (`features.rs:714`). + +## How to extend / modify + +- To add a new Extended family: create a module under `stages/features/` + exposing `pub const NAMES: &[&str]` and `pub fn values(&Evidence) -> Vec` + of matching length and order, `mod`-declare it (`features.rs:32`), and append + `(name, values)` to `FAMILIES` (`features.rs:52`). Appending at the end keeps + all prior schema positions stable. Reuse `crate::stats` and the parent helpers + (`mean`, `normalize_sum`, `best_xcorr`, `smooth3`, `peak_bounds`) rather than + reimplementing kernels. Add an arity unit test (`values(&e).len() == + NAMES.len()`) and a degenerate-evidence finiteness test. +- To add a Minimal/Rich feature: append the name to `MINIMAL_FEATURES` or + `RICH_EXTRA` and push its value in the serial loop (`features.rs:820`); make + sure no Extended family already uses the name, or it will be dropped as a + reserved collision. +- New names must be globally unique across Minimal, Rich, and every family; a + collision is silently dropped by the dedup filter, so run + `feature_sets_sized` (`features.rs:1590`) after any change to confirm the + size and uniqueness invariants. +- Do not reach into vendor formats or duplicate the mass model / stats kernel; + Evidence is the sole per-PSM interface for families, and any new scalar a + family needs must be added to `Evidence` (`features.rs:293`) and filled by the + caller (`features.rs:784`). +- Prefer adding a config field (backed by a default) over hardcoding a + threshold, consistent with the project convention; the current hardcoded + fragment tolerance in `mass_accuracy` is a documented exception awaiting a + tolerance carried on `Evidence`. diff --git a/docs/11_compete_rescore_fdr.md b/docs/11_compete_rescore_fdr.md new file mode 100644 index 0000000..e8a4c06 --- /dev/null +++ b/docs/11_compete_rescore_fdr.md @@ -0,0 +1,548 @@ +# compete, rescore, and FDR + +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +This subsystem is the tail of the identification chain (PLAN.md Stage F). It +turns the per-PSM feature table produced by the `features` stage into a scored, +FDR-controlled result set. It has three parts: + +1. **compete** (`mumdia compete`): within each competition group, resolve + redundant candidates for the same elution peak before target-decoy counting, + so multiple plausible candidates for one peak cannot each be counted as a + discovery. Default behaviour keeps only the best-scoring candidate per group. +2. **rescore** (`mumdia rescore`): train a semi-supervised classifier over the + competed PSMs of the whole experiment, produce a single discriminant `score` + per PSM, and derive native target-decoy q-values at several aggregation + levels (PSM, per-run PSM, precursor, peptide, protein group). +3. **fdr** (`crate::fdr`): the shared, stateless q-value kernels. Both + `search-seed` and `rescore` call these; they are not a stage. + +The design principle behind the sensitivity work is "preserve candidate evidence +until the workflow can make a well-calibrated decision". That is why every +non-default competition mode and the label rule keep decoys and redundant +variants alive so the rescorer and FDR have a valid null to work against. + +## Files + +| path | role | +|---|---| +| `rust/mumdia/crates/mumdia/src/stages/compete.rs` | `compete` stage: grouping, within-group competition resolution, competed table + optional audit | +| `rust/mumdia/crates/mumdia/src/stages/rescore.rs` | `rescore` stage: input concat, classifier dispatch, multi-context q-values, scored table | +| `rust/mumdia/crates/mumdia/src/rescoring.rs` | `percolator_lite`, the native semi-supervised linear rescorer (the `native_tda` path) | +| `rust/mumdia/crates/mumdia/src/fdr.rs` | q-value kernels: `target_decoy_q`, `entrapment_q`, `count_targets_at_q`, `validate_labels`, `ln_factorial` | +| `rust/mumdia/crates/mumdia-core/src/rejection.rs` | `RejectionReason` codes; compete's audit `rejection_reason` uses `code()` | +| `rust/mumdia/crates/mumdia-core/src/config.rs` | `CompeteConfig`, `CompetitionMode`, `CompeteGroupBy`, `RescoreConfig`, `RescorerKind` | +| `scripts/mokapot_worker.py` | Mokapot PIN sidecar (`RescorerKind::Mokapot`) | +| `scripts/nn_rescore_worker.py` | PyTorch MLP PIN sidecar (`RescorerKind::NnTorch`) | +| `scripts/entrapment_worker.py` | entrapment GBM sidecar (`RescorerKind::Entrapment`) | + +## Inputs and outputs + +### compete + +Consumes the features artifact `psms_features` (path passed as `--features`), +plus its schema companion `.schema.json` (read at compete.rs:41 via +`FeatureSchema::read`). Reads these columns (compete.rs:31-47): `candidate_id` +(u32), `label` (str), `base_peptide_id` (u32), `peptidoform` (str), `protein` +(str), `apex_rt`, `elution_lo`, `elution_hi` (f64), `precursor_mz` (f64), +`prelim_score` (f64), `charge` +(f64, only needed for `peptidoform_charge` grouping), and every feature column +named in the schema. + +Produces `psms_competed` (schema version 2, `artifact::PSMS_COMPETED`, +schema.rs:20) at `--out`. Column schema includes `candidate_id` +(u32), `label` (str), `base_peptide_id` (u32), `peptidoform` (str), `protein` +(str), `apex_rt`, `elution_lo`, `elution_hi` (f64), `precursor_mz` (f64), +`prelim_score` (f64), then every +feature column (f64) carried through unchanged. It also writes +`.schema.json` (compete.rs:153) so rescore recovers the exact feature list, +and `.report.json`. + +Optional sidecar `.compete_audit.parquet` (only when +`compete.emit_competition_audit = true`, compete.rs:158-203): one row per removed +candidate with `candidate_id` (u32), `label` (str), `peptidoform` (str), +`winner_candidate_id` (u32), `loser_prelim` (f64), `winner_prelim` (f64), +`rejection_reason` (str). The reason string is `RejectionReason::code()` +(rejection.rs:50), which is `SCREAMING_SNAKE_CASE`, so the two values written here +are `OUTCOMPETED_BY_DECOY` and `OUTCOMPETED_BY_TARGET` (rejection.rs:64-65), not +lowercase. The reason is keyed on the **loser's own label** (`reason_of`, +compete.rs:159-165): a removed decoy is `OUTCOMPETED_BY_DECOY`, a removed target is +`OUTCOMPETED_BY_TARGET`. Because competition is within-label, the winner shares the +loser's label, so this equivalently describes the winner. `RejectionReason` +(rejection.rs:19-45) has 17 variants spanning the whole loss ladder; compete only +ever emits these two. + +### rescore + +Consumes one or more `psms_competed` tables (`--competed`, a `Vec`). +The ordered feature schema from the first input is the expected contract, and +every later schema companion must match it exactly before tables are +concatenated. Reads `candidate_id`, `label`, `base_peptide_id`, `peptidoform`, +`protein`, `charge` (f64, cast to i32), `prelim_score`, `precursor_mz`, +`apex_rt`, `elution_lo`, `elution_hi`, and the feature columns. + +Produces `psms_scored` (schema version 3, `artifact::PSMS_SCORED`, schema.rs:21) +at `--out`. Column schema includes `candidate_id` (u32), +`peptidoform` (str), `charge` (i32), `label` (str), `protein` (str), +`base_peptide_id` (u32), the carried identification `apex_rt`/`elution_lo`/ +`elution_hi` (f64), `score` (f64, the classifier discriminant), +`q_value` (f64, pooled PSM q), `peptide_q_value` (f64), `protein_group` (str, +the protein-accession-set string, a duplicate of `protein`), `pg_q_value` (f64), +`global_q_value` (f64, byte-identical alias of `q_value`), `prelim_score` (f64), +`source` (u32, index into `--competed` identifying the run), `run_psm_q` (f64), +`experiment_psm_q` (f64, alias of `q_value`), `precursor_q` (f64). Plus +`.report.json` whose `params` records `classifier` (the path actually taken), +`classifier_requested`, `strict`, `folds`, `num_iter`, `train_fdr`, +`feature_schema_id`, `competed_inputs`, `config_hash` (rescore.rs:501-511), +`model_identity` (e.g. `native-percolator-lite-v1`, `mokapot-` +(default `mokapot-nn`), `nn-torch-semisup-sidecar-v1`, +`entrapment-gbm-sidecar-v1`, `native-percolator-lite-entrapment-v1`, +rescore.rs:144/169/196/251/261/272), and `stats` records `psms`, `classifier`, +`target_psms_at_1pct`, `target_peptides_at_1pct`, `target_protein_groups_at_1pct`, +`target_precursors_at_1pct`, plus `entrapment_ratio` and +`entrapment_peptides_at_1pct` in entrapment mode (rescore.rs:480-493). + +Sidecar working files are written under `--work-dir` (created with +`create_dir_all`, rescore.rs:689, 755). Two distinct file contracts exist: + +- **PIN sidecars** (`RescorerKind::Mokapot`, `RescorerKind::NnTorch`): input + `rescore.pin` (Percolator tab format), output `rescore_sidecar_out.parquet` + (rescore.rs:756-757). The PIN header is `SpecId Label ScanNr ExpMass + CalcMass Peptide Proteins` (rescore.rs:760-762); each row is + `psm_{i}`, `Label` = `-1` for a decoy else `+1` (rescore.rs:770), `ScanNr` = `i` + (the flat row index), `ExpMass` = `CalcMass` = `precursor_mz` (5 decimals), + the feature columns in schema order (6 decimals), `Peptide` = `-..-`, + `Proteins` = `` (rescore.rs:771-776). The worker echoes the SpecId tail + as a `candidate_id` column (here the row index `i`) and emits a `score` column; + scores are mapped back by row index through `align_sidecar_scores`, which + requires every input row to be covered exactly once with a finite score and + hard-errors on a missing, duplicate, out-of-range, or non-finite entry + (rescore.rs:802-805, 811-847). There is no worst-score fallback for a missing + row. +- **Entrapment GBM** (`RescorerKind::Entrapment` with `rescore.python` set): input + `entrapment_in.parquet`, output `entrapment_out.parquet` (rescore.rs:690-691). + The input parquet columns are `row_id` (u32, the flat row index used for + score readback), `candidate_id` (u32), `base_peptide_id` (u32), `is_entrapment` + (i32, 0/1), `is_decoy` (i32, 0/1), then the feature columns in schema order + (f64) (rescore.rs:693-714). The worker is invoked as + `python entrapment_worker.py ` (rescore.rs:717-724, `folds` + from `cfg.folds`); it reads back `row_id` (u32) + `score` (f64) and maps by + `row_id` through the same `align_sidecar_scores` coverage contract, so an + incomplete or duplicated response hard-errors (rescore.rs:729-732, 811-847). + Both sidecars are run with `PYTHONUTF8=1` and a non-zero exit is a hard error + (rescore.rs:723/725-727, 785/794-796). + +## How it works + +### compete: grouping + +`compete::run` (compete.rs:28) builds a competition group key per PSM +(compete.rs:74-95). The key is a fixed-size tuple `(u32, u8, i64)` rather than a +freshly allocated String, chosen so grouping does not allocate per PSM: + +- element 0 is `base_peptide_id` for `Precursor`/`Apex`, or a dense + first-appearance `pform_id` for `PeptidoformCharge` (built at compete.rs:54-64); +- element 1 is the label code `0=target, 1=decoy, 2=other` (compete.rs:76-80); +- element 2 is a bucket: constant `0` for `Precursor`; the rounded apex-RT bucket + `(apex_rt / apex_rt_tolerance_s).round()` for `Apex` (compete.rs:84); the + rounded charge for `PeptidoformCharge` (compete.rs:90). + +**The label is part of the key on purpose.** A target is never competed against +its own decoy; competition only arbitrates redundant charge/mod variants within +the target population and, separately, within the decoy population. If a target +could evict its paired decoy, the decoy population would be depleted and the +target-decoy null would collapse, badly underestimating FDR. This is stated at +compete.rs:66-73 and re-stated on `CompetitionMode` in config.rs:666-671. + +`PeptidoformCharge` (config.rs:698-701) is the precursor-level grouping DIA-NN and +Spectronaut report at: sibling charges of one peptide are kept as separate groups +rather than collapsed. It requires the `charge` column and bails if absent +(compete.rs:47-50). + +### compete: resolution + +`resolve_competition` (compete.rs:283) is a pure, unit-tested function. It visits +group keys in **sorted order** (compete.rs:292-293) for determinism, and picks +the winner as the highest `prelim_score`, ties broken by smallest row index +(compete.rs:298-306). The per-mode behaviour (`CompetitionMode`, config.rs:674): + +- `WinnerTakeAll` (default): keep only `win`; everything else is a removal pair + `(loser, winner)` (compete.rs:311-320). +- `None`: keep all members, remove nothing (compete.rs:308-310). FDR handles the + ambiguity downstream. +- `FeaturesOnly`: identical retained set to `None` (same match arm, + compete.rs:308); the distinct name documents intent for the experiment matrix, + the idea being that conflict/contested features carry the interference signal + into rescoring instead of removal. +- `UniqueEvidence`: keep the winner; keep a loser only when its unique-fragment + evidence `>= unique_evidence_min_fragments` (compete.rs:321-334). Evidence comes + from `unique_evidence` (compete.rs:242-260): prefers an explicit + `unique_fragment_count` column, else approximates it as + `n_matched_fragments * (1 - contested_frac)` (the multiplier `(1 - contested)` + clamped to [0,1]), else raw `n_matched_fragments`, else `None`. The contested + fraction is resolved by `prefer_peak_contested_fraction` (compete.rs:263-268), + which prefers the Extended-feature `peak_contested_frac` column and falls back to + the legacy `contested_frac` spelling. Each column is read through `col_f64` + (compete.rs:271-277), which accepts either an f64 or an i32 encoding, so an + integer-typed fragment count is handled without a schema mismatch. If the mode is + selected but no column is available it warns and falls back to winner-take-all + (compete.rs:102-108, 328 `.unwrap_or(false)`). +- `MarginGated`: keep the winner; remove a loser only when + `prelim[win] - prelim[m] >= margin`, otherwise keep it (compete.rs:335-347). + Conservative removal for the low-FDR region. + +The returned `keep` indices are sorted and deduped (compete.rs:350-351). Kept +rows are projected into the output columns (compete.rs:120-150), and the schema +is forwarded. The report records `group_by`, `mode`, `input_rows`, `kept`, +`removed` (compete.rs:206-221). + +### rescore: input concat and classifier dispatch + +`rescore::run` (rescore.rs:41) concatenates all `--competed` tables into flat +vectors (rescore.rs:76-107). It records a per-PSM `source` = the index of the +input file the PSM came from (rescore.rs:70, 104); for a single-run rescore this +is all-zero. `source` is why the PIN and entrapment sidecars key on a unique flat +row index rather than `candidate_id`: `candidate_id` is the library index and +repeats across runs, so an experiment-wide table would collide on it +(rescore.rs:65-69, 693-697, 763-768). + +Immediately after loading, `crate::fdr::validate_labels` (rescore.rs:108) rejects +any label that is not exactly `"target"` or `"decoy"`. `is_decoy` is derived at +rescore.rs:109; entrapment status is derived separately from the protein string by +`classify_entrapment` (rescore.rs:110, 577). Its exact rule (rescore.rs:588-603): a +decoy is neither entrapment nor real; a non-decoy is `is_entrapment` when its +protein **contains** `entrapment_marker` **and** (no `entrapment_exclude` set, or +the protein does not contain it) **and** the protein matches none of +`entrapment_contaminant_markers`; every other non-decoy is a real target. When +`entrapment_marker` is `None` nothing is entrapment and every non-decoy is real. +The contaminant carve-out exists so genuine contaminants living inside the spike-in +proteome (keratins, albumin) are not mislabeled as false negatives. + +The classifier is dispatched on `RescoreConfig::classifier` (rescore.rs:152-277). +The stage tracks `classifier_used`, `model_identity`, and `qmode` so the report +reflects the path actually taken rather than the requested one (rescore.rs:143-145). + +**`RescorerKind::NativeTda`** (config.rs:104, the default): calls `native_scores` +-> `percolator_lite` (rescore.rs:216, 552-568). `percolator_lite` +(rescoring.rs:100) is a Percolator/Mokapot-style linear model: + +- Fold assignment is `fold_key % folds` where `fold_key = base_peptide_id` + (rescoring.rs:109), so every charge/mod variant of a peptide lands in the same + fold and no peptide leaks between train and test. "CV folds by candidate + hashing" is the modulo-of-the-id scheme; the id used is the base-peptide id. +- Folds are processed in parallel (`rayon`, rescoring.rs:114-115) but each fold is + independent and scores only its disjoint test set, so the result is + order-independent and deterministic. +- Each fold fits its **own** standardizer on its training rows only + (`fit_standardizer`, rescoring.rs:14-40, 122), which is the leak-free choice: + test-fold statistics never enter standardization. std < 1e-9 is clamped to 1.0. +- Semi-supervised loop (`num_iter` iterations, rescoring.rs:131-176): compute + target-decoy q on the current train scores, take confident targets + (`q <= train_fdr`) as positives and all decoys as negatives, fit an L2 logistic + regression (`logreg_fit`, rescoring.rs:50-77; full-batch gradient descent, + `l2=1e-3`, `epochs=200`, `lr=0.5`, weight[0]=bias), and re-score the train fold. + If fewer than 10 confident targets exist, it falls back to the top-scoring half + as positives (rescoring.rs:155-171). +- Final score for each test row is `score_row(w, std_row(...))` written back by + original index (rescoring.rs:178-190). Weights start at zero and there is no + RNG, so the whole path is deterministic. +- **The result vector is seeded with `init_score` (the `prelim_score`)** + (rescoring.rs:185). Only rows a fold actually scores are overwritten, so any row + a fold cannot cover retains its prelim score. A fold whose training or test set + is empty produces nothing and is skipped (rescoring.rs:119-121); this happens + when a `fold_key` bucket is empty, e.g. very few peptides or all peptides landing + in one fold. `folds` is clamped to at least 1 (rescoring.rs:105) and `num_iter` + to at least 1 (rescoring.rs:131), so degenerate config still runs one fold and + one iteration. `fit_standardizer` divides by `idx.len().max(1)` (rescoring.rs:16) + so an empty index does not divide by zero. + +**`RescorerKind::Mokapot`** (config.rs:106): runs `mokapot_worker.py` through +`run_pin_sidecar` (rescore.rs:153-181, 740). On success it uses the returned +scores; on failure it either hard-errors (if `rescore.strict`) or warns and falls +back to `native_scores` (rescore.rs:172-180). Requires `rescore.python`. + +**`RescorerKind::NnTorch`** (config.rs:107-113): runs `nn_rescore_worker.py` +through the same `run_pin_sidecar` contract (rescore.rs:182-208). A nonlinear +PyTorch MLP with the same CV-fold + iterative positive-reselection scheme. The +initial feature/sign and every positive set are selected from that fold's +training rows only; empty, single-class, or zero-positive folds hard-error, so +held-out labels do not influence OOF scoring. The +worker receives the NN hyperparameters through environment variables +`MUMDIA_NN_FOLDS`, `MUMDIA_NN_ITERS`, `MUMDIA_NN_TRAIN_FDR`, set from +`cfg.folds/num_iter/train_fdr` (rescore.rs:790-792), so the report reflects the +values actually used. Same strict/fallback logic as Mokapot. + +> **NnTorch is seeded but not bit-deterministic.** The worker seeds numpy/torch +> (`nn_rescore_worker.py:284-285`) but floating-point training and numerical +> kernels mean runs are only approximately reproducible. It also has a +> **scaler leak**: standardization statistics are computed over the **full** +> feature matrix, not per training fold. In-memory backend uses global +> median/IQR (`nn_rescore_worker.py:138-141`); streaming backend accumulates a +> global mean/std in one pass (`:169-171`). This differs from the leak-free +> native `percolator_lite`, which fits the scaler on the train fold only. Both +> facts are properties of the sidecar, not the Rust dispatch. + +**`RescorerKind::Percolator`**: **not wired.** Normal config loading rejects this +classifier before execution; the defensive rescore arm still errors under strict +mode or falls back only for a manually constructed compatibility config. +`rescore.percolator_bin` is a dead field until an adapter exists. + +**`RescorerKind::Entrapment`** (config.rs:116-122): a decoy-independent path for +spike-in entrapment experiments. It requires `entrapment_marker` and at least one +matching PSM, else it warns/errors and falls back to native (rescore.rs:217-234). +With `rescore.python` set it runs `run_entrapment_gbm` (rescore.rs:236, 675): a +gradient-boosted sidecar trained out-of-fold by base peptide, positives = real +targets, negatives = spike-in targets (rescore.rs:675-733). Without python it +uses the native linear rescorer but with `is_entrapment` (not `is_decoy`) as the +negative label (rescore.rs:263, 274). Any of these sets `qmode = Entrapment`, so +q-values are computed by `entrapment_q` instead of `target_decoy_q`. + +### rescore: the multi-context q columns + +After scoring, the stage computes q-values at several aggregation levels, each an +**independent** target-decoy (or entrapment) analysis run on the appropriate +best-per-group reduction. This is deliberate: q-values at different levels are not +derived from one another, they are separately calibrated nulls. + +| output column | grouping | how computed | file:line | +|---|---|---|---| +| `q_value` | none (pooled PSM) | `target_decoy_q` / `entrapment_q` over all PSMs | rescore.rs:288-303 | +| `experiment_psm_q` | none (pooled PSM) | clone of `q_value` | rescore.rs:346 | +| `global_q_value` | none (pooled PSM) | clone of `q_value`, backward-compat alias | rescore.rs:345 | +| `run_psm_q` | by `source` | independent TDA within each run, scattered back by row index | rescore.rs:350-376 | +| `precursor_q` | `(peptidoform, charge)` | best PSM per precursor, TDA over that set | rescore.rs:379-396 | +| `peptide_q_value` | `base_peptide_id` | best PSM per base peptide, TDA over that set | rescore.rs:309-317 | +| `pg_q_value` | protein-accession-set string | best PSM per protein group, TDA over that set | rescore.rs:322-339 | + +The per-level reduction is `grouped_q` (rescore.rs:609): for each key, keep the +best-scoring member `(score, is_decoy, is_entrapment, is_real, row)` with an exact +score tie resolved in favour of the active null (decoy or entrapment) so input row +order cannot make the accepted set anti-conservative (rescore.rs:622-642), run the +chosen q kernel over the one-row-per-group set (rescore.rs:644-655), then assign +the group q **only to the winning row** of each group and give every losing sibling +q = 1.0 (rescore.rs:662-666). A lower-scoring charge/mod variant (possibly itself a +false target) must not inherit the winner's low q. Group counts dedup by key on the +winner, so counts are unchanged, but per-PSM peptide/pg/precursor q no longer +propagate to losers. + +`run_psm_q` groups rows by `source` in a `BTreeMap` for deterministic iteration +(rescore.rs:351-355) and runs a full independent TDA within each run, so a per-run +report gets a genuine per-run FDR instead of the pooled value. For a single-run +rescore (`source` all-zero) it equals `q_value`. + +**Why peptide q >= precursor q at the same threshold.** Both are best-per-group +TDA, but the peptide grouping (`base_peptide_id`) is coarser than the precursor +grouping (`peptidoform+charge`): several precursors collapse into one peptide. A +coarser grouping has fewer, higher-scoring representatives and a different +target/decoy balance among the survivors, so the monotonized q at a given level +is generally not lower than at the finer level. Reporting both lets a consumer +pick the FDR granularity that matches its claim (a peptide-level ID list vs a +precursor-level one). + +Interned dense u32 ids are used for the protein and precursor groupings +(rescore.rs:322-330, 379-387) purely as a performance optimization: interning the +accession-set string and the `(peptidoform, charge)` tuple to first-seen integers +avoids hashing and cloning hundreds of thousands of strings inside `grouped_q`. +The mapping is bijective, so grouping and the resulting q-values are unchanged. + +`is_reported` (rescore.rs:400-403) selects which rows count toward the 1% +summaries: real targets in entrapment mode (spike-in excluded), all non-decoys +otherwise. The report records `target_psms_at_1pct`, `target_peptides_at_1pct`, +`target_protein_groups_at_1pct`, `target_precursors_at_1pct`, and in entrapment +mode the `entrapment_ratio` and the entrapment-leak count `entrapment_peptides_at_1pct` +(spike-in peptides passing the 1% gate, a running FDR-validity check, +rescore.rs:437-445, 487-493). + +### fdr: the kernels + +`target_decoy_q` (fdr.rs:7) is the no-pi0 estimator +`q = (n_decoys + 1) / max(1, n_targets)`, monotonized: + +1. Sort record indices by descending score (fdr.rs:12-18). +2. Walk in score order, processing **tied-score blocks together** so every PSM in + a block gets the same FDR regardless of its arbitrary within-tie order + (fdr.rs:27-43). This is a determinism requirement (PLAN.md Section 7): a + target/decoy interleave inside one tie block must not change the q. +3. FDR at rank = `(td + 1) / max(1, tt)` where `td`, `tt` are cumulative decoy and + target counts at that score (fdr.rs:38). The `+1` is the conservative + finite-sample pseudocount; the bare `n_decoys/n_targets` is optimistic in the + low-count regime. +4. Monotonize from worst-scoring to best so q is non-increasing with score + (fdr.rs:45-51): `q[i] = min(fdr at all ranks worse-or-equal)`. + +The best target with perfect separation gets `q = 1/n_targets`, not 0 +(test at fdr.rs:149-165). + +`entrapment_q` (fdr.rs:64) is the empirical-null analog: +`FDR(t) = (ratio * n_entrap(>=t) + 1) / max(1, n_real(>=t))`. `ratio` = +`N_real_lib / N_entrap_lib` corrects for unequal library sizes; the `+1` is the +same pseudocount. Rows that are neither entrapment nor real (decoys) are ranked +but enter no count (fdr.rs:87-98). Same tied-block walk and worst-to-best +monotonization as `target_decoy_q`. Unlike in-silico decoys, the entrapment +population experiences the same chimeric DIA interference as real targets, so the +estimate is not optimistic (fdr.rs:54-63). Uses a stable sort so ties keep input +order (fdr.rs:75-80). + +`count_targets_at_q` (fdr.rs:115): count of non-decoy records with `q <= threshold`. + +`validate_labels` (fdr.rs:127): hard-error on any label other than `"target"` or +`"decoy"`. An unknown or malformed label must not silently count as a target +because the target-decoy null depends on exact labeling. Entrapment status is +derived from the protein accession, not the label, which is why only two label +values are valid here. + +`ln_factorial` (fdr.rs:137): `ln(n!)` via summed logs, used where matched-fragment +counts feed a hyperscore-style term; `n` is small so the naive loop is fine. + +## Key types and functions + +| name | file:line | what it does | +|---|---|---| +| `compete::run` | compete.rs:28 | full compete stage: group, resolve, write competed table + optional audit | +| `compete::resolve_competition` | compete.rs:283 | pure per-mode within-group resolution; returns kept indices + removal pairs | +| `compete::unique_evidence` | compete.rs:242 | derive per-candidate unique-fragment evidence for `UniqueEvidence` mode | +| `compete::prefer_peak_contested_fraction` | compete.rs:263 | choose the Extended `peak_contested_frac`, else the legacy `contested_frac` | +| `compete::col_f64` | compete.rs:271 | read a numeric column as f64, accepting an f64 or i32 encoding | +| `CompeteParams` | compete.rs:21 | `features`, `out`, `cfg`, `config_hash` | +| `rescore::run` | rescore.rs:41 | full rescore stage: concat, dispatch, multi-context q, scored table | +| `rescore::validate_feature_schema` | rescore.rs:532 | reject a concat whose feature companions differ in id or ordered columns | +| `rescore::native_scores` | rescore.rs:552 | thin wrapper calling `percolator_lite` with the config knobs | +| `rescore::classify_entrapment` | rescore.rs:577 | per-PSM `(is_entrapment, is_real_target)` from the protein string | +| `rescore::grouped_q` | rescore.rs:609 | best-per-group reduction + level q, winner-only assignment | +| `rescore::run_entrapment_gbm` | rescore.rs:675 | write parquet, run entrapment GBM worker, read scores by row_id | +| `rescore::run_pin_sidecar` | rescore.rs:740 | write PIN, run Mokapot/NnTorch worker, read scores by row index | +| `rescore::align_sidecar_scores` | rescore.rs:811 | validate exact/unique/finite sidecar coverage, map scores by row id | +| `QMode` | rescore.rs:23 | which null q is computed against: `Decoy` or `Entrapment` | +| `RescoreParams` | rescore.rs:30 | `competed`, `out`, `work_dir`, `script_dir`, `cfg`, `config_hash` | +| `percolator_lite` | rescoring.rs:100 | native semi-supervised L2 logreg rescorer with per-fold scaler | +| `RescoreInput` | rescoring.rs:87 | features, is_decoy, fold_key, init_score, folds, num_iter, train_fdr | +| `fit_standardizer` | rescoring.rs:14 | per-fold mean/std over training rows only (leak-free) | +| `logreg_fit` | rescoring.rs:50 | full-batch GD logistic regression with L2, weight[0]=bias | +| `std_row` | rescoring.rs:42 | standardize one feature row with a fold's mean/std | +| `score_row` | rescoring.rs:79 | linear discriminant `w[0] + sum(w[j+1]*r[j])` | +| `RejectionReason` | rejection.rs:19 | loss-ladder reason codes; compete emits `OUTCOMPETED_BY_{TARGET,DECOY}` | +| `target_decoy_q` | fdr.rs:7 | `(D+1)/T` monotonized tied-block q from `(score, is_decoy)` | +| `entrapment_q` | fdr.rs:64 | `(ratio*E+1)/R` monotonized tied-block entrapment q | +| `count_targets_at_q` | fdr.rs:115 | count non-decoy records at or below a q threshold | +| `validate_labels` | fdr.rs:127 | label whitelist; rejects anything but target/decoy | +| `ln_factorial` | fdr.rs:137 | `ln(n!)` via summed logs | + +## Configuration + +The config was recently pruned of dead fields; the fields below are the live ones +this subsystem reads. + +### `compete` (`CompeteConfig`, config.rs:626-664) + +| field | default | effect | +|---|---|---| +| `group_by` | `precursor` | groups charge/mod variants of a base peptide separately within target and decoy labels; `apex` also buckets by rounded apex RT; `peptidoform_charge` keeps each precursor form separate | +| `apex_rt_tolerance_s` | `5.0` | RT bucket width (s) for `group_by = apex` (compete.rs:84) | +| `mode` | `winner_take_all` | within-group resolution (`CompetitionMode`, config.rs:674): `winner_take_all`, `none`, `features_only`, `unique_evidence`, `margin_gated` | +| `margin` | `0.0` | score margin required to remove a loser under `margin_gated` (compete.rs:341) | +| `unique_evidence_min_fragments` | `2` | min unique-fragment count for a loser to survive under `unique_evidence` (compete.rs:323) | +| `emit_competition_audit` | `false` | write `.compete_audit.parquet` (compete.rs:158) | + +All non-default `mode` values are part of the sensitivity program and are +default-off; the production chain uses `winner_take_all` and is byte-identical +unless a knob is set. + +### `rescore` (`RescoreConfig`, config.rs:913-965) + +| field | default | effect | +|---|---|---| +| `classifier` | `native_tda` | which rescorer (`RescorerKind`, config.rs:100): `native_tda`, `mokapot`, `nn_torch`, `percolator` (unwired), `entrapment` | +| `folds` | `3` | CV folds: native `percolator_lite`, PIN sidecars via `MUMDIA_NN_FOLDS` env (rescore.rs:790), entrapment GBM via positional arg (rescore.rs:722) | +| `train_fdr` | `0.01` | q threshold for the confident-positive set in the semi-supervised loop | +| `num_iter` | `10` | semi-supervised iterations for the native rescorer | +| `python` | `None` | interpreter for the Mokapot/NnTorch/entrapment sidecars; required for those paths | +| `percolator_bin` | `None` | dead field until the `percolator` path is wired | +| `entrapment_marker` | `None` | protein substring marking spike-in negatives; required for `entrapment` | +| `entrapment_exclude` | `None` | substring that, if also present, keeps a PSM as a real target (shared peptides) | +| `entrapment_contaminant_markers` | `[]` | substrings marking genuine contaminants inside the spike-in proteome; matching PSMs stay real targets | +| `entrapment_ratio` | `1.0` | `N_real_lib / N_entrap_lib`, scales the entrapment FDR estimate | +| `strict` | `true` | production default: any sidecar failure / misconfiguration is a hard error; false explicitly enables compatibility fallback | + +## Invariants, determinism, gotchas + +- **Label stays in the competition key** (compete.rs:66-95, config.rs:670-671). A + target never directly eliminates its own decoy in the `compete` stage. + `rescore` still reduces target and decoy representatives by the requested + biological unit and compares those populations when estimating q values. + Removing the stage-level label partition would prematurely deplete the null. +- **compete is deterministic.** Groups are visited in sorted key order and the + winner tie-breaks to the smallest row index (compete.rs:292-306). Kept indices + are sorted+deduped (compete.rs:350-351). No floats are summed across an unordered + map. +- **Tied-score blocks share one q** in both `target_decoy_q` and `entrapment_q` + (fdr.rs:27-43, 86-104). Within-tie order is arbitrary and must not change the q. +- **`native_tda` is fully deterministic**: zero-initialized weights, no RNG, + per-fold work is order-independent even under rayon (rescoring.rs:114-183). +- **A row a fold cannot score keeps its `prelim_score`.** `percolator_lite` seeds + its output with `init_score` and only overwrites rows a fold actually scores + (rescoring.rs:185-190); an empty train/test fold is skipped (rescoring.rs:119-121). + So the discriminant `score` is a mix of learned scores and, for uncovered rows, + the raw prelim. This is the intended safe fallback, not a bug. +- **`nn_torch` is seeded but not bit-deterministic** and has a **scaler leak** (standardization + fit over the full matrix, not per fold; `nn_rescore_worker.py:138-141,169-171`). + Do not treat its scores as reproducible; do not use it where byte-identity is + required. +- **`percolator` is unwired.** Config loading rejects it. The defensive fallback + arm matters only to manually constructed compatibility configs; + `percolator_bin` remains a dead field. +- **Sidecars key on the flat row index, not `candidate_id`** (rescore.rs:65-69, + 693-697, 763-768). `candidate_id` is the library index and repeats across runs; + keying on it collides in an experiment-wide (multi-file) rescore. A missing row + in the sidecar output is a hard error, not a worst-score fallback: coverage must + be exact/unique/finite (`align_sidecar_scores`, rescore.rs:811-847, missing-row + bail at rescore.rs:843-845). +- **`global_q_value` and `experiment_psm_q` are exact clones of `q_value`** + (rescore.rs:345-346). `global_q_value` is kept only for backward-compat. +- **Losing siblings get q = 1.0** at the peptide/precursor/pg levels + (rescore.rs:662-666); do not read a loser's level q as its FDR. +- **Multi-context q-values are independent per level**, each a separate TDA on its + own best-per-group reduction. They are not derived from `q_value` by + aggregation. +- **`validate_labels` runs before any counting** (rescore.rs:108). Any label other + than target/decoy aborts the stage. +- **Schema companion is mandatory.** compete writes `.schema.json` + (compete.rs:153); rescore reads the first input's schema (rescore.rs:74) and + validates every later companion against it for identity and ordered feature + columns (`validate_feature_schema`, rescore.rs:76-78, 532-549). A divergent + feature schema hard-errors rather than silently misreading columns. + +## How to extend / modify + +- **Add a competition mode**: extend `CompetitionMode` (config.rs:674), add a match + arm in `resolve_competition` (compete.rs:307-348), and add a unit test alongside + the existing ones (compete.rs:365-450). Keep the label in the key and keep group + visitation in sorted order so determinism holds. +- **Add a grouping**: extend `CompeteGroupBy` (config.rs:695) and add a key arm in + compete.rs:81-93. If it needs a new column, guard for its presence as + `PeptidoformCharge` does (compete.rs:48-50). +- **Wire the `percolator` path**: replace the warn/bail at rescore.rs:209-215 with a + PIN round-trip. The PIN writer already exists (`run_pin_sidecar`, + rescore.rs:740); percolator consumes the same PIN, so the work is invoking + `percolator_bin` (config.rs:922) and parsing its output back into a per-row score + vector aligned to input order. Honor `strict`. +- **Add a rescorer sidecar** that follows the PIN contract: reuse `run_pin_sidecar` + (rescore.rs:740), which writes SpecId `psm_i` / ScanNr `i`, ExpMass=CalcMass=mz, + the feature columns in schema order, and `-..-` / `` + (rescore.rs:759-777). The worker must echo the SpecId tail as `candidate_id` and + emit a `score` column; scores are mapped back by that row index through + `align_sidecar_scores` (rescore.rs:802-805). + NN hyperparameters are passed as `MUMDIA_NN_*` env vars (rescore.rs:790-792). +- **Add a q-value context**: add a grouping vector and a `grouped_q` call + (mirror `precursor_q` at rescore.rs:379-396), then add the output column at + rescore.rs:447-477 and a 1% count if desired. Reuse the interning pattern for + string keys to avoid hashing large columns. +- **Change the FDR estimator**: `target_decoy_q` (fdr.rs:7) and `entrapment_q` + (fdr.rs:64) are the only two kernels; both `search-seed` and `rescore` call them, + so a change here is global. Preserve the tied-block walk and the worst-to-best + monotonization or determinism breaks. The `+1` pseudocount is intentional; do not + drop it to chase counts. +- **Fix the NnTorch scaler leak**: fit standardization per training fold in + `nn_rescore_worker.py` (mirror `fit_standardizer` in rescoring.rs:14-40) instead + of over the full matrix at `:138-141` / `:169-171`. diff --git a/docs/12_quant_lfq_align_mbr_report_audit.md b/docs/12_quant_lfq_align_mbr_report_audit.md new file mode 100644 index 0000000..0f17187 --- /dev/null +++ b/docs/12_quant_lfq_align_mbr_report_audit.md @@ -0,0 +1,594 @@ +# quant, quant-lfq, align, mbr, report, audit + +> Part of the MuMDIA developer documentation (see docs/README.md). + +## Purpose + +This document covers the six "tail" subcommands of the pipeline: the ones that run +after `rescore` has produced `psms_scored.parquet`, plus the two experiment-level +stages that operate across multiple runs. + +- **quant** (Stage G, `mumdia quant`): integrate per-fragment chromatograms over the + elution peak, sum the top-N fragments into a per-peptidoform quantity, and roll up + to protein groups. Single-run. +- **quant-lfq** (Stage G, `mumdia quant-lfq`): combine several per-run quant tables + into a protein-by-run abundance matrix by MaxLFQ (peptide-level) or directLFQ + (ion/fragment-level), with optional cross-run median-ratio normalization. +- **align** (Stage D2, `mumdia align`): put >=2 runs on a common RT coordinate by + fitting a reference LOESS RT map per run and recording the residual spread. +- **mbr** (Stage D3, `mumdia mbr`): partially wired match-between-runs identification + transfer. A Rust CLI + config gate that shells out to `scripts/mbr_worker.py`. +- **report** (`mumdia report`): emit human-readable `peptides.tsv` + `proteins.tsv` + from the scored PSM table, joined to quant. +- **audit** (`mumdia audit`): reconstruct, per candidate, the pipeline stage flags and + the earliest rejection reason across the artifact chain, without re-running compute. + +## Files + +| path | role | +|---|---| +| `rust/mumdia/crates/mumdia/src/stages/quant.rs` | `quant` stage + `run_lfq_combine`/`size_factors` for `quant-lfq` | +| `rust/mumdia/crates/mumdia/src/quant_lfq.rs` | MaxLFQ/directLFQ ratio-alignment core (`lfq_profile`, `maxlfq`, `directlfq`) | +| `rust/mumdia/crates/mumdia/src/stages/align.rs` | Stage D2 cross-run RT alignment | +| `rust/mumdia/crates/mumdia/src/stages/report.rs` | `peptides.tsv` + `proteins.tsv` writer | +| `rust/mumdia/crates/mumdia/src/stages/audit.rs` | candidate identification-loss ladder | +| `rust/mumdia/crates/mumdia-core/src/rejection.rs` | `RejectionReason` enum + ladder ordering | +| `rust/mumdia/crates/mumdia-core/src/config.rs` | `QuantConfig`, `MbrConfig`, `RtImTrainConfig`, and the strategy enums | +| `scripts/mbr_worker.py` | MBR transfer sidecar (rescuable + re-extraction tiers) | +| `rust/mumdia/crates/mumdia/src/main.rs` | CLI definitions + handlers for all six subcommands | +| `rust/mumdia/crates/mumdia/src/sidecar.rs` | `run_mbr` (builds argv for `mbr_worker.py`) | + +## Inputs and outputs + +### quant +Consumes `psms_scored.parquet` (from `rescore`) and `chromatograms.parquet` (from +`extract`). Produces two mandatory artifacts and two optional ones. + +`peptide_quant.parquet` (schema `PEPTIDE_QUANT` v2): + +| column | type | meaning | +|---|---|---| +| `candidate_id` | u32 | library candidate id | +| `base_peptide_id` | u32 | stripped/base-peptide key used to deduplicate protein rollup | +| `peptidoform` | str | ProForma peptidoform | +| `charge` | i32 | precursor charge | +| `protein_group` | str | protein-group key | +| `quantity` | nullable f64 | sum of the top-N positive finite fragment areas; null when not quantifiable | +| `quant_status` | str | `quantified` or the explicit reason quantity is missing | +| `n_fragments_used` | i32 | number of positive finite fragments actually summed | +| `integration_apex_rt` | nullable f64 | apex actually used for integration | +| `integration_lo_rt` / `integration_hi_rt` | nullable f64 | integration bounds actually applied | + +`protein_group_quant.parquet` (schema `PROTEIN_GROUP_QUANT` v2): +`protein_group` (str), nullable `quantity` (f64), `quant_status` (str), and +`n_peptides` (i32, number of unique positive base peptides before Top-N +truncation). Charge/modification siblings contribute only their maximum +single-run quantity to the base-peptide representative. + +`fragment_quant.parquet` (optional, `--out-fragment`; `quant.rs:558`): one row per +fragment area, `candidate_id`, `peptidoform`, `charge`, `protein_group`, +`fragment_name` (str), `quantity` (f64). Only positive finite fragment areas are +emitted. This is the input for ion-level directLFQ. + +`.parquet` (optional diagnostic, `--out-peak-bounds`, only when +`bound_peak` is on; `quant.rs:419`): `candidate_id`, `lo_rt`, `hi_rt`, `width_s` +(all f64). Not part of the quant contract; it is a view of the integration windows. +A row is emitted only for a candidate whose chosen `(lo_rt, hi_rt)` are both finite +(`quant.rs:399`), so unbounded/degenerate windows are absent. + +`peptide_quant`, `protein_group_quant`, and an emitted `fragment_quant` receive a +sidecar `.report.json`; the peak-bounds diagnostic does not. Recorded +params include the q filter, integration settings, config hash, and exact scored +and chromatogram inputs. Stats distinguish row counts from quantified and +nonquantifiable rows. + +### quant-lfq +Consumes N per-run tables (peptide_quant for maxlfq, fragment_quant for directlfq). +Produces a long-form matrix (`quant.rs:720`): `protein_group` (str), `run` (i32, +0-based input index), `quantity` (f64), `n_features` (i32). `n_features` is the +count of feature keys for that protein group, constant across its rows (`quant.rs:717`), +not the per-run non-missing count. One row per `(protein_group, run)` is written for +every run, including runs where the protein's profile is 0. Unlike `quant` and +`align`, `run_lfq_combine` writes no `.report.json`: it calls `write_table` and +logs, with no `ArtifactReport` (`quant.rs:720`). + +### align +Consumes one `seed_psms.parquet` per run (`--seeds`, first is the reference). +Produces `alignment.parquet` (`align.rs:130`): `run_id` (u32), `source_rt` (f64), +`reference_rt` (f64), `residual_spread` (f64). The mapping is emitted on a grid of +`grid_n` points per run (`grid_n = 100`, hardcoded in `main.rs:667`). + +### mbr +Consumes an experiment-wide `scored_combined.parquet` (must carry a `source` column) +and one `psms.parquet` per run in `source` order. Produces `.parquet`, one row +per accepted transfer (`mbr_worker.py:254`): `candidate_id`, `source`, `peptidoform`, +`charge`, `protein_group`, `label`, `expected_rt`, `observed_rt`, `rt_delta`, +`transfer_q` (10 columns). When there are no transfer candidates at all the worker +short-circuits and writes a placeholder table with a single empty `candidate_id` +column (`pa_write_empty`, `mbr_worker.py:289`), so `.parquet` always exists. +Optionally writes an augmented scored table (`--out-scored`) that lowers each accepted +transfer's `q_value` to `min(q_value, transfer_q)` on the matching `(candidate_id, +source)` row and adds an `is_transferred` bool (`mbr_worker.py:272`); this requires the +scored table to carry a `source` column. All MBR outputs are written by Python and +have no `report.json`. + +The unreachable re-extraction tier (`--emit-transfer-targets

`) instead writes +per-run `transfer_targets_.parquet` and `transfer_decoys_.parquet` in +`run_windows` format (`mbr_worker.py:142`): `candidate_id` (u32), `rt_pred_cal`, +`rt_lo`, `rt_hi` (f64), plus null `im_pred_cal`/`im_lo`/`im_hi` (f64) columns for the +3D data model. `run_mbr` never passes this flag, so `mumdia mbr` never produces these. + +### report +Consumes `psms_scored.parquet` and optionally the two quant tables. The CLI takes +`--out-dir` and derives `/peptides.tsv` + `/proteins.tsv` +(`main.rs:723`); `run` passes the run out-dir and `q_threshold = quant.q_threshold` +(`run.rs:467`). `peptides.tsv` header (`report.rs:95`): `precursor`, +`stripped_sequence`, `charge`, `protein`, `q_value`, `score`, `quantity`. +`proteins.tsv` header (`report.rs:134`): `protein_group`, `q_value`, `quantity`. No +Parquet or `report.json` is written; `report::run` returns `(n_precursors, +n_protein_groups)`, and the `mumdia report` handler prints a one-line summary +(`main.rs:733`). + +### audit +Consumes `library_precursors.parquet` (the full search space) plus `psms` +(extract), `competed` (compete), `scored` (rescore). It also attempts to read +`.audit.parquet` (`load_extract_reasons`, `audit.rs:51`) for extract-reason +refinement, but no stage in the current chain writes that file (see the gotchas), so +the read returns empty and refinement is inert. Writes `candidate_audit.parquet` +(16 columns, `audit.rs:180`) and `.metrics.json` (`audit.rs:216`). + +`candidate_audit.parquet` schema (`audit.rs:180`): + +| column | type | meaning | +|---|---|---| +| `run_id` | str | run identifier (`--run-id`; in `run` it is the out-dir path) | +| `precursor_id` | u32 | library `candidate_id` | +| `modified_sequence` | str | peptidoform (ProForma) | +| `charge` | i32 | precursor charge | +| `target_decoy_label` | str | `target`/`decoy` from the library | +| `entrapment_label` | bool | `protein` contains `entrapment_substr` (empty substr = always false) | +| `candidate_generated` | bool | always true (in the search space by construction, `audit.rs:168`) | +| `traces_extracted` | bool | present in `psms` (extract produced an accepted peak) | +| `peak_generated` | bool | equals `traces_extracted` (artifact resolution, `audit.rs:170`) | +| `peak_selected` | bool | equals `traces_extracted` | +| `variant_selected` | bool | present in `competed` | +| `target_decoy_winner` | bool | present in `scored` (survived to rescore) | +| `passed_precursor_fdr` | bool | scored `q_value <= q_threshold` | +| `passed_peptide_fdr` | bool | `passed_pep && passed_prec` (`audit.rs:175`) | +| `reported` | bool | equals `passed_precursor_fdr` (`audit.rs:176`) | +| `rejection_reason` | str | earliest-loss `RejectionReason::code()` | + +`.metrics.json` (`audit.rs:206`) records `run_id`, `q_threshold`, +`search_space`, `extracted`, `competed`, `reported`, `trace_recall` +(`extracted / max(1, search_space)`), and the per-reason `waterfall` map (as a sorted +`BTreeMap`). + +## How it works + +### quant (`quant.rs:259`, `run`) + +1. Load `psms_scored`. The q-value column to filter on is selected by + `cfg.q_filter`: `PeptideQ` -> `peptide_q_value`, `PrecursorQ` -> + `precursor_q`, `PsmQ` -> pooled `q_value`, `RunPsmQ` -> `run_psm_q`. + `PeptideQ`/`PrecursorQ` are grouped values suitable for a single-run rescore; + `RunPsmQ` is the run-local gate for a slice of an experiment-wide rescore. + Quant has no source selector: before per-run quantification, slice a pooled + scored table by `source` and pair that slice with the matching chromatograms. + Changing `q_filter` does not select a run. +2. Group chromatogram rows by `candidate_id` into `cand_rows` (`quant.rs:301`). Rows + whose `frag_name` starts with `ms1_` are MS1 isotope XIC pseudo-traces, not + fragment ions; they are excluded from both peak detection and the top-N sum. +3. **Phase 1** (only when `cfg.bound_peak`): compute a per-candidate elution + window `(lo_rt, hi_rt, apex_rt)` via `peak_window`. Schema-v3 scored tables + carry the exact identification apex through compete/rescore; quant anchors the + window at that apex. A missing, null, or non-finite apex from an older scored + artifact falls back to the legacy robust summed-XIC apex detector. The + summed XIC across all of a candidate's fragments is built in a `BTreeMap` keyed by + the f32 RT bit pattern, so both the union RT axis and the f64 summation order are + fixed (determinism, docs/18_findings_and_decisions.md contract B2; non-negative RTs make bit order equal value + order). When the summed XIC has fewer than two distinct RT samples nothing can be + bound: `peak_window` returns `(NEG_INFINITY, INFINITY)` with the lone RT as apex + (or NaN when empty), an unbounded window (`quant.rs:107`). Otherwise the apex is + chosen by a co-elution rule (`quant.rs:134`): among scans whose co-eluting + nonzero-fragment count is at least `thresh = max(max_cnt - 1, 1)` (`quant.rs:135`, + the `-1` for robustness, the `.max(1)` floor requiring at least one co-eluting + fragment), take the highest summed intensity; fall back to a plain summed argmax + only if no scan qualifies. That detector is now a compatibility fallback, not + the normal source of the quant apex. `peak_bounds` then walks out from the apex with + `peak_fraction` and `peak_grace`. A collapsed `lo==hi` window is widened to the + adjacent grid scans so `trapezoid_window` never returns a raw height (units bug, + `quant.rs:164`). +4. **Consensus mode** (`quant.rs:342`, `peak_window_mode == Consensus`): peak width is + treated as a near-constant instrument/gradient property. Over confident target + peptides (`pep_q <= reliable_q`) it takes the median left half-width `apex - lo` and + right half-width `hi - apex`, and applies `(apex - ml, apex + mr)` around each + candidate's apex. Requires `>= 20` anchors (`quant.rs:364`), else falls back to the + per-candidate windows. The consensus is local to one quant invocation; separate + runs estimate separate widths unless an external workflow supplies a shared + policy. +5. **Phase 2** (`quant.rs:385`): integrate each fragment trace. With `bound_peak` off, + integrate the whole trace with `trapezoid` (`quant.rs:38`). With it on, restrict to + the chosen window via `trapezoid_window` (`quant.rs:53`). Areas are accumulated per + candidate in `areas` and `frag_areas`. +6. **Top-N sum:** for each accepted target (`passes_quant_filter`, `quant.rs:177`), + `summarize_fragment_areas` (`quant.rs:185`) retains only positive finite fragment + areas, sorts descending, and sums the top `top_n_fragments`. Missing traces, + all-zero/non-finite areas, or `top_n_fragments=0` produce a null quantity plus + an explicit `quant_status`; they are not converted to biological zero. The + applied apex and bounds are written with the row. +7. **Protein rollup:** within each protein group, `add_protein_base_quantity` + (`quant.rs:220`) deduplicates charge/modification siblings by `base_peptide_id` + using their maximum positive quantity. `rollup_protein_bases` (`quant.rs:238`) + then sums the top `top_n_peptides` unique base peptides under `TopNSum`; `Sum` uses + all unique bases. A group with no quantifiable base peptide has null quantity + and `quant_status=no_quantifiable_peptide`. +8. Optional fragment export (`quant.rs:531`) and peak-bounds diagnostic + (`quant.rs:419`). `ArtifactReport` records params + stats for each table + (`quant.rs:603`). + +Trapezoid math: `trapezoid` sums `dt*(y_i+y_{i+1})/2` in f64 over consecutive RT +samples; a single sample returns its raw intensity (`quant.rs:39`). `trapezoid_window` +first filters to samples with `lo <= rt <= hi`, then calls `trapezoid` on the subset +so the single-sample rule is identical; an empty window integrates to 0. + +**Remaining limits.** A single positive fragment can still yield a quantity; the +status and `n_fragments_used` expose that evidence level but do not enforce a +minimum-clean-ion rule. Peak-window and fragment-selection policies remain +single-run choices rather than a learned cross-run consensus. Evaluate changes on +known-ratio data rather than identification count alone. + +### quant-lfq (`quant.rs:657`, `run_lfq_combine`) + +Reads each input table and builds `data: protein_group -> feature_key -> Vec>` +of length N runs (`quant.rs:666`). The feature key is `peptidoform|charge` for MaxLFQ +and `peptidoform|charge|fragment_name` for directLFQ (`quant.rs:682`). Missing entries +stay `None`. `size_factors` (`quant.rs:753`) computes one global size factor per run; +when `normalize != None` every present value is divided by its run factor before +rollup (`quant.rs:703`). For each protein group the feature-by-run matrix is passed to +`lfq_profile` (`quant_lfq.rs:84`) and the per-run abundances are written long-form. + +`lfq_profile` is the MaxLFQ least-squares reconstruction: +- Column sums/counts per sample give the fallback and the anchoring total + (`quant_lfq.rs:89`). Single sample returns the column sum (`quant_lfq.rs:101`). +- For each sample pair `(a,b)`, the median over shared features of `ln(va)-ln(vb)` is + an edge weight (`quant_lfq.rs:106`). Median-of-log-ratios is robust to a minority of + genuinely changing features. +- Connected components of the sample graph are found by union-find (`quant_lfq.rs:126`). + A singleton component falls back to its column sum. +- Each multi-sample component solves a Laplacian normal system `L x = c` with the first + variable fixed at 0 (`solve_fixed`, `quant_lfq.rs:28`, dense Gaussian elimination + with partial pivoting), giving log-abundances. The exp-profile is then scaled so the + component preserves its measured total intensity (`quant_lfq.rs:176`). + +`size_factors` methods (`quant.rs:753`): +- `MedianRatio` (default, DESeq-style): over complete-case features (positive in all + runs) take each run's log2 deviation from the per-feature mean; the run factor is + `2^median` of those deviations. Robust so a spike-in design's real fold changes are + not flattened (test `median_ratio_recovers_global_scale_not_real_changes`, + `quant.rs:918`). +- `Median`: align each run's median log2 intensity to the median of the per-run + medians. +- `None`: all factors 1.0. + +Determinism: medians sort in place, the matrix is iterated in `BTreeMap` key order. +With a single input, `run_lfq_combine` reduces to the per-run sum. +If no positive feature is complete across all runs, `MedianRatio` returns identity +factors (`1.0`) without estimating a scale correction. Inspect the logged +`size_factors`; identity may mean either balanced data or no usable complete cases. + +### align (`align.rs:53`, `run`) + +`confident_rts` (`align.rs:32`) reads a seed table's `base_peptide_id`, `spectrum_q`, +`observed_rt`, `score`, `label`, validates labels, and returns the best-scoring +observed RT per base peptide among confident targets (`q <= q_train`, target only). +The reference is `seeds[0]`. For each run, shared base peptides with the reference give +paired `(this_rt, ref_rt)`. A LOESS map is fit only when there are `>= 4` shared +peptides (`align.rs:93`, `Loess::fit(xs, ys, span=0.4, grid_n)`); the span is hardcoded +at 0.4 and is independent of `rt_im_train.loess_span`. The residual spread is the p95 of +`|ref - loess(this)|` on the shared set (`align.rs:105`); it is what sets how tight an +MBR window can be. The mapping is emitted on `grid_n.max(2)` evenly-spaced grid points; +the reference-run grid spans `[min(ref_rt, 0.0), max(ref_rt, 1.0)]` (`align.rs:58`) and +each other run's grid extends that span to cover its own shared RTs (`align.rs:88`). The +reference run and any run with too few shared peptides emit the identity map with +residual 0 (the insufficient-anchor guard, `align.rs:93`/`align.rs:113`). `align::run` +asserts at least one seed (`align.rs:55`) and writes an `alignment.report.json` +(schema `alignment` v1, `align.rs:141`, the only literal-string schema here since there +is no `schema.rs` constant for it). This is an experiment-level stage: with one run it +degenerates to identity, and it is not part of the `run` chain. Real multi-run +validation needs a multi-file experiment; only crafted two-run unit input exercises it. + +### mbr (partially wired Stage D3) + +The Rust side is a gate, not the algorithm. `Cmd::Mbr` (`main.rs:671`): loads config; +bails if `cfg.mbr.strategy == None` (`main.rs:680`); bails if fewer than 2 psms paths +(`main.rs:685`); requires `cfg.mbr.python` (`main.rs:688`); resolves `mbr_worker.py` +relative to the binary; calls `sidecar::run_mbr`. `run_mbr` (`sidecar.rs:162`) joins +the psms paths into a comma-separated `psms_csv` and forwards a fixed set of flags. + +**Which `MbrConfig` knobs are forwarded, and which are NOT.** Forwarded to the worker: +`q_anchor` (`--q-anchor`), `min_anchor_runs` (`--min-anchor-runs`), `q_transfer` +(`--q-transfer`), `consensus_corr_min` (`--consensus-corr-min`, only together with +`--frag-csv` and only when `frag` is non-empty and `consensus_corr_min > 0`, +`sidecar.rs:209`), plus `cfg.rng_seed` as `--seed` and the optional `--out-scored`. +NOT forwarded (present in `MbrConfig`, `config.rs:873`, but dead in the wired path): +- `strategy` beyond the None/not-None gate: `EmpiricalLibrary`, `RtTransfer`, `Full` + are indistinguishable to the worker, which always runs the rescuable transfer tier. +- `rt_window_s` (`config.rs:884`): the worker's `--rt-window` (default 20 s) is only + read by the `--emit-transfer-targets` re-extraction tier, which `run_mbr` never + invokes, so this knob has no effect on the wired path. +- `decoy_transfer` (`config.rs:886`): the worker hardcodes the permuted-RT null; + `ReverseSequence`/`Both` are not implemented in the worker. +- `requant_all` (`config.rs:892`): `Full`-only requantification, unused. + +The worker reads the scored table columns `candidate_id`, `source`, `label`, `q_value`, +`peptidoform`, `charge`, `protein_group` (`mbr_worker.py:81`) and per-run `apex_rt` from +each psms path. Run index 0 is the RT reference `REF` (`mbr_worker.py:101`); the +cross-run RT calibration (`to_ref`/`from_ref`, `mbr_worker.py:102`) uses `binned_map` +only when a run shares `>= 200` confident anchors with the reference, else the identity +map. The confident-anchor set uses targets only (a decoy anchor would inject a random +cross-run RT pair); decoys ride the same transfer test to measure the empirical +decoy-transfer fraction (`mbr_worker.py:87`). + +The worker (`scripts/mbr_worker.py`) implements two tiers: +- **Rescuable transfer** (default path, `mbr_worker.py:156`): for each precursor + confident (`q <= q_anchor`, target) in `>= min_anchor_runs` OTHER runs, sub-threshold + in a target run where it WAS extracted (`c in rt_all[i]`), predict its RT in that run + from the median of the other runs' binned-median-aligned apex RTs (`expected_rt`, + `mbr_worker.py:116`; calibration via `binned_map`, `mbr_worker.py:31`, default 80 + bins). The false-transfer FDR is a permuted-RT decoy-transfer null: each candidate is + given a shuffled candidate's predicted RT (`mbr_worker.py:177`). The transfer q-value + is standard target/decoy competition on `rt_delta = |observed - predicted|` (smaller + is better), computed as a running min from the tail (`mbr_worker.py:198`). Accept when + `q <= q_transfer`. When no transfer candidates exist at all it writes the empty + placeholder and returns (`mbr_worker.py:187`). +- **Fragment-consensus guard** (`mbr_worker.py:209`, M4 enhancement, active only with + `--frag-csv` and `--consensus-corr-min > 0`): reject an accepted transfer whose + observed fragment pattern in the target run has cosine `< corr_min` with the + empirical consensus (per-run L1-normalized, averaged over the confident runs) over its + confident runs. A transfer is also rejected if the candidate has no fragment data in + the target run or fewer than `min_anchor_runs` confident runs with fragments + (`mbr_worker.py:229`). Removes RT-concordant interference. +- **Re-extraction tier** (`--emit-transfer-targets`, `mbr_worker.py:126`): for the + ABSENT set (confident elsewhere, not confident here, and NOT extracted here) emit + per-run `run_windows`-format tables (`transfer_targets_.parquet`) at the tight + predicted-RT window plus a permuted-RT decoy target file (`transfer_decoys_.parquet`), + to feed `extract --restrict-candidates --run-windows`. This tier is fully implemented + in the worker but has no Rust CLI plumbing, so it is unreachable from `mumdia mbr`. + +The M5 augmented scored output (`--out-scored`, `mbr_worker.py:272`) requires the +scored table to have a `source` column and matches transfers on `(candidate_id, +source)`, taking `min(q_value, transfer_q)` and setting `is_transferred`. The worker +prints a validation summary (accepted counts per run, empirical decoy fraction, and the +RT window `delta_star` at `q_transfer`, `mbr_worker.py:245`). + +### report (`report.rs:49`, `run`) + +Reads the scored table columns `peptidoform`, `charge`, `protein`, `label`, +`peptide_q_value`, `protein_group`, `pg_q_value`, `score`. Builds two quant lookup +maps: peptide quant keyed `(peptidoform, charge)` (`report.rs:61`), protein quant keyed +`protein_group` (`report.rs:73`). Peptides: sort rows by `peptide_q_value` ascending, +keep the first (best-q) row per unique `(peptidoform, charge)`, targets only with +`peptide_q_value <= q_threshold` (`report.rs:101`). The row unit is the precursor +(peptidoform + charge), not the stripped sequence; `strip` (`report.rs:24`) removes a +`DECOY_` prefix and bracketed/parenthesized mod blocks for the `stripped_sequence` +column only, and a separate stripped-sequence count is logged. `q_value` is printed at +6 decimals, `score` at 4, `quantity` via `qcell` (1 decimal, empty on NaN, +`report.rs:39`). Proteins: sort by `pg_q_value`, unique non-empty protein groups, +targets with `pg_q_value <= q_threshold` (`report.rs:137`). Returns `(n_precursors, +n_protein_groups)`. + +This is a hybrid identification report: `peptides.tsv` has precursor-shaped rows +but is filtered and labeled with peptide-level q. It is neither a stripped-peptide +table nor a `precursor_q`-controlled precursor table. Report filtering is also +independent of `quant.q_filter`, so a report row can legitimately have a blank +quantity when quant excluded it or marked it nonquantifiable. Use +`peptide_quant.parquet`/`protein_group_quant.parquet` for numerical analysis: +Parquet retains nullable f64 precision and status/bounds, while TSV quantity is a +presentation value rounded to one decimal. + +### audit (`audit.rs:64`, `run`) + +The search space is every library precursor (`candidate_id`, `peptidoform`, `charge`, +`label`, `protein`). Survivor sets are built as `HashSet` of `candidate_id` from +`psms` (extracted), `competed`, and `scored`; scored also yields `q_by_cid` and +optional `pepq_by_cid` (peptide-level q, only present in some scored schemas, +`audit.rs:90`). `load_extract_reasons` (`audit.rs:51`) tries to read a +`.audit.parquet` sidecar to refine the extract-stage bucket, but nothing in the +current chain writes that file (see the gotchas), so the map is empty and every +extract-stage loss buckets to `NO_PEAK_GROUP`. For each candidate the earliest +rejection reason is assigned along the ladder (`audit.rs:136`): +- not in `extracted` -> refined from the sidecar + (`NO_FRAGMENT_TRACES`/`NO_VALID_FRAGMENTS`/`PEAK_NOT_SELECTED`/`RT_PRUNED`/ + `WRONG_ISOLATION_WINDOW`, `audit.rs:139`) or the generic `NO_PEAK_GROUP` when no + sidecar (the only outcome today); +- extracted but not in `competed` -> `OUTCOMPETED_BY_DECOY` (decoy) or + `OUTCOMPETED_BY_TARGET` (target); +- competed but `q > q_threshold` -> `FAILED_PRECURSOR_FDR`; +- passes precursor but fails peptide q -> `FAILED_PEPTIDE_FDR` (peptide q falls back to + the precursor gate when absent, `audit.rs:133`); +- else `REPORTED`. + +The waterfall counts per reason and is logged sorted by descending count +(`audit.rs:220`); `.metrics.json` records the fields listed under +`## Inputs and outputs`. This stage never re-runs compute and never mutates a pipeline +output, so it is safe to run after any search. In the `run` orchestrator the stage is +invoked only when `extract.emit_candidate_audit` is set (`run.rs:405`), with +`q_threshold = 0.01`, `run_id = out_dir`, and no entrapment substring; standalone, +`mumdia audit` takes those as CLI args. + +`RejectionReason` (`rejection.rs:19`) is a 17-variant enum with a stable +SCREAMING_SNAKE_CASE `code()` (`rejection.rs:50`), a `stage_order()` ladder position +(0 earliest, `Reported` = 255, `rejection.rs:76`), an `is_rejection()` predicate +(true for any non-`Reported` reason, `rejection.rs:100`), and `earliest()` to keep the +smaller stage (`rejection.rs:106`). The variants are grouped by pipeline stage in the +source (search space, candidate generation/pruning, extraction, ranking, competition, +FDR/reporting). + +## Key types and functions + +| name | file:line | what it does | +|---|---|---| +| `QuantParams` | quant.rs:21 | inputs/outputs + `&QuantConfig` for the quant stage | +| `trapezoid` | quant.rs:38 | f64 trapezoidal integral of one trace; single point = raw intensity | +| `trapezoid_window` | quant.rs:53 | trapezoid restricted to `[lo,hi]`; empty window = 0 | +| `peak_window` | quant.rs:80 | summed-XIC apex (co-elution rule) + descent-walk window | +| `passes_quant_filter` | quant.rs:177 | accept a scored row: non-decoy (`label != "decoy"`), finite q, q <= threshold | +| `summarize_fragment_areas` | quant.rs:185 | top-N positive fragment sum + quant_status; missing/zero -> null | +| `add_protein_base_quantity` | quant.rs:220 | record one base peptide's max quantity for protein rollup | +| `rollup_protein_bases` | quant.rs:238 | TopNSum/Sum over unique quantifiable base peptides | +| `quant::run` | quant.rs:259 | full quant stage | +| `run_lfq_combine` | quant.rs:657 | build the protein-by-run matrix for `quant-lfq` | +| `size_factors` | quant.rs:753 | per-run normalization factors (MedianRatio/Median/None) | +| `median_sorted` | quant.rs:818 | in-place median helper (empty = 0.0; assumes finite, no NaN) | +| `lfq_profile` | quant_lfq.rs:84 | MaxLFQ least-squares per-sample profile | +| `median` | quant_lfq.rs:12 | private in-place median (duplicate of `median_sorted`, local to `quant_lfq`) | +| `solve_fixed` | quant_lfq.rs:28 | dense Laplacian solve, first var fixed at 0 | +| `maxlfq` / `directlfq` | quant_lfq.rs:191 / 198 | granularity-specific wrappers over `lfq_profile` | +| `AlignParams` | align.rs:22 | seeds + q_train + grid_n | +| `confident_rts` | align.rs:32 | best observed RT per confident target base peptide | +| `align::run` | align.rs:53 | reference LOESS RT map per run | +| `run_mbr` | sidecar.rs:162 | build argv and spawn `mbr_worker.py` | +| `binned_map` | mbr_worker.py:31 | monotone binned-median RT calibration | +| `expected_rt` | mbr_worker.py:116 | cross-run predicted RT for a candidate in a run | +| `pa_write_empty` | mbr_worker.py:289 | placeholder output when there are no transfer candidates | +| `ReportParams` / `report::run` | report.rs:13 / 49 | TSV writer | +| `strip` | report.rs:24 | stripped sequence from a peptidoform | +| `qcell` | report.rs:39 | quantity cell formatting (1 decimal; empty on NaN) | +| `AuditParams` / `audit::run` | audit.rs:28 / 64 | candidate loss ladder | +| `load_extract_reasons` | audit.rs:51 | optional `.audit.parquet` refinement (no producer yet) | +| `RejectionReason` | rejection.rs:19 | earliest-loss category enum (`code`/`stage_order`/`is_rejection`/`earliest`) | +| `resolve_script` | sidecar.rs:20 | locate a worker script relative to CWD or the binary | +| `run_worker` | sidecar.rs:217 | spawn `python