Feat/sensitivity improvements - #45
Merged
Merged
Conversation
… docs)
- scripts/: keep only the 10 engine-invoked sidecars + DIA-NN recipe helpers;
remove ~30 one-off diagnostic/analysis/probe scripts, the exploratory notebook,
and feature_registry.yaml (consumed only by the removed diagnostics)
- config: remove dead/unwired surface (threads; extract.{k_select,
max_fragment_charge, scan_scale, scan_window_mode} + ScanWindowMode;
digest.decoy.{ratio, source} + DecoySource; search_seed.precursor_tol_ppm;
rt_im_train.tolerance_regime + ToleranceRegime; FeatureSet::Custom;
MatcherKind::Naive; CompetitionMode::from_token). Kept DiannShift / None guards
and the MBR config surface as documented hooks.
- docs (local, gitignored): consolidated sensitivity_plan/ into rewritten
plan.md + CLAUDE.md; also removed ~60GB of gitignored run outputs/notebooks/
scratch from the working tree (preserved the DIA-NN library under lib/).
cargo test --workspace green (105 tests).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Deep code + workflow reference for handoff: 14 subsystem docs + an index, grounded in the current post-cleanup code with file:line citations. Covers the end-to-end dataflow, config + data model, IO layer, every pipeline stage (convert -> ... -> rescore), quant/align/mbr/report/audit, the 10 Python sidecars, and build/test/deploy/gotchas. Sits between CLAUDE.md (orientation) and plan.md (spec + findings). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ounts) Adversarial audit of each doc against its source (one agent per doc) + a coverage audit. Closed the gaps found: - Filled undocumented items in every doc: all CLI subcommands + flags, every config field / strategy-enum variant semantics, error types, core data types, the candidate-audit rejection ladder, output-column schemas, MBR wiring gaps, and per-family feature names. +1,189 lines (5,775 -> 6,964). - Closed the single orphan source file: mumdia/src/lib.rs (crate bin+lib split + module map) now documented in doc 01. - Corrected the Extended feature count 383 -> 381 (per active_features test, features.rs:1436-1443) in docs 01 and 02; other docs/CLAUDE.md/plan.md already said 381. Fixed a stray em-dash; verified zero em-dashes remain. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tainment) Gap analysis (4 lenses) then closure: - New reference docs: 15_data_dictionary (every artifact's columns from source), 16_glossary, 17_troubleshooting (symptom->fix), 18_findings_and_decisions (self-contained findings + interstage/determinism/sidecar contracts, no plan.md dependency), 19_getting_started (env setup, msconvert, lib/ provenance, worked example with expected counts) + a host-runnable config.local-diann-lib.json. - Accuracy fixes: config_hash/schema ids are provenance-only (never read back or used to invalidate); run has no caching/resume; emit_candidate_audit gates the audit stage (candidate_audit.parquet), not a <psms>.audit.parquet; sidecar fallback only for rescore (strict-gated), predict-frag/finetune/mbr hard-bail; apex_top_fragments=0 -> default 3; isolation_windows column is window_id; benchmark reconciled (native ~1,213 / mokapot ~9.3-9.5k / nn_torch ~10.3k); README quickstart --out-dir; Extended feature count 381 (not 383). - Self-containment: repointed plan.md / "finding N" citations to docs/18. docs/ now 8,903 lines; zero em-dashes; no dangling plan.md sole-authority refs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…inalize Finalizes a large uncommitted working tree left by a prior session: a correctness/hardening pass across the engine plus a docs overhaul. Verified to the CLAUDE.md validation gate (cargo fmt --check clean, cargo clippy --workspace --all-targets -D warnings clean, cargo test --workspace 126 passed, python compileall scripts, JSON configs parse). Engine (from the diff; tests added alongside): collision-checked native decoys (collision_safe_decoy, drops impossible low-complexity pairs); calibrated observed-only RT/mass error (calibrated_rt_error); peptidoform mod enumeration rules with fixed/variable overlap + unknown-mod rejection (validated_rules, expand_site_choices); quant quantifiability + base-peptide protein rollup + missing-vs-zero fragment handling (passes_quant_filter, summarize_fragment_areas, rollup_protein_bases); unique-evidence competition via contested fraction; config contract tests (precursor_q filter, rescore-contract rejection). Docs: rewritten CLAUDE.md policy (three objectives, validation discipline; local, gitignored) and new docs/20_sensitivity_and_quantification_playbook.md; consequent edits across docs/01-19 + README + configs. Gate-green finalize by this session: 9 clippy lints (needless_range_loop, identical if-blocks, too_many_arguments, large_enum_variant, neg_cmp_op_on_partial_ord) fixed behavior-preservingly + cargo fmt. NOTE: the gate is not scientific validation; the sensitivity/quant behavior changes still need the docs/20 benchmark + entrapment gates before being trusted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Re-verify and correct the developer guide (docs/01-20) and README against the current source after the e7d7fa5 correctness/hardening pass. Updates drifted file:line citations, new/changed functions (collision_safe_decoy, calibrated_rt_error, validated peptidoform mod rules, quant quantifiability and base-peptide protein rollup, unique-evidence competition), config fields and defaults (min_frag_corr=0.2, gate_mode, q_filter, compete modes), and artifact schema versions. Verified: Extended feature count is 381 everywhere, 0 em-dashes, docs/README indexes 01-20, no dangling plan.md sole-authority citations. Docs only; no code, script, or config changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add DigestConfig.n_term_met_excision (default on): when a protein begins with the initiator methionine, also emit the Met-removed form of its N-terminal peptides, matching DIA-NN's --met-excision. Without it the search database structurally cannot contain these (biologically dominant) peptides. Add scripts/augment_library.py: augment an imported spectral library with the tryptic FASTA peptides it is missing (a completeness fix). Reuses the engine's own digest -> peptidoforms -> predict-frag stages so peptidoform strings are byte identical to what the search consumes, offsets ids disjoint from the imported set, and hands off to make_shift_decoys.py for paired decoys. The per-run DeepLC fine-tune re-predicts the whole library's iRT, so imported and augmented entries share one RT axis without explicit reconciliation. On the LFQ_Orbitrap_AIF_Ecoli_01 benchmark this closed the not-in-database gap (209 missing peptides -> 0) and lifted confident IDs at a valid 0.98% empirical decoy fraction; every previously-missing peptide was an N-terminal Met-excision peptide. Tests: met_excision_emits_both_n_term_forms, met_excision_only_at_protein_n_terminus. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.