Merged single-cell workflow and notebooks addition - #96
Closed
KevinMLanderos wants to merge 5 commits into
Closed
Conversation
Purely additive integration — bulk pipeline unchanged (no files removed). New: --mode dispatcher (bulk default | singlecell); single-cell modality with two routes (full-SC with GEX Seurat, and VDJ-only). Single-cell data is pseudobulked and run through the shared bulk engine (ANNOTATE / SAMPLE / PATIENT / COMPARE incl. GIANA / GLIPH2 / TCRdist3 / OLGA), then CoNGA / consensus / repertoire / master summary. Additive hooks in shared files (no bulk behaviour removed): - ANNOTATE_FROM_CONCAT entrypoint; PATIENT cluster emits + --patient_col - SAMPLE tcrdist emits (superset of the sample_csv emit); SAMPLE_PLOT gated to bulk - GLIPH2 patient-prefixed cluster_member_details emit; tcrdist3 empty-nucseq handling - Per-process container routing (tcrtoolkit + scratch-tcr) via --sc_container - SC params added to nextflow.config and nextflow_schema.json (singlecell_options) Validated end-to-end on real data for both single-cell routes (on the prior base). Bulk cellranger/pseudobulk path, channel-based reports, and convert subworkflow are kept exactly as in main.
single-cell test coverage Merges the all_notebooks branch (report notebook pipeline wiring) into v2, incorporating PR #93 review feedback from Dima/Copilot, replaces the samplesheet's timepoint_order column with an optional pipeline parameter, and establishes real-execution test coverage for --mode singlecell - finding and fixing 8 real bugs in the process, all confirmed via actual container execution. Pipeline wiring (from all_notebooks) - Wire 3 more report notebooks into the pipeline: discovery brief, patient details part 1/2, on top of the existing QC report. - Add named emits for sample/patient/compare outputs needed by the new reports (v/j family, tcrdist/olga/vdjdb, convergence, tcrpheno, shared_cdr3, giana/gliph2). - Stage sibling {{< include >}}'d .qmd files alongside their parent notebook; resolve template_pheno.qmd / template_patient_clustering.qmd generic-include shims to the right notebook at render time, since Quarto includes have no native runtime conditional. - Fix legacy-vs-AIRR column naming mismatches, a convergence-file merge silently returning zero rows, and VDJdb/phenotype join-key bugs, all found by running the new notebooks end-to-end against real output. Merge conflicts with v2's single-cell modality (gliph2.nf, nextflow.config, nextflow_schema.json, subworkflows/local/{patient,sample}.nf) resolved by keeping both branches' additive changes, and rewriting patient.nf's gliph2_cluster_details emit as a direct ternary on GLIPH2_TURBOGLIPH.out per this Nextflow version's strict-syntax requirement. PR #93 review feedback (Copilot + Dima) - Fix a dead subject_col != subject_col conditional in template_discovery_brief.qmd. - Replace an outdated GLIPH2 example in RENDER_NOTEBOOK's staged_layout comment with the template_pheno.qmd case that's actually used. - Strengthen render_notebook.nf.test to assert staged symlinks actually exist, not just that HTML rendered. - Document the stage_cmds join separator, the ch_discovery_report .combine() safety argument, and trim an overlong comment. Timepoint ordering (team decision) - Remove the redundant timepoint_order column from the samplesheet. - Add --timepoint_order: an optional ordered comma-separated list (e.g. Base,Week4,EOT); unset defaults to numeric rank if every value parses as a number, otherwise alphabetical. - Compute timepoint_order_col at notebook-render time in every notebook that loads its own copy of the samplesheet; replace a dead fallback in template_overlap.qmd with a reuse of the rank already carried on clonotypes_df. Single-cell mode: Built synthetic-but-real fixtures (a real Seurat object with computed PCA/UMAP, real Cell Ranger-schema VDJ contigs) and nf-test coverage for VDJ_QC, TCELL_INTEGRATION, CLUSTER_TO_SC, and CONSENSUS_CLUSTERING - all now passing for real through the actual pipeline containers, plus a pipeline-level VDJ-only test in tests/main.nf.test (62 processes, full route from VDJ_QC through MASTER_SUMMARY).
…ts/nextflow.config, since GitHub Actions runners only have 4 CPUs while conf/base.config requests 8, causing every SC test (VDJ_QC, CONGA, CONSENSUS_CLUSTERING, TCELL_INTEGRATION, CLUSTER_TO_SC) to fail immediately with Process requirement exceeds available CPUs. This only overrides the CI test config, not production defaults.
Unit Test Results17 tests 17 ✅ 7m 31s ⏱️ Results for commit ab7da22. ♻️ This comment has been updated with latest results. |
… fixture CoNGA's own find_gex_cluster_degs -> dotplot call crashes (matplotlib "left cannot be >= right") when its DE-gene filtering finds zero significant genes, which this small 180-cell synthetic fixture always triggers - a bug in the external CoNGA library, not TCRtoolkit. Real execution got through the actual TCRtoolkit<->CoNGA integration (clonotype reduction, graph-vs-graph clustering, correlation tables) before hitting this, but it was undetectable in CI until now because the process previously failed earlier on a CPU resource limit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Collaborator
|
Close, GH action tests now trigger on push to v2 branch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.