Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
172 changes: 70 additions & 102 deletions README.md

Large diffs are not rendered by default.

114 changes: 114 additions & 0 deletions assets/seqsubmit_schema.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
76 changes: 10 additions & 66 deletions docs/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,44 +184,12 @@ If you update images or graphics, follow the nf-core [style guidelines](https://

To ensure consistency and maintainability of the nf-core/seqsubmit pipeline, contributors should follow these guidelines when implementing new features or modifying existing functionality.

1. Understand Workflow Structure

The pipeline consists of independent workflows, each corresponding to a certain data type:

- GENOMESUBMIT (mags, bins)
- ASSEMBLYSUBMIT (metagenomic_assemblies)
- READSUBMIT (raw reads)

When adding new functionality (e.g. a new submission mode such as `isolate_genomes`), ensure that:

- It integrates cleanly into the appropriate workflow or introduces a well-structured new one.
- Existing workflows remain unaffected unless changes are explicitly required.
- Shared logic is abstracted into reusable modules or subworkflows where possible.


2. Follow Samplesheet and Schema-Driven Design

All inputs in seqsubmit are defined via strict samplesheet schemas.

- Any new feature must:
- Extend or introduce a corresponding schema (assets/schema*input*\*.json)
- Clearly define required and optional fields
- Maintain:
- Column order consistency (critical for pipeline execution)
- Backward compatibility where possible
- Avoid duplicating metadata logic—reuse existing parsing and validation patterns.


3. ENA Submission Requirements
### ENA Submission Requirements

Submissions to [ENA](https://www.ebi.ac.uk/ena/browser/home) are governed by a well-defined data model and submission process. These include relationships between entities (e.g. studies, samples, experiments, runs, analyses), required submission steps, and accepted file types and formats.

Contributors should refer directly to the [official ENA documentation](https://ena-docs.readthedocs.io/en/latest/submit/general-guide.html) to understand latest submission requirements and ensure that pipeline behaviour remains aligned with ENA standards.

Follow ENA-defined submission workflows and required steps for each data type

- Ensure that:
- New pipeline workflows follow ENA-defined submission workflows and required steps for each data type
- Metadata fields align with ENA requirements
Expand All @@ -232,23 +200,23 @@ Contributors should refer directly to the [official ENA documentation](https://e


4. Integration with Webin-CLI
### Integration with Webin-CLI

All submissions are performed via [Webin-CLI](https://github.com/enasequence/webin-cli). Contributors should rely on official Webin-CLI documentation to understand expected inputs, parameters, and modes of operation. Make sure to always use the latest version of the tool.

- Contributions affecting submission logic must:
- Respect --webincli_mode (validate vs submit)
- Support --test_upload for safe testing against ENA test server
- Respect `--webincli_mode` (validate vs submit)
- Support `--test_upload` for safe testing against ENA test server
- Never introduce changes that:
- Risk accidental submission to production without explicit user intent
- Ensure compatibility with:
- Credential handling via Nextflow secrets (ENA_WEBIN, ENA_WEBIN_PASSWORD)


5. Testing Requirements
### Testing Requirements

All new functionality must be covered by tests.
All new functionality must be covered by tests. Running nf-tests that perform ENA submission locally requires ENA webin credentials. Contributors may ask for testing credentials on the nf-core Slack [#seqsubmit](https://nfcore.slack.com/channels/seqsubmit) channel.

At minimum, include:

Expand All @@ -268,36 +236,12 @@ Where applicable:


6. Performance and Resource Awareness

Some data processing steps may be resource-intensive (e.g. database preparation for CAT_pack). When introducing new workflows or features avoid unnecessary recomputation. Add option to reuse cached results or provide prebuilt databases when possible.


7. Documentation and Usability

Every contribution should improve usability:

- Update:
- README.md
- docs/usage.md
- Schema descriptions
- Provide:
- Clear examples (e.g. samplesheets)
- Meaningful parameter descriptions

The goal is to ensure that users can successfully submit data without needing deep knowledge of ENA submission requirements.


8. Design Principles
### Performance and Resource Awareness

When contributing, prioritise:
Some data processing steps may be resource-intensive (e.g. database preparation for CAT_pack).

- Reproducibility (consistent results across environments)
- Automation (minimise manual user intervention)
- Clarity (explicit inputs and outputs)
- Extensibility (easy to add new submission types)
- When introducing new workflows or features avoid unnecessary recomputation.
- Add option to reuse cached results or provide prebuilt databases when possible.


Expand Down
50 changes: 20 additions & 30 deletions docs/methods.md
Original file line number Diff line number Diff line change
@@ -1,38 +1,33 @@
# nf-core/seqsubmit: Methods

## Overview
This page provides a detailed overview of the methods and procedures that are executed throughout the nf-core/seqsubmit pipeline.

`nf-core/seqsubmit` currently contains two workflow implementations:

- `GENOMESUBMIT` for `--mode mags` and `--mode bins`
- `ASSEMBLYSUBMIT` for `--mode metagenomic_assemblies`

This page documents the methods that are currently implemented in the pipeline.

## `GENOMESUBMIT` methods
## `GENOMESUBMIT`

### Overview

The `GENOMESUBMIT` workflow:

1. Reads the samplesheet and associated genome FASTA files.
2. Validates genome FASTA files.
3. Reuses provided or calculates missing values for RNA genes presence, coverage, taxonomy, and genome quality metrics.
3. Reuses provided or calculates missing values for tRNA and rRNA genes presence, coverage, taxonomy, and genome quality metrics.
4. Collects genome metadata into the tabular format required by `genome_uploader`.
5. Generates submission manifests for ENA.
6. Performs submission to ENA.

### Genome FASTA validation

Genome FASTA files are validated with the `FASTAVALIDATOR` module before downstream processing. Each file is checked for FASTA format validity and contig count. A genome must contain at least two contigs to pass validation, which is an ENA requirement for contig-level submissions.
Genome FASTA files are validated with the `fa-lint` before downstream processing. Each file is checked for FASTA format validity and contig count. A genome must contain at least two contigs to pass validation, which is an ENA requirement for contig-level submissions.

Only FASTA files that pass validation are retained for downstream processing and submission.

### RNA presence detection
### tRNA and rRNA genes detection

The workflow only runs internal RNA detection for entries where the `RNA_presence` column is empty. If a value is already supplied in the samplesheet, that value is passed through unchanged.
The workflow only runs tRNA and rRNA genes detection for entries where the `RNA_presence` column is empty. If a value is already supplied in the samplesheet, that value is passed through unchanged.

RNA detection is implemented through the `RNA_DETECTION` subworkflow and combines:
The resulting `RNA_presence` value is a decisive factor affecting MISAG/MIMAG assembly quality classification of a genome.

tRNA and rRNA genes detection is implemented through the `RNA_DETECTION` subworkflow and combines:

- `barrnap` for rRNA prediction
- `tRNAscan-SE` for tRNA prediction
Expand All @@ -42,11 +37,7 @@ RNA detection is implemented through the `RNA_DETECTION` subworkflow and combine

`barrnap` is run in bacterial mode (`"bac"`).

The custom parser then scans the GFF output and keeps only the following ribosomal RNA subunits:

- `16S_rRNA`
- `23S_rRNA`
- `5S_rRNA`
The custom parser then scans the GFF output and keeps only the following rRNA subunits: `16S_rRNA`, `23S_rRNA`, `5S_rRNA`.

For each detected feature, the recovered length is calculated as:

Expand All @@ -66,7 +57,7 @@ $$

If multiple hits are found for the same subunit, the workflow keeps the best recovered percentage for that subunit.

A subunit is considered present when its best recovered percentage is greater than or equal to `--rrna_limit`. The current default is `80`.
A subunit is considered present when its best recovered percentage is greater than or equal to `--rrna_limit`. The current default is `80`. This threshold is arbitrary, as no strict consensus length is defined by ENA or MISAG/MIMAG for considering an rRNA gene present.

#### tRNA detection

Expand All @@ -79,16 +70,16 @@ The parser sums the counts assigned to the 20 standard amino-acid isotypes:
- `Leu`, `Lys`, `Met`, `Phe`, `Pro`
- `Ser`, `Thr`, `Trp`, `Tyr`, `Val`

The total number of predicted tRNAs is compared against `params.trna_limit`. The current default is `18`.
The total number of predicted tRNAs is compared against `params.trna_limit`. The current default is `18`. This value is defined by the MISAG/MIMAG standard and must not be modified.

#### Final RNA presence decision
#### Final `RNA_presence` decision

The final decision stored in `RNA_presence` is:

- `Yes` when at least 18 tRNAs are detected and all three required rRNA subunits pass the recovery threshold
- `No` otherwise

The result is written as a two-column TSV file containing the genome identifier and the final `Yes`/`No` decision, and this value is then merged back into the submission metadata.
The final `Yes`/`No` decision is then merged back into the submission metadata.

### Genome coverage calculation

Expand All @@ -98,7 +89,7 @@ Genome coverage values from `coverm genome` output TSV are parsed and merged int

### Taxonomy assignment

If `NCBI_lineage` is already present in the input samlesheet, the value is retained. If it is missing, the workflow runs taxonomy classification using the `CAT_pack` tool.
If `NCBI_lineage` is already present in the input samplesheet, the value is retained. If it is missing, the workflow runs taxonomy classification using the `CAT_pack` tool.

Before classification, input FASTA files are normalized to a `.fasta` suffix by `RENAME_FASTA_FOR_CATPACK`. Classification is then run in bin mode (`CAT_pack bins`, followed by `CAT_pack add_names`).

Expand All @@ -112,22 +103,21 @@ The workflow checks three samplesheet fields: `completeness`, `contamination`, a

For records that run `CheckM2`, completeness and contamination are extracted from the generated quality report (`quality_report.tsv`) and used `CheckM2` version is recorded as `stats_generation_software`.

## `ASSEMBLYSUBMIT` methods
## `ASSEMBLYSUBMIT`

### Overview

The `ASSEMBLYSUBMIT` workflow:

1. Reads the samplesheet and associated assembly FASTA files.
2. Validates assembly FASTA files.
3. Reuses `coverage` values supplied in the samplesheet when they are already present.
4. Calculates `coverage` internally for entries where this field is missing and reads are available.
5. Builds the metadata table used for manifest generation.
6. Generates assembly manifests and submits the assemblies to ENA.
3. Reuses provided or calculates missing `coverage` values.
4. Builds the metadata table used for manifest generation.
5. Generates assembly manifests and submits the assemblies to ENA.

### Assembly FASTA validation

Assembly FASTA files are validated with `FASTAVALIDATOR` before downstream processing. Each file is checked for FASTA format validity and contig count. A genome must contain at least two contigs to pass validation, which is an ENA requirement for contig-level submissions.
Assembly FASTA files are validated with `fa-lint` before downstream processing. Each file is checked for FASTA format validity and contig count. A genome must contain at least two contigs to pass validation, which is an ENA requirement for contig-level submissions.

Only assemblies with successful validation are forwarded to coverage estimation, metadata/manifest generation and submission.

Expand Down
Loading
Loading