Pre-commit hooks to enforce snakemake formatting

[dgruano]

What & why

Add development environment configuration, include pre-commit and snakefmt to ensure proper formatting and document all this in the CONTRIBUTING.md docs.

According to the Snakemake workflow catalog, formatting is faling.

[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/common.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/validate_input.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/Snakefile":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/prepare_input_vhd.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/explore_genes.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/generate_qupath_he_ate.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/spaceranger_count_vhd.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/prepare_input_ate.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/spatial_niches.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/annotate_cells.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/pseudobulk_aggregate.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/prepare_input_x5k.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/ingest_ref.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/subcluster.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/qc_sweep.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/generate_qupath_ate.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/neighbourhood_analysis.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/prepare_input_mer.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/leiden_analysis.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/generate_qupath_vhd.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/sample_report.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/generate_qupath_mer.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/preprocess_umap.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/convert_zarr_x5k.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/convert_zarr_ate.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/pseudobulk_de.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/generate_qupath_x5k.smk":  Formatted content is different from original
[DEBUG] 
[DEBUG] In file "/tmp/tmpz6rgqcu6/cbib-SpaceBlocks-196b97b/workflow/rules/integrate_samples.smk":  Formatted content is different from original
[INFO] 28 file(s) would be changed 😬
[INFO] 1 file(s) would be left unchanged 🎉

snakefmt version: 0.11.5

pre-commit is installed with conda. Hooks are configured in .pre-commit-config.yaml and installed with pre-commit install`. Hooks include snakefmt and fixes for other common formatting errors.

pre-commit hooks are ran automatically on all committed files before a commit, which fails if any hook needs to modify a file. In that case, git add -u to include all changes made by pre-commit. The next git commit will have proper formatting.

For all files that were misformatted, I ran the hooks with pre-commit run --all-files. I included all rule files in one commit and the rest in another commit. @Sergio-ote , feel free to cherry-pick the configuration and documentation commits and run the hooks yourself to avoid a bump in the modified lines by me seen by git.

Also @Sergio-ote , can you share how you configured the environment to build the docs? This is the only check missing on my end.

Cheers!

Validation

• Workflow lint passes against .test/
• Decoupled .test/ dry-run builds the CoreBlock DAG
• mkdocs build --strict passes, if documentation was changed

[dgruano]

Some notes after discussing with @Sergio-ote:

• Docstrings from snakemake rules were losing indentation after formatting. This was due to a different formatting standard in snakefmt v1.1.x . The fix was to configure hooks to use the latest version of snakefmt (v2.0.3).
• Formatting hooks were modifying configuration or data files that come from upstream tools (e.g. sample tables, GEOJSON etc). I solved this by excluding several file patterns in the pre-commit config:
  • demos/ directory
  • .tsv and .csv files in config/ directory
  • .svg files (like the rulegraph schematic)
  • mkdocs.yml config file, due to incompatibility of yaml formatter with some tags needed for the docs config

These formatting changes should balance keeping the commit diffs clean and avoid formatting data files. Virtually, formatting changes will be reduced to:

• Snakemake formatting of Snakefile and .smk files to comply with Snakemake formatting standards. This is essential for the workflow catalog. This generally means:
  • Reorganising blocks within rules.
  • Expanding brackets into multiple lines.
• General formatting rules for .md (docs and READMEs) and .yaml files (config and envs):
  • Trailing spaces
  • YAML formatting
  • End-of-file

Let me know what you think!

[dgruano]

As a last change, I included the packages needed to build the documentation in the dev.env environment configuration. Docs build cleanly under a new environment.

PR is ready for review.