Bactopia v2 Overview

With tremendous effort by @Mxrcon and @abhi18av, the foundation for migrating Bactopia to DSL2 has been laid out. This transition represents the key milestone to push Bactopia to v2! (Super excited about this Davi and Abhinav!)

By switching to DSL2, the door for creating custom Bactopia workflows has been opened. For example, let's say you have some Staphylococcus aureus samples, and you want to run Bactopia and then the Bactopia Tool staph-typer. Instead with DSL2, we can create a sub-workflow (e.g. Staphopia) that will automatically run Bactopia and staph-typer. In other words, we can start creating organism-specific sub-workflows, as well as sub-workflows that only include certain steps such as assembly.

I think this also a good time to start cleaning up some things and adding features that will make long-term maintenance more sustainable.

House Cleaning

These are to help reduce the burden required to maintain Bactopia long-term. These are really about standardizing things in such a way that we can automate things. For example, printing usage across each of the workflows can be configured through config files (e.g. nf-core json schema. There are also a lot of shared functions for checking inputs, creating channels, etc. These duplications are no longer necessary in DSL2.

• Automate Version and Citation tracking

  • Add meta.yml to every module
    • Use nf-core/modules meta.yml a template
  • Integrate this into bactopia citations
  • Integrate this into bactopia versions (now handled by versions.yml)

• Reduce code duplication

  • Schema for printing Usage (using modified nf-core pipelines lib/*)
  • Input file checking (using modified nf-core pipelines lib/*)
  • Merging tables add csvtk/concat module nf-core/modules#785
  • Input validation (e.g. type checking) (using modified nf-core pipelines lib/*)

• Organize DSL2 structure

  • Create structure to follow (using structure loosely based off nf-core dsl2 pipelines)

• Fixes

  • [bug] WARN: Unable to resume cached task -- See log file for details #212
  • [bug] System memory incorrectly parses > 1000GB as 1Gb #240
  • [bug] download_references fails when the reference genome has been removed #245

Additional Features

• Support for Nanopore reads
• Drop support for some uncompressed outputs (e.g. assemblies)
  • Defaults to compressed outputs, --skip_compression disables this feature
• GenBank compatible assembly
  • Currently does not like gnl|
• Outputs from the tutorial https://doi.org/10.6084/m9.figshare.17097156.v1

Implement pytest for testing

I'd like to create a suite of tests that are operated by pytest and pytest-workflows. The nf-core/modules team has a framework that can be extended to Bactopia.

• Setup Test-Data repo - Done! https://github.com/bactopia/bactopia-tests
• Setup walk through for testing - Done! https://github.com/bactopia/bactopia/tree/dsl2/tests
• Add tests to Github Actions -Done! https://github.com/bactopia/bactopia/actions/runs/1256507737
• Create Tests for Bactopia Modules
  • annotate_genome
  • antimicrobial_resistance
  • ariba_analysis
  • assemble_genome
  • assembly_qc
  • blast
  • call_variants
  • count_31mers (merged into minmer_sketch)
  • download_references (merged into call_variants)
  • estimate_genome_size (merged into gather_samples)
  • fastq_status (merged into gather_samples)
  • gather_samples
  • mapping_query
  • minmer_query
  • minmer_sketch
  • qc_reads
  • sequence_type
• Create Tests for Bactopia Tool Modules
  • agrvate
  • bakta
  • ectyper
  • emmtyper
  • eggnog
  • fastani
  • hicap
  • ismapper
  • kleborate
  • lissero
  • mashtree
  • meningotype
  • ngmaster
  • pangenome
  • seqsero2
  • spatyper
  • staph-typer
  • staphopiasccmec
  • tbprofiler

Convert some processes to nf-core/modules

There are a few tools used by Bactopia that are the only tool in the process. Most of these tools are in the Bactopia Tools. I think its best that these tools be transferred to nf-core/modules. Many of these will need to be added to nf-core but they are in need of some bacterial genomic tool love, so its ok!

• Need to be added (25 total)
  • agrvate module for agrvate nf-core/modules#693
  • bakta add bakta module nf-core/modules#1085
  • clonalframe add clonalframeml module nf-core/modules#974
  • ectyper add ectyper module nf-core/modules#948
  • eggnog_mapper add eggnog-mapper module nf-core/modules#1020
  • emmtyper add emmtyper module nf-core/modules#1028
  • fastani module fastani nf-core/modules#695
  • fastq-scan add module for fastq-scan nf-core/modules#935
  • gtdb Add gtdbtk/classifywf module nf-core/modules#765
  • hicap add hicap module nf-core/modules#772
  • ismapper add ismapper module nf-core/modules#773
  • kleborate Add module kleborate nf-core/modules#711
  • lissero add lissero module nf-core/modules#1026
  • mashtree add mashtree module nf-core/modules#767
  • meningotype add meningotype module nf-core/modules#1022
  • ngmaster add ngmaster module nf-core/modules#1024
  • phyloflash Add phyloflash module nf-core/modules#786
  • pirate add pirate module nf-core/modules#777
  • roary add roary module nf-core/modules#776
  • scoary add scoary module nf-core/modules#1034
  • seqsero2 add seqsero2 module nf-core/modules#1016
  • snp-dists module: snp-dists nf-core/modules#697
  • spatyper add spatyper module nf-core/modules#784
  • staphopia-sccmec add staphopia-sccmec module nf-core/modules#702
  • tb-profiler add module for tbprofiler nf-core/modules#947

Curated Datasets

I think one of the best features of Bactopia is the ability to include public datasets. This works great for general datasets, but organism-specific datasets are kind of lost. I think it would be great to start a set of curated datasets that users can add data to.

Here's an example of a curated Staphylococcus aureus Bactopia Dataset. This dataset can easily be imported and allow users to rapidly analyze their samples with a curated dataset specific to their organism.

I think it would also be nice if these curated datasets, included SRA accessions linked to publications. But this exceeds my capabilities and would require extensive community support.

Species specific Workflows

With DSL2, we can create Species Specific workflows by combining the main Bactopia workflow with some Bactopia Tools. The main example, and thus shall act as a proof-of-concept will be Staphopia. Staphopia is essentially Bactopia + the Bactopia Tool staph-typer.

• Create a Staphopia workflow