Lazypipe User Guide

Running Lazypipe v3.0 on a Linux cluster

Table of Content

1. About Lazypipe
2. Running on CSC
3. Installing
   • Setting up directories
   • Cloning repository
   • Installing dependencies
     • Installing with Conda
     • Installing manually
   • Installing Perl modules
   • Installing R libraries
   • Installing reference databases

4. Running Lazypipe

   • Example 1: analyzing sample data
   • Example 1: generated reports
     • abundance tables
     • annotation tables
     • Krona graph
     • Quality control plots
   • Retrieving reads for a contig or taxid

5. command-line options

6. config.yaml options
7. Citing Lazypipe
8. Contact

About Lazypipe

Lazypipe is a bioinformatic pipeline for analyzing virus and bacteria metagenomics from NGS data.

Figure 1. Lazypipe workflow

Lazypipe supports:

• fastq preprocessing
• de novo assembling
• taxonomic binning
• taxonomic profiling
• reporting
  • mapped contigs sorted by taxa
  • virus contigs
  • unmapped contigs
  • contig annotations (tsv and excel)
  • taxon abundancies (tsv and excel)
• quality control plots

Running Lazypipe on CSC

Lazypipe can be quickly assessed using a preinstalled module at the Finnish Center of Scientific Computing.

Installing Lazypipe

Setting up directories

Create root directory $data and subdirectories for storing reference databases, NCBI taxonomy, host genomes and Lazypipe results (change /my/data/path/ according to your preferences):

data=/my/data/path
mkdir -p $data $data/databases $data/taxonomy $data/hostgen $data/results

For convenience add environment variable $data pointing to your root directory. To add the variable locate .bashrc file in your home directory and add this line to the file:

export data=/my/data/path

Cloning the repository

git clone https://plyusnin@bitbucket.org/plyusnin/lazypipe.git
cd lazypipe

Installing dependencies

Installing dependencies with Conda

We recommend installing BLAST under a separate Conda environment labeled blast:

conda create -n blast -c bioconda blast

All other dependencies can be installed under environment labeled Lazypipe:

conda create -n lazypipe -c bioconda -c eclarke bwa csvtk fastp krona megahit mga minimap2 samtools seqkit spades taxonkit trimmomatic numpy scipy requests

Mac users installing to M1/M2 ARM64 architecture: Prior to installing bio-packages configure Conda with conda config --add subdirs osx-64. You may also need to install MGA binary manually (see Table 1).

To activate all installed dependencies type:

conda activate blast
conda activate --stack lazypipe

Set taxonomy database location for KronaGraph:

 rm -rf $CONDA_PREFIX/conda/env/lazypipe/opt/krona/taxonomy
 ln -s $data/taxonomy $CONDA_PREFIX/conda/env/lazypipe/opt/krona/taxonomy

Set env variable $TM to point to trimmomatic directory:

 export TM=$CONDA_PREFIX/share/trimmomatic

Download PANNZER (version 02/2022 or later) and set runsanspanz.py as executable to your path:

wget http://ekhidna2.biocenter.helsinki.fi/sanspanz/SANSPANZ.3.tar.gz
tar -zxvf SANSPANZ.3.tar.gz
echo '#!'$(which python) 1> SANSPANZ.3/runsanspanz.ex.py
cat SANSPANZ.3/runsanspanz.py >> SANSPANZ.3/runsanspanz.ex.py
chmod 755 SANSPANZ.3/runsanspanz.ex.py
ln -sf $(pwd)/SANSPANZ.3/runsanspanz.ex.py ~/bin/runsanspanz.py

Installing dependencies manually

Download and unpack dependencies listed in Table 1. Then copy or link these executables to your ~/bin folder. For example:

wget https://github.com/lh3/minimap2/releases/download/v2.24/minimap2-2.24_x64-linux.tar.bz2
tar -xjvf minimap2-2.24_x64-linux.tar.bz2
cp minimap2-2.24_x64-linux/minimap2 ~/bin/

Table 1: Lazypipe dependencies Tools in square brackets mark binaries that are not required for basic Lazypipe runs. When installed, these will provide additional options/functionalities.

Installing Perl modules

Install modules to local-lib ~/perl5

cpan --local-lib=~/perl5 File::Basename File::Temp Getopt::Long YAML::Tiny
export PERL5LIB=~/perl5/lib/perl5:{$PERL5LIB}

Installing R libraries

Open R console and type

install.packages( c("reshape","openxlsx") );

Installing reference databases

Install NCBI Taxonomy to default location ($data/taxonomy) by running:

perl perl/install_db.pl --db taxonomy

Download and unpack reference databases for 1st and 2nd round annotations. You can choose to install RefSeq/UniRef100 databases (Table 2), NT/UniRef100 databases (Table 3) or Viral databases (Table 4). RefSeq/UniRef100 databases are suited for annotating established taxa with small disk and time overhead. NT/UniRef100 databases have better coverage for novel taxa and may produce more accurate annotations, however the disk/time overhead is also higher. Viral databases are small databases intended for annotating only viral taxa with minimum disk/time overhead.

Use install_db.pl script to install databases from URLs listed in config.yaml. Too install RefSeq/UniRef100 databases to default path ($data/databases) call:

perl perl/install_db.pl --db minimap.refseq.abv -v
perl perl/install_db.pl --db minimap.refseq.vi -v
perl perl/install_db.pl --db blastn.refseq.ab -v
perl perl/install_db.pl --db blastn.refseq.vi -v
perl perl/install_db.pl --db blastp.vi -v

Table 2: RefSeq/UniRef100 databases

Table 3: NT/UniRef100 databases

Table 4: Viral databases

Open config.yaml and check that database paths match the location and version of the installed databases. Edit these line in config.yaml:

ann1.databases:
   minimap.nt:        "$data/databases/nt.abv.2024_01_01.fa"
   minimap.refseq:    "$data/databases/refseq.abv.release221.fa"
   ..

ann2.databases:
   blastn.ab.nt:      "$data/databases/blastn.nt.ab.2024_01_01"
   blastn.vi.nt:      "$data/databases/blastn.nt.vi.2024_01_01"
   ..

If you wish to annotate bacteriophages, specify minimap.ph, blastn.ph and blastp.ph in your config.yaml. Use blastn/blastp virus databases or your custom bacteriophage databases:

ann1.databases:
   minimap.ph: $data/databases/minimap.GPD.ph.fasta
   blastn.ph:  $data/databases/blastn.nt.vi.2024_01_01
   blastp.ph:  $data/databases/blastp.uniref100.vi.2024_01_24
ann2.databases:
   blastn.ph:  $data/databases/blastn.nt.vi.2024_01_24
   blastp.ph:  $data/databases/blastp.uniref100.vi.2024_01_24

Naming convention for reference databases

Reference sequence databases are defined in config.yaml as key-value pairs under ann1.databases and ann2.databases. Here, each key is a string referring to the SearchTool and TargetTaxa, and each value is a path to the applied database. For the 1st round, annotation keys are named SearchTool[.dbid] and for the 2nd round SearchTool.TargetTaxa[.dbid]. In both rounds SearchTool can be sans, minimap, blastn or blastp. TargetTaxa can be abv (ie Archaea, Bacteria and Viruses), ab (ie Archaea and Bacteria), vi (Viruses), ph (Bacteriophages) or un (unmapped). You can use an optional dbid string to differentiate between similar databases. For any annotation step you can use any database, default or custom. For BLASTN/BLASTP use blast indices. For minimap2 you can use .fasta* or .mmi files; note that these must have an accomponing .acc2taxid tsv-map (see default minimap2 databases for an example).

Running Lazypipe

Example 1

In this example we will use a sample PE library that is included with the repository (data/M15small_R*.fastq).

Preprocess reads with fastp:

perl lazypipe.pl -1 data/samples/M15small_R1.fastq --pipe pre -t 8 -v

Download Neovison vison genome and use it to filter host reads. Note that running host filtering with a newly downloaded genome will take some time to index the genome:

mkdir -p $data/hostgen
wget https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/900/108/605/GCA_900108605.1_NNQGG.v01/GCA_900108605.1_NNQGG.v01_genomic.fna.gz -P $data/hostgen/
perl lazypipe.pl -1 data/samples/M15small_R1.fastq --pipe flt --hostgen $data/hostgen/GCA_900108605.1_NNQGG.v01_genomic.fna.gz -t 8 -v

Run assembling with Megahit and realign reads to assembly

perl lazypipe.pl -1 data/samples/M15small_R1.fastq -p ass,rea --ass megahit -t 8 -v

Run 1st round annotation with Minimap2 against your local minimap.refseq database:

perl lazypipe.pl -1 data/samples/M15small_R1.fastq -p ann1 --ann1 minimap.refseq -t 8 -v

Run 1st round annotation with SANSparallel against UniProt TrEMBL. Note that SANSparallel runs on a remote server and requires internet connection. Append results to Minimap2 annotations from the previous step:

perl lazypipe.pl -1 data/samples/M15small_R1.fastq -p ann1 --ann1 sans --append -t 8 -v

Now run a more complex 1st round annotation. Start by mapping contigs with Minimap2, then map unmapped contigs with SANSparallel then map unmapped contigs with BLASTN against blastn.vi database. Note that without --append flag this will overwrite existing 1st round annotations:

perl lazypipe.pl -1 data/samples/M15small_R1.fastq -p ann1 --ann1 minimap.refseq,sans,blastn.vi -t 8 -v

Run 2nd round annotation. In the second round you can target archaeal+bacterial (=ab), bacteriophage (=ph), viral (=vi) and unmapped (=un) contigs, based on labeling from the 1st round. Local databases for the 2nd round annotations are defined in ann2.databases section of the config.yaml. For example, to map viral contigs with BLASTN and BLASTP against local viral databases type:

perl lazypipe.pl -1 data/samples/M15small_R1.fastq --pipe ann2 --ann2 blastn.vi.refseq,blastp.vi -t 8 -v

Run 2nd round annotation for bacteria with BLASTN. Append results to BLASTN and BLASTP annotations from the previous step:

perl lazypipe.pl -1 data/samples/M15small_R1.fastq --pipe ann2 --ann2 blastn.ab.refseq --append -t 8 -v

You can also combine these runs in any order. For example:

perl lazypipe.pl -1 data/samples/M15small_R1.fastq --pipe ann2 --ann2 blastn.ab.refseq,blastn.vi.refseq,blastp.vi -t 8 -v

The most common combinations of 1st and 2nd round annotations can be saved to config.yaml in the ann.strategies section. Each annotation strategy is saved as a key-value pair. There are several annotation strategies predifined:

• abv.fast -- run only the 1st round with Minimap2 against RefSeq.abv
• abv.nt -- 1st round: Minimap2 against NT.abv, 2nd round: BLASTN viral reads against NT.vi and archaeal+bacterial reads against NT.ab
• abv.refseq -- 1st round: Minimap2 against RefSeq.abv, 2nd round: BLASTN viral reads against RefSeq.vi and archaeal+bacterial reads against RefSeq.ab
• abv.extend -- 1st round: Minimap2 against NT.abv + SANSparallel unmapped reads against TrEMBL, 2nd round: BLASTN viral reads against NT.vi and archaeal+bacterial reads against NT.ab, additionally BLASTP viral reads against UniRef100.vi and archaeal+bacterial reads against UniRef100.ab
• vi.nt -- 1st round: Minimap2 against NT.vi, 2nd round: BLASTN viral reads against NT.vi
• vi.refseq -- 1st round: Minimap2 against RefSeq.vi, 2nd round: BLASTN viral reads against RefSeq.vi

Generate reports based on created annotations:

perl lazypipe.pl -1 data/samples/M15small_R1.fastq --pipe rep -t 8 -v

Generate assembly stats, pack for sharing and remove temporary files:

perl lazypipe.pl -1 data/samples/M15small_R1.fastq -p stats,pack,clean -t 8 -v

For convenience, routine analysis steps (pre,flt,ass,rea,ann1,ann2,rep,sta,pack,clean) can be called with main tag. To run main analysis with abv.refseq annotation strategy type:

perl lazypipe.pl -1 data/samples/M15small_R1.fastq -p main --anns abv.refseq -t 8 -v

Example 1: generated reports

Results are output to $res/$sample. Default value for $res is set in config.yaml and default value for $sample is created from the name of the input reads. These can be changed during runtime with --res mydir --sample mysample.

In example 1 results were output to $data/results/M15small.

Assembled contigs and predicted ORFs

Table 5: Lazypipe results: contigs and ORFs.

Abundance tables

Figure 2. abund_table.xlsx

Spreadsheets with taxon abundancies are printed to abund_table.xlsx. Abundancies are displayed in separate tables for viruses (excluding bacteriophages), bacteria, bacteriophages and eukaryots. For each domain abundancies are displayed at three taxonomic levels: species, genus and family.

For raw abundance data see abund_table.tsv.

Table 6: Columns in abund_table.xlsx

Annotation tables

Figure 3. annot_table.xslx

Spreadsheets with contig annotations are printed to contig_annot.xslx. Spreadsheets are displayed separately for viruses (excluding bacteriophages), bacteria, bacteriophages and eukaryots.

For raw annotation data see contigs_annot.tsv.

Table 7: Columns in contigs_annot.xslx

Quality control plots

Figure 5. Quality control plogs

Quality Control (QC) plots include length histograms for reads and contigs, and survival plots. The survival plots track retained reads after each pipeline step.

Table 8: Quality Control plots

Retrieving reads for a contig or taxid

Start by unzipping your source fasta:

gunzip -k results/M15small/read*.trim.fq.gz

To retrieve all reads mapped to contig k99.17 type:

bin/retrieve_reads -r results/M15small -v -c k99.17

To retrieve all reads mapped to Circovirus mink use the following command. Note that the exact species name may change with taxonomy updates.

bin/retrieve_reads -r results/M15small -v -s "Circovirus mink"

To retrieve all reads mapped to staxid 1239574 (Mamastrovirus) type:

bin/retrieve_reads -r results/M15small -v -t 1239574

Command line options

Table 9: Lazypipe command line options.

Default options and additional settings are defined in config.yaml file. Note that command line options take precedence over options in config.yaml file.

Additional options in config.yaml:

Table 10: Default options in in config.yaml

Citing Lazypipe

1. Plyusnin Ilya, Olli Vapalahti, Tarja Sironen, Ravi Kant, and Teemu Smura. “Enhanced Viral Metagenomics with Lazypipe 2.” Viruses 15, no. 2 (February 4, 2023): 431. https://doi.org/10.3390/v15020431

2. Ilya Plyusnin, Ravi Kant, Anne J. Jaaskelainen, Tarja Sironen, Liisa Holm, Olli Vapalahti, Teemu Smura. (2020) Novel NGS Pipeline for Virus Discovery from a Wide Spectrum of Hosts and Sample Types. Virus Evolution, veaa091, https://doi.org/10.1093/ve/veaa091

Contact

Project website: https://www.helsinki.fi/en/projects/lazypipe

Contact email: grp-lazypipe@helsinki.fi