LAZYPIPE User Guide

Table of Content

1. About Lazypipe
2. Running on CSC
3. Installing
   • Setting up directories
   • Cloning repository
   • Installing dependencies
     • Installing with Conda
     • Installing manually
   • Installing Perl modules
   • Installing R libraries
   • Installing reference databases
4. Running Lazypipe with lazypipe.pl
   • command-line options
   • config.yaml options
   • Example 1: analyzing sample data
   • Example 1: generated reports
     • abundance tables
     • annotation tables
     • Krona graph
     • Quality control plots
5. Running Lazypipe with Snakemake
   • Example 2: analyzing sample data
6. Retrieving reads for a contig or taxid
7. Citing Lazypipe
8. Contact

About Lazypipe

Lazypipe is a bioinformatic pipeline for analyzing virus and bacteria metagenomics from NGS data.

Figure 1. Lazypipe workflow

Lazypipe supports:

• fastq preprocessing
• de novo assembling
• taxonomic binning
• taxonomic profiling
• reporting
  • mapped contigs sorted by taxa
  • virus contigs
  • unmapped contigs
  • contig annotations (tsv and excel)
  • taxon abundancies (tsv, excel and KronaGraph)
• quality control plots

Running Lazypipe on CSC

Lazypipe can be quickly assessed using a preinstalled module at the Finnish Center of Scientific Computing.

Installing Lazypipe

Setting up directories

Create root directory for storing reference and taxonomy databases. Change /my/data/path/ according to your preferences:

mkdir /my/data/path/taxonomy

For convenience add environment variable $data referring to your /my/data/path. To add the variable locate the .bashrc file in your home directory and add this line to the file:

export data=/my/data/path

Load your variables (will autoload on the next login):

source ~/.bashrc

Cloning the repository

git clone https://plyusnin@bitbucket.org/plyusnin/lazypipe.git
cd lazypipe

Installing dependencies

Installing dependencies with Conda

conda create -n blast -c bioconda blast
conda create -n lazypipe -c bioconda -c eclarke bwa centrifuge csvtk fastp krona megahit mga minimap2 samtools seqkit spades snakemake-minimal taxonkit trimmomatic numpy scipy fastcluster requests

Or from conda yaml files:

conda env create -f blast.yml
conda env create -f lazypipe.yml

This will create separate conda environment for blast. All other tools are installed under lazypipe. To activate all installed binaries type:

conda activate blast
conda activate --stack lazypipe

Set taxonomy database location for KronaGraph (replace $CONDA_PREFIX and $data according to your settings):

 rm -rf $CONDA_PREFIX/conda/env/lazypipe/opt/krona/taxonomy
 ln -s $data/taxonomy $CONDA_PREFIX/conda/env/lazypipe/opt/krona/taxonomy

Set env variable $TM to point to trimmomatic directory:

 export TM=$CONDA_PREFIX/share/trimmomatic

Download PANNZER (version 02/2022 or later) and set runsanspanz.py as executable to your path:

wget http://ekhidna2.biocenter.helsinki.fi/sanspanz/SANSPANZ.3.tar.gz
tar -zxvf SANSPANZ.3.tar.gz
sed -i "1 i #!$(which python)" SANSPANZ.3/runsanspanz.py
ln -sf  $(pwd)/SANSPANZ.3/runsanspanz.py ~/bin/runsanspanz.py

Installing dependencies manually

Download and unpack dependencies listed in Table 1. Then copy or link these executables to your ~/bin folder. For example:

wget https://github.com/lh3/minimap2/releases/download/v2.24/minimap2-2.24_x64-linux.tar.bz2
tar -xjvf minimap2-2.24_x64-linux.tar.bz2
cp minimap2-2.24_x64-linux/minimap2 ~/bin/

Note that Snakemake requires conda for installation (for details see https://snakemake.readthedocs.io/):

conda create -c bioconda -n snakemake snakemake-minimal
conda activate snakemake

Table 1: Lazypipe dependencies Tools in square brackets mark binaries that are not required for basic Lazypipe runs. When installed, these will provide additional options/functionalities.

Installing Perl modules

Install modules to local-lib ~/perl5

cpan --local-lib=~/perl5 File::Basename File::Temp Getopt::Long YAML::Tiny
export PERL5LIB=~/perl5/lib/perl5:{$PERL5LIB}

Installing R libraries

Open R console and type

install.packages( c("reshape","openxlsx") );

Installing reference databases

Download and install reference databases using Table 1 and the following instructions.

Start by installing NCBI Taxonomy. In config.yaml set local path to taxonomy. Then install by running:

perl perl/install_db.pl --db taxonomy

Running 1st round annotations with SANS or Minimap2 and 2nd round annotations with BLASTN (recommended):

• SANS: no databases required

• Minimap2: in config.yaml set local path to minimap_db. Then download and unpack the latest NCBI NT abv minimap database to that location.

• BLASTN: in config.yaml set local path to blastn_vi_db. Then install by running:

  perl perl/install_db.pl --db blastn_vi
  

Running 1st round annotations with BLASTP or Centrifuge:

• BLASTP: in config.yaml set local path to blastp_db. Then download your preferred BLAST database to that location.

• Centrifuge: in your config.yaml set local path to centrifuge_db. Then download and unpack NCBI NT habv centrifuge index to that location.

Running 2nd round annotations for bacteriophages:

• in config.yaml set local paths to minimap_db_phages and minimap_db_phages_metadata. Then download Gut Phage Database (GPD_sequences.fa and GPD_metadata.tsv) to these locations.

• index the database by running:

  minimap2 -t 4 -x asm20 -d GPD_metadata.fa.mmi GPD_metadata.fa
  

Table 2. Databases used by Lazypipe.

Test Perl and Snakemake interfaces

Perl interface

perl lazypipe.pl

Snakemake interface

snakemake -np all

Running Lazypipe with lazypipe.pl

lazypipe.pl runs your metagenomic analysis step-by-step. For example, to run preprocessing and assembling type

perl lazypipe.pl -1 data/sample_R1.fq.gz --pipe pre,ass -v

###lazypipe.pl command-line options:

Default options and additional settings are defined in config.yaml file. Note that command line options take precedence over options in config.yaml file.

###Additional options in config.yaml:

Example 1: analyzing sample data with lazypipe.pl

For this example, we will use data/samples/M15small_R\*.fastq (PE Illumina reads from mink feces env sample).

Run main steps with default options

perl lazypipe.pl -1 data/samples/M15small_R1.fastq -s M15 -p main -t 8

Run preprocessing with Trimmomatic

perl lazypipe.pl -1 data/samples/M15small_R1.fastq -s M15 -p pre --pre trimm -t 8 -v

Filter host reads with Neovison vison genome

wget https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/900/108/605/GCA_900108605.1_NNQGG.v01/GCA_900108605.1_NNQGG.v01_genomic.fna.gz
mv GCA_900108605.1_NNQGG.v01_genomic.fna.gz $data/hostgen/
perl lazypipe.pl -1 data/samples/M15small_R1.fastq -s M15 -p flt --hostgen $data/hostgen/GCA_900108605.1_NNQGG.v01_genomic.fna.gz -t 8 -v

Run assembling with SPAdes + realign reads to assembly

perl lazypipe.pl -1 data/samples/M15small_R1.fastq -s M15 -p ass,rea --ass spades -t 8

Run annotation with minimap2 + update reports

perl lazypipe.pl -1 data/samples/M15small_R1.fastq -s M15 -p ann,rep --ann minimap -t 8 -v

Confirm virus contigs with local blastn

perl lazypipe.pl -1 data/samples/M15small_R1.fastq -s M15 -p blastv -t 8 -v

Search for viruses in unmapped contigs with local blastn

perl lazypipe.pl -1 data/samples/M15small_R1.fastq -s M15 -p blastu -t 8 -v

Pack results to *.tar.gz

perl lazypipe.pl -1 data/samples/M15small_R1.fastq -s M15 -p pack -v

Example 1: generated reports

By default, all results are printed to ./res-dir/sample-dir/, in this case to ./results/M15/:

Assembled contigs and predicted ORFs

Abundance tables

Figure 2. abund_table.xlsx

Spreadsheets with taxon abundancies are printed to abund_table.xlsx. In the bundancies are displayed in separate tables for viruses (excluding bacteriophages), bacteria, bacteriophages and eukaryots. For each domain abundancies are displayed at three taxonomic levels: species, genus and family.

For raw abundance data see abund_table.tsv.

Columns in abund_table.xlsx

Annotation tables

Figure 3. contigs.annot.xslx

Spreadsheets with contig annotations are printed to contigs.annot.xslx. Spreadsheets are displayed separately for viruses (excluding bacteriophages), bacteria, bacteriophages and eukaryots. Columns displayed depend on the applied homology search (sans/blastp/minimap2).

Running -p blastv will also print blastn annotation for contigs_vi.fa to contigs_vi.annot.xlsx.

Running -p blastu will also print blastn annotation for contigs_un.fa to contigs_un.annot.xlsx.

For raw annotation data see contigs[_un|_vi].annot.tsv.

Key columns in contigs[_un|_vi].annot.xslx:

Krona graph

Figure 4. krona_graph.html

Estimated taxon abundancies are also displayed as an interactive Krona graph: krona_graph.html.

Quality control plots

Figure 5. Read survival plots

Quality Control (QC) plots include length histograms for reads and contigs, and survival plots. The survival plots track retained reads after each pipeline step.

Running Lazypipe with Snakemake

Example2: analyzing sample data

Snakemake works by declaring the end file you wish to produce.

Start by listing your input fastq files under datain key in config.yaml file. Pretend each file with sample id.

For this example, we will use data/samples/M15small. In your config.yaml type:

datain:
    M15: data/samples/M15small_R1.fastq

Run main steps with default options

snakemake --cores 8 results/M15.tar.gz -p

Run preprocessing with Trimmomatic. Overwrite any trimmed reads produced by previous runs with --force:

snakemake --config pre="trimm" --cores 8 results/M15/trimmed_paired1.fq.gz --force -p

Run assembling with SPAdes. Overwrite any contigs produced by previous runs with --force:

snakemake --config ass="spades" --cores 8 results/M15/contigs.fa --force -p

Redo annotation with minimap2:

snakemake --config ann="minimap" --cores 16 results/M15.tar.gz --force -p

Confirm viral contigs with local blastn:

snakemake --config blastv=1 --cores 16 results/M15/contigs_vi.annot.xlsx -p

Search for viruses in unmapped contigs with local blastn

snakemake --config blastu=1 --cores 16 results/M15/contigs_un.annot.xlsx -p

Retrieving reads for a contig or taxid

Start by unzipping your source fasta:

gunzip -k results/M15/trimmed_paired*_hostflt.fq.gz

Retrieve all reads mapped to contig k141.100 in sample M15

bin/retrieve_reads -c k141.100 -r results/M15 -v

Retrieve all reads mapped to Mamastrovirus (taxid 1239574) in sample M15:

bin/retrieve_reads -t 1239574 -r results/M15 -v

Citing Lazypipe

1. Plyusnin Ilya, Olli Vapalahti, Tarja Sironen, Ravi Kant, and Teemu Smura. “Enhanced Viral Metagenomics with Lazypipe 2.” Viruses 15, no. 2 (February 4, 2023): 431. https://doi.org/10.3390/v15020431

2. Ilya Plyusnin, Ravi Kant, Anne J. Jaaskelainen, Tarja Sironen, Liisa Holm, Olli Vapalahti, Teemu Smura. (2020) Novel NGS Pipeline for Virus Discovery from a Wide Spectrum of Hosts and Sample Types. Virus Evolution, veaa091, https://doi.org/10.1093/ve/veaa091

Contact

Project website: https://www.helsinki.fi/en/projects/lazypipe

Contact email: grp-lazypipe@helsinki.fi