ZARP

Snakemake workflow that covers common steps of short read RNA-Seq library analysis developed by the Zavolan lab.

Reads are analyzed (pre-processed, aligned, quantified) with state-of-the-art tools to give meaningful initial insights into the quality and composition of an RNA-Seq library, reducing hands-on time for bioinformaticians and giving experimentalists the possibility to rapidly assess their data.

Below is a schematic representation of the individual steps of the workflow ("pe" refers to "paired-end"):

For a more detailed description of each step, please refer to the workflow documentation.

Requirements

Currently the workflow is only available for Linux distributions. It was tested on the following distributions:

• CentOS 7.5
• Debian 10
• Ubuntu 16.04, 18.04

Installation

Cloning the repository

Traverse to the desired directory/folder on your file system, then clone/get the repository and move into the respective directory with:

git clone ssh://git@git.scicore.unibas.ch:2222/zavolan_group/pipelines/zarp.git
cd zarp

Installing Conda

Workflow dependencies can be conveniently installed with the Conda package manager. We recommend that you install Miniconda for your system (Linux). Be sure to select Python 3 option. The workflow was built and tested with miniconda 4.7.12. Other versions are not guaranteed to work as expected.

Installing dependencies

For improved reproducibility and reusability of the workflow, each individual step of the workflow runs either in its own Singularity container or in its own Conda virtual environemnt. As a consequence, running this workflow has very few individual dependencies. However, for the container execution it requires Singularity to be installed on the system where the workflow is executed. As the functional installation of Singularity requires root privileges, and Conda currently only provides Singularity for Linux architectures, the installation instructions are slightly different depending on your system/setup:

For most users

If you do not have root privileges on the machine you want to run the workflow on or if you do not have a Linux machine, please install Singularity separately and in privileged mode, depending on your system. You may have to ask an authorized person (e.g., a systems administrator) to do that. This will almost certainly be required if you want to run the workflow on a high-performance computing (HPC) cluster. We have successfully tested the workflow with the following Singularity versions:

• v2.4.5
• v2.6.2
• v3.5.2

After installing Singularity, install the remaining dependencies with:

conda env create -f install/environment.yml

As root user on Linux

If you have a Linux machine, as well as root privileges, (e.g., if you plan to run the workflow on your own computer), you can execute the following command to include Singularity in the Conda environment:

conda env create -f install/environment.root.yml

Activate environment

Activate the Conda environment with:

conda activate zarp

Installing non-essential dependencies

Most tests have additional dependencies. If you are planning to run tests, you will need to install these by executing the following command in your active Conda environment:

conda env update -f install/environment.dev.yml

Testing the installation

We have prepared several tests to check the integrity of the workflow and its components. These can be found in subdirectories of the tests/ directory. The most critical of these tests enable you execute the entire workflow on a set of small example input files. Note that for this and other tests to complete successfully, additional dependencies need to be installed.

Test workflow on local machine

Execute the following command to run the test workflow on your local machine (with singularity):

bash tests/test_integration_workflow/test.local.sh

Alternatively execute the following command to run the test workflow on your local machine (with conda):

bash tests/test_integration_workflow_with_conda/test.local.sh

Test workflow via Slurm

Execute the following command to run the test workflow on a Slurm-managed high-performance computing (HPC) cluster:

bash tests/test_integration_workflow/test.slurm.sh

or

bash tests/test_integration_workflow_with_conda/test.slurm.sh

NOTE: Depending on the configuration of your Slurm installation or if using a different workload manager, you may need to adapt file cluster.json and the arguments to options --config and --cores in the file test.slurm.sh, both located in directory tests/test_integration_workflow. Consult the manual of your workload manager as well as the section of the Snakemake manual dealing with cluster execution.

Running the workflow on your own samples

1. Assuming that your current directory is the repository's root directory, create a directory for your workflow run and traverse inside it with:

   mkdir config/my_run
   cd config/my_run

2. Create empty sample table, workflow configuration and, if necessary, cluster configuration files:

   touch samples.tsv
   touch config.yaml
   touch cluster.json

3. Use your editor of choice to populate these files with appropriate values. Have a look at the examples in the tests/ directory to see what the files should look like, specifically:

   • samples.tsv
   • config.yaml
   • cluster.json

4. Create a runner script. Pick one of the following choices for either local or cluster execution. Before execution of the respective command, you must replace the data directory placeholders in the argument of the --singularity-args option with a comma-separated list of all directories containing input data files (samples and any annoation files etc) required for your run.

   Runner script for local execution:

   cat << "EOF" > run.sh
   #!/bin/bash
   snakemake \
       --snakefile="/path/to/Snakefile" \
       --configfile="config.yaml" \
       --cores=4 \
       --printshellcmds \
       --rerun-incomplete \
       --use-singularity \
       --singularity-args="--bind <data_dir_1>,<data_dir_2>,<data_dir_n>"
   EOF

   OR

   Runner script for Slurm cluster exection (note that you may need to modify the arguments to --cluster and --cores depending on your HPC and workload manager configuration):

   cat << "EOF" > run.sh
   #!/bin/bash
   mkdir -p logs/cluster_log
   snakemake \
       --snakefile="/path/to/Snakefile" \
       --configfile="config.yaml" \
       --cluster-config="cluster.json" \
       --cluster="sbatch --cpus-per-task={cluster.threads} --mem={cluster.mem} --qos={cluster.queue} --time={cluster.time} --job-name={cluster.name} -o {cluster.out} -p scicore" \
       --cores=256 \
       --printshellcmds \
       --rerun-incomplete \
       --use-singularity \
       --singularity-args="--bind <data_dir_1>,<data_dir_2>,<data_dir_n>"
   EOF

   When running the pipeline with conda you should use the --use-conda flag instead of --use-singularity and --singularity-args.

5. Start your workflow run:

   bash run.sh

Configuring workflow runs via LabKey tables

Our lab stores metadata for sequencing samples in a locally deployed LabKey instance. This repository provides two scripts that give programmatic access to the LabKey data table and convert it to the corresponding workflow inputs (samples.tsv and config.yaml), respectively. As such, these scripts largely automate step 3. of the above instructions. However, as these scripts were written specifically for the needs of our lab, they are likely not directly usable or, at least, will require considerable modification for other setups (e.g., different LabKey table structure). Nevertheless, they can serve as an example for interfacing between LabKey and your workflow.

NOTE: All of the below steps assume that your current working directory is the repository's root directory.

1. The scripts have additional dependencies that can be installed with:

   pip install -r scripts/requirements.txt

2. In order to gain programmatic access to LabKey via its API, a credential file is required. Create it with the following command after replacing the placeholder values with your real credentials (talk to your LabKey manager if you do not have these):

   cat << EOF | ( umask 0377; cat >> ${HOME}/.netrc; )
   machine <remote-instance-of-labkey-server>
   login <user-email>
   password <user-password>
   EOF

3. Generate the workflow configuration with the following command, after replacing the placeholders with the appropriate values (check out the help screen with option '--help' for further options and information):

   python scripts/prepare_inputs.py \
       --labkey-domain="my.labkey.service.io"
       --labkey-domain="/my/project/path"
       --input-to-output-mapping="scripts/prepare_inputs.dict.tsv" \
       --resources-dir="/path/to/my/genome/resources" \
       --output-table="config/my_run/samples.tsv" \
       --config_file="config/my_run/config.yaml" \
       <table_name>

Additional information

The metadata field names in the LabKey instance and those in the parameters in the Snakemake workflow have different names. A mapping between LabKey field identifiers and Snakemake parameters is listed below: