Use biocontainers images for star, gffread, salmon, kallisto, cutadapt, samtools, fastqc, alfa, bedtools, bedgraphtobigwig. Change container from bash to ubuntu. Fixes #149

Use biocontainers images for star, gffread, salmon, kallisto, cutadapt, samtools, fastqc, alfa, bedtools, bedgraphtobigwig. Change container from bash to ubuntu. Fixes #149

feat: enable user to configure CLI params per rule

See merge request !83

Remove default arguments from rules

See merge request !80

BIOPZ-Katsantoni Maria authored 3 years ago and BIOPZ-Gypas Foivos committed 3 years ago

and are therefore not required in the rules.
- Snakefile
 star_rpm: --outWigNorm (default RPM was used)
 star_rpm: --outWigStrand (default Stranded was used)
 rename_star_rpm_for_alfa: orientation in params is redundant (Fixes #152)
- single_end.snakefile.smk
 map_genome_star: outFilterMismatchNoverLmax
 map_genome_star: outFilterScoreMinOverLread
 map_genome_star: outFilterMatchNminOverLread
 quantification_salmon: --writeUnmappedNames
- paired_end.snakefile.smk
  pe_map_genome_star: outFilterMismatchNoverLmax
  pe_map_genome_star: outFilterScoreMinOverLread
  pe_map_genome_star: outFilterMatchNminOverLread
  quantification_salmon: --writeUnmappedNames

- use of [`temp()`](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html?highlight=temp#protected-and-temporary-files) statements in snakemake rules
  - extract_transcriptome
  - concatenate_transcriptome_and_genome
  - extract_transcripts_as_bed12
  - calculate_TIN_scores
  - star_rpm
  - rename_star_rpm_for_alfa
  - alfa_qc
  - sort_bed_4_big
  - pe_remove_adapters_cutadapt
  - pe_remove_polya_cutadapt
  - remove_adapters_cutadapt
  - remove_polya_cutadapt
- use of [`--notemp`](https://snakemake.readthedocs.io/en/stable/executing/cli.html?highlight=notemp#BEHAVIOR) in test scripts (local and slurm) for successful execution without removing temp files
  - test_alfa
  - test_create_dag_image
  - test_create_rule_graph
  - test_integration_worklow
  - test_integration_workflow_multiple_lanes
- use of [`shadow`](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html?highlight=shadow#shadow-rules) directive in rules
  - star_rpm
  - pe_map_genome_star
  - pe_quantification_salmon
  - pe_genome_quantification_kallisto
  - map_genome_star
  - quantification_salmon
  - genome_quantification_kallisto
- See comment in #137 for setting an optional `shadow prefix`, especially useful in cluster execution
- See comment in #137 for impact of disk space savings on a real-world dataset
- correctly add snakemake `{threads}` in execution in rules
  - fastqc
  - quantification_kallisto

Fixes #137.

MultiQC plugins for TIN scores and ALFA
See merge request !79

PCA analysis on genes and transcripts TPM for kallisto and salmon

See merge request !77

- Delete unused files scripts/fg_extract_transcripts.py, scripts/heatmap_and_clustermap.py, scripts/perform_PCA.py
- Add rules (pca_kallisto, pca_salmon) that run zpca (https://github.com/zavolanlab/zpca) on genes and transcripts TPM tables from kallisto and salmon.
- The output is wired to multiqc_report but the plots are not visualized to multiqc. Update documentation.
- Update dag and rulegraph.
Fixes #140 #142

docs: revised terminology for non-experts

See merge request !73

Merge kallisto

See merge request !75

BIOPZ-Iborra de Toledo Paula authored 4 years ago and BIOPZ-Gypas Foivos committed 4 years ago

The rules rely on https://github.com/zavolanlab/merge_kallisto
Update info in pipeline_documentation.md

Cluster.json was updated

See merge request !74

feat: add support for multi-lane samples

See merge request !71

BIOPZ-Katsantoni Maria authored 4 years ago and Alex Kanitz committed 4 years ago

fix: Renamed samples_concat.tsv to samples.multiple_lanes.tsv. Renamed rows with split with the same name as the other test samples, so that I do not change the tests (md5 and sunch). Removed the one lane samples. Created config that uses this tsv file

refactor: rename LabKey/input table column

See merge request !70

Add Snakemake report

See merge request !64

docs: rename project/workflow

See merge request !69

fix(prepare_inputs): support relative paths

See merge request !68

Salmon argument fix

See merge request !67

BIOPZ-Katsantoni Maria authored 4 years ago and BIOPZ-Gypas Foivos committed 4 years ago

fix: added arguments for fragment size mean and sd required for the salmon quantification approach in single-end mode

Gtf2bed update

See merge request !65

Extend workflow description

See merge request zavolan_group/pipelines/rhea!63

- add linked ToC
- add table of third party software
- add rule graph
- clean up sample table description
- formalize structure for each rule description
- cross-reference rules via inputs/outputs
- clean list of references

Major refactoring before release

Closes #117, #112, #111, #108, #107, #101, and #74

See merge request zavolan_group/pipelines/rhea!62

- clean up command line interface
  - improve descriptions
  - add consistent structure
  - remove or merge superfluous CLI arguments
  - set defaults
  - update test calls
  - update docs
  - when importing data from LabKey, table is saved to 'samples.tsv.labkey' in same directory as Snakemake sample table
- allow user to specify environment variables and relative paths in input table and on CLI
  - relative paths in the input table are interpreted with respect to the directory containing the input table
  - relative paths will are interpreted with respect to the current working directory; this is to achieve portability with respect to tests but is discouraged in production because its behavior is not very predictable from the user's perspective; consequently a warning is thrown
- set STAR index size to read length - 1
- remove `gtf_filtered` and `tr_fasta_filtered` and update Snakefiles and test sample tables accordingly
- rename some MultiQC report-related parameters and update Snakefiles and test config files accordingly
- add logging
- add docstrings to module and all functions
- add typing definitions to all functions
- restructure and comment code to improve readability
- linters `flake8` and `mypy` pass

BIOPZ-Katsantoni Maria authored 4 years ago and Alex Kanitz committed 4 years ago

* Sequencing mode-related changes:
  * allowed sequencing modes in Snakemake input table changed from `paired_end` and `single_end` to `pe` and `se`, respectively
  * remove sequencing mode from output paths for each rule
  * corresponding wild cards removed entirely from all rules that do not depend on sequencing mode (currently all rules that are defined in the main `Snakefile` in the project root directory)
  * where absolutely necessary, sequencing mode is added as part of output file or directory instead
  * remove dependency of sequencing mode for rule for `FastQC`; now runs separately for each strand
* Changes related to MultiQC and output file/directory structure
  * moving and renaming outputs for MultiQC is no longer required
  * code to create MultiQC custom config externalized into script `scripts/rhea_multiqc_config.py`
  * add MultiQC output files with deterministic output to md5 sum checks performed during execution of `tests/test_integration_workflow/test.{local,slurm}.sh`
  * output filenames for each rule now follow this general structure: `samples/{sample_name}/{rule}/{output_file}`
  * change log directory structure matches results directory structure
* Miscellaneous changes
  * consistent, PEP8-compliant formatting in most parts, including Snakemake files, where allowed
  * remove rule `extract_decoys_salmon`; equivalent file `chrName.txt` produced by `star_index` is used instead
  * add rule `start` which copies sample data to the results directory and enforces uniform naming
  * refactoring of ALFA rules and modification of the CI/CD test to ensure compatibility