BIOPZ-Katsantoni Maria authored 5 years ago and Alex Kanitz committed 4 years ago

* Sequencing mode-related changes:
  * allowed sequencing modes in Snakemake input table changed from `paired_end` and `single_end` to `pe` and `se`, respectively
  * remove sequencing mode from output paths for each rule
  * corresponding wild cards removed entirely from all rules that do not depend on sequencing mode (currently all rules that are defined in the main `Snakefile` in the project root directory)
  * where absolutely necessary, sequencing mode is added as part of output file or directory instead
  * remove dependency of sequencing mode for rule for `FastQC`; now runs separately for each strand
* Changes related to MultiQC and output file/directory structure
  * moving and renaming outputs for MultiQC is no longer required
  * code to create MultiQC custom config externalized into script `scripts/rhea_multiqc_config.py`
  * add MultiQC output files with deterministic output to md5 sum checks performed during execution of `tests/test_integration_workflow/test.{local,slurm}.sh`
  * output filenames for each rule now follow this general structure: `samples/{sample_name}/{rule}/{output_file}`
  * change log directory structure matches results directory structure
* Miscellaneous changes
  * consistent, PEP8-compliant formatting in most parts, including Snakemake files, where allowed
  * remove rule `extract_decoys_salmon`; equivalent file `chrName.txt` produced by `star_index` is used instead
  * add rule `start` which copies sample data to the results directory and enforces uniform naming
  * refactoring of ALFA rules and modification of the CI/CD test to ensure compatibility

- log and, if workflow is executed on cluster, cluster log directories are explicitly created in `Snakefile`
- location of main log directory can be configured in `config.yaml` (field `log_dir`, previously: `local_log`; requires change in script `labkey_to_snakemake.py` as well as subworkflows as field name is hard-coded there)
- location of cluster log directory can be configured in `cluster.json` (in field `__default__` -> `out`)
- `config.yaml` and `cluster.json` in `tests/input_files` are set such that a directory `logs/` is created in the directory where Snakemake is run (i.e., the directory of each test); cluster logs are stored in a subdirectory `logs/cluster`
- removes instructions to explicitly create log directories from docs and all test scripts
- cleans up main `Snakefile` (apart from Snakemake-specific syntax, now passes `flake8` linter test)

- trap call functionalized through cleanup() function
- function added to all test scripts
- function prints out exit status of last command before trap
- flag `--verbose` added to Snakemake calls in all test scripts
- script tests rename to follow naming convention 'test_script_<script_name>_<script_run_mode>

BIOPZ-Katsantoni Maria authored 5 years ago and Alex Kanitz committed 5 years ago

- add script that prepares Snakemake input files 'samples.tsv' and 'config.yaml' from LabKey table
- script either connects to API directly (with '--remote' and related options) or processes a tab-separated LabKey dump file
- add tests for both use cases
- common input files for tests now in 'tests/input_files'
- update all other tests to account for new file locations
- update documentation

add tests to workflow integration test `tests/test_workflow_integration` that
- verify that STAR alignments match expected alignments (based on "ground truth" files)
- verify that Salmon gene quantification assign the correct number of reads to each gene (based on "ground truth files)

resolves #49

- replaces existing larger libraries and annotations in test cases `test_create_dag_chart` and `test_integration_workflow`
- adds the following new test files:
  - `chr1-10000-20000.fa`: artificial chromosome of length 10'000 (based on human chromosome 1)
  - `chr1-10000-20000.gtf`: matching gene annotation file with two gene and three multi-exon transcripts entries
  - `chr1-10000-20000.transcripts.fa`: sequences of the transcripts listed in the gene annotation file
  - `synthetic.mate_?.fastq.gz`: 10 read pairs randomly sampled from the genic regions of the artificial chromosome
  - `synthetic.*.bed`: BED files with expected alignments for each read; names of overlapping genes are specified in a 7th column
- updates file paths in the relevant sample tables
- extends and updates checksum checking of result files in CI/CD pipeline

`README.md` file describes
- aim and background of the project (including the workflow DAG representation)
- how to install requirements (including setting up a `conda` environment for the project)
- how to execute the workflow run integration test
- how to run the workflow on your own samples (including how to auto-generate required params from LabKey metadata)

Additional minor changes:
- minor changes in various test and related files, including updates of paths
- root directory now includes subdirectory `runs/` for a user's workflow runs (contents not version-controlled)

- set up integration test for Snakefile in dedicated folder; current test case was left untouched for the time being, despite requiring large input files
- set up DAG chart creation test in dedicated folder; script creates an SVG representation of the workflow DAG at `images/workflow_dag.svg`
- both tests have been added to the GitLab CI/CD configuration; the latter test ensures that always the latest version of the
- all tests are now located inside subdirectories of `tests/`; test scripts and configuration files for test runs etc. have been moved to the appropriate test directories
- for the time being, required input files for each test are placed within the individual test directories; a layout for common test files should be introduced later and paths and bind paths in tests adapted
- make script `scripts/labkey_api.py` executable