BIOPZ-Katsantoni Maria authored 5 years ago and Alex Kanitz committed 5 years ago

- add script that prepares Snakemake input files 'samples.tsv' and 'config.yaml' from LabKey table
- script either connects to API directly (with '--remote' and related options) or processes a tab-separated LabKey dump file
- add tests for both use cases
- common input files for tests now in 'tests/input_files'
- update all other tests to account for new file locations
- update documentation

repo follows recommended structure

Closes #31

See merge request zavolan_group/pipelines/rnaseqpipeline!22

hot fix integration test script

See merge request zavolan_group/pipelines/rnaseqpipeline!21

LabKey-like input to Snakmake input

See merge request zavolan_group/pipelines/rnaseqpipeline!20

BIOPZ-Katsantoni Maria authored 5 years ago and Alex Kanitz committed 5 years ago

- separate organism genome architecture (different input folder)
- change MD5 checksums to match the new output

update install instructions

See merge request zavolan_group/pipelines/rnaseqpipeline!17

CJHerrmann authored 5 years ago and Alex Kanitz committed 5 years ago

- describe installation of Singularity inside `conda` environment when user has root access
- put clone repo section before install section

display rule graph instead of DAG

Closes #52

See merge request zavolan_group/pipelines/rnaseqpipeline!18

CJHerrmann authored 5 years ago and Alex Kanitz committed 5 years ago

- add script `tests/test_rule_graph/test.sh` to generate a rule graph in `images/rule_graph.svg`
- display rule graph created in `README.md` instead of specific workflow DAG
- add test script to GitLab CI config
- renamed test to create workflow DAG from `test_create_dag_chart` to `test_create_dag_image` (also output file is renamed from `images/workflow_dag.svg` to `images/dag_test_workflow.svg`

Ground truth checks

Closes #49

See merge request zavolan_group/pipelines/rnaseqpipeline!19

add tests to workflow integration test `tests/test_workflow_integration` that
- verify that STAR alignments match expected alignments (based on "ground truth" files)
- verify that Salmon gene quantification assign the correct number of reads to each gene (based on "ground truth files)

resolves #49

Replace test files with small synthetic ones

See merge request zavolan_group/pipelines/rnaseqpipeline!16

- replaces existing larger libraries and annotations in test cases `test_create_dag_chart` and `test_integration_workflow`
- adds the following new test files:
  - `chr1-10000-20000.fa`: artificial chromosome of length 10'000 (based on human chromosome 1)
  - `chr1-10000-20000.gtf`: matching gene annotation file with two gene and three multi-exon transcripts entries
  - `chr1-10000-20000.transcripts.fa`: sequences of the transcripts listed in the gene annotation file
  - `synthetic.mate_?.fastq.gz`: 10 read pairs randomly sampled from the genic regions of the artificial chromosome
  - `synthetic.*.bed`: BED files with expected alignments for each read; names of overlapping genes are specified in a 7th column
- updates file paths in the relevant sample tables
- extends and updates checksum checking of result files in CI/CD pipeline

Wire all available rules

See merge request zavolan_group/pipelines/rnaseqpipeline!15

BIOPZ-Iborra de Toledo Paula authored 5 years ago and Alex Kanitz committed 5 years ago

- Rules now integrated include #7, #8, #9, #10, #11, #12, #13, #14, #15, #16, #17
- Applies both for single-end and paired-end subworkflows
- DAG image updated
- Test sample tables and outcomes updated
- Unused preprocessing subworkflow removed

Closes #48
Closes #6

Labkey input merge

See merge request zavolan_group/pipelines/rnaseqpipeline!14

fix various small issues

See merge request zavolan_group/pipelines/rnaseqpipeline!13

- remove log files and add '.snakemake' directories to '.gitignore'
- update wrong link in 'README.md'
- delete superfluous script documentation 'scripts/labkey_api.md'
- add Snakemake-specific file extension '.smk' to subworkflows
- remove non-deterministic workflow output from md5 sums

add documentation

See merge request zavolan_group/pipelines/rnaseqpipeline!12

`README.md` file describes
- aim and background of the project (including the workflow DAG representation)
- how to install requirements (including setting up a `conda` environment for the project)
- how to execute the workflow run integration test
- how to run the workflow on your own samples (including how to auto-generate required params from LabKey metadata)

Additional minor changes:
- minor changes in various test and related files, including updates of paths
- root directory now includes subdirectory `runs/` for a user's workflow runs (contents not version-controlled)

clean up tests

See merge request zavolan_group/pipelines/rnaseqpipeline!11

- set up integration test for Snakefile in dedicated folder; current test case was left untouched for the time being, despite requiring large input files
- set up DAG chart creation test in dedicated folder; script creates an SVG representation of the workflow DAG at `images/workflow_dag.svg`
- both tests have been added to the GitLab CI/CD configuration; the latter test ensures that always the latest version of the
- all tests are now located inside subdirectories of `tests/`; test scripts and configuration files for test runs etc. have been moved to the appropriate test directories
- for the time being, required input files for each test are placed within the individual test directories; a layout for common test files should be introduced later and paths and bind paths in tests adapted
- make script `scripts/labkey_api.py` executable

Input parser

Closes #39

See merge request zavolan_group/pipelines/rnaseqpipeline!10

BIOPZ-Katsantoni Maria authored 5 years ago and Alex Kanitz committed 5 years ago

Adds script `scripts/labkey_to_snakemake.py` which
- maps LabKey table fields to Snakemake parameters
- assembles required parameters from the table data
- infers required parameters from the input data
- produces files `config.yaml` and `samples.tsv` required by the Snakemake pipeline

A self-contained integration test for the script is located at `tests/test_scripts_labkey_to_snakemake` (execute script `test.sh`) and was added to the CI/CD pipeline.

Note that intermittent changes to the `master` branch were merged into this branch to forego conflicts during merging.

Closes #39

Fix params.direcionality for kallisto rule

See merge request zavolan_group/pipelines/rnaseqpipeline!9

Update fake samples file: tests/samples.tsv

See merge request zavolan_group/pipelines/rnaseqpipeline!7

BIOPZ-Gypas Foivos authored 5 years ago and Alex Kanitz committed 5 years ago

- replace corrupt input files
- add script that runs locally named `run_test.sh` in the snakemake directory
- add script to CI/CD pipeline
- fix typo in `paired_end.snakefile`
- update samples.tsv with fake adapter `XXXXXXX` when we do not want to trim them

Labkey api

See merge request zavolan_group/pipelines/rnaseqpipeline!8

BIOPZ-Börsch Anastasiya authored 5 years ago and Alex Kanitz committed 5 years ago

- add usage information in file `scripts/labkey_api.md`

Added sample fastq.gz paired-end input 1000 record files, adjusted dummy sample.tsv

See merge request zavolan_group/pipelines/rnaseqpipeline!6

add continuous integration support

See merge request zavolan_group/pipelines/rnaseqpipeline!5

Subpipelines

Closes #7, #8, #16, #17, #14, and #15

See merge request zavolan_group/pipelines/rnaseqpipeline!4

Rename snakemake/paired_end.snakemake to snakemake/paired_end.snakefile. Fix wiring of rules. Add fake tests.

Clean up unused code in snakemake/Snakefile. Add 2 threads in htseq_qa (paired end mode). Add example of tsv files from LabKey.