Results directory uses lots of disk space
The results directory contains a lot of results. For example, the GCN4 project with 9 SE samples yields a 89GB big results directory. Each samples takes around 8-10GB. Big chuncks of this are used by quant_kallisto, start and map_genome with each >1GB.
Also, I think there could be a redundancy in STAR_coverage and the results in ALFA as there is a copy of BED files at some point.
Are all the files needed or could some be marked as temporary in the workflow?
E.g. results/star_indexes is already 1606M. Could it be deleted once the workflow finished successfully?
A potential side effect of cleaning up the results, besides the reduction of disk space, is that browsing the results becomes easier.
Activity
added Discuss label
changed milestone to %pipeline
assigned to @burri0000
After the discussion on 02.10.2020, we realised the need to make sure that the files are only removed once the CI tests have run.
The idea is to
- mark the appropriate files as
temp()in the respective rules - adjust the snakemake call with the flag
--notempin order to ignore the temp statements. - After the tests, a snakemake call with
--delete-temp-outputremoves the temp files.
- mark the appropriate files as
@burri0000 Makes sense to me. The only thing that I am not sure, is whether we need to run the last command
--delete-temp-outputas the tests are anyway deleted once the CI pipeline finishes. So just for the tests, we can provide the--notempflag, while for normal runs we run it without. And yes, you should encapsulate the rules withtemp()when necessary.I have some questions regarding which files to keep in the results directory, besides the ones used for the finish rule:
- is there a need for the various index directories (kallisto, salmon, star, alfa)?
- per sample, there are various directories with the mapped reads, in particular
map_genome,sample.salmon.se,quant_kallisto. Should we retain these? - Can the various fastq files be removed?
Snakemake does not support to remove whole directories. I think I have to use the snakemake command
onsuccesssee here which is executed after the workflow finishes and in which I can use shell commands to remove directories. For testing, this can be ignored with--no-hooks.- I think we need to be careful with the indexes, as the indexes can be used/shared between runs. Let's discuss this tomorrow.
- I think that we need to keep the
.bam,.bam.baiand thebigwigfiles. We can discard the individualsample.salmon.seandquant_kallistoas we have a rule where we concatenate the individual count tables. - I think that you can remove intermediate fastq files (e.g. the output of cutadapt), but I would not delete the original fastq files.
Let's discuss then also the
onsuccessand--no-hookstomorrow.
added Doing label
Provide some optional cleanup of intermediate files ( #135 (closed))
mentioned in commit 9308bd60
The use of
temp(),shadowand manual deletion of indexes, the results directory is cleaned up. Only the files for creating multiqc should've been retained.I tested GCN4_project on slurm and got with
du -h -d 0 resultsthe following:- before: 87G
- after: 31G
Test on local machine
I also tested the samples on a local machine with 12 cores and 32GB RAM. With
/usr/bin/timefor the snakemake command,- it took 2:25 hours
- with 1037% percentage CPU and
- a maximum resident set size of 4.3 GBytes.
Important note
By default, the shadow rules are executed in
.snakemake/shadow. In an attempt to decrease I/O on the cluster and with the help of https://snakemake.readthedocs.io/en/stable/project_info/faq.html?highlight=shadow#how-can-i-make-use-of-node-local-storage-when-running-cluster-jobs I added the following in the snakemake call:--shadow-prefix $TMPDIR --singularity-args="--bind $TMPDIR"This means that the rules with
shadoware executed within$TMPDIR/shadow/and only the output files are copied to the actual output directory.mentioned in merge request !78 (merged)
mentioned in commit 6b6b2830
Fixed in !78 (merged)
