Error in prepare_files_for_report with single end only

I ran some samples, all single end, and got this error:

FileNotFoundError in line 1251 of /scicore/home/zavolan/burri0000/zavolab_projects/rnaseqpipeline/Snakefile:
[Errno 2] No such file or directory: 'results/paired_end'
  File "/scicore/home/zavolan/burri0000/zavolab_projects/rnaseqpipeline/Snakefile", line 1251, in __rule_prepare_files_for_report
  File "/scicore/home/zavolan/burri0000/miniconda3/envs/rnaseq_pipeline/lib/python3.6/shutil.py", line 471, in rmtree
  File "/scicore/home/zavolan/burri0000/miniconda3/envs/rnaseq_pipeline/lib/python3.6/shutil.py", line 469, in rmtree
  File "/scicore/home/zavolan/burri0000/miniconda3/envs/rnaseq_pipeline/lib/python3.6/concurrent/futures/thread.py", line 56, in run

the error seems to come from line 1251 in Snakefile. The snippet shows the complete call:

shutil.rmtree(
            os.path.join(
                params.results_dir,
                "paired_end"),
            ignore_errors=False,
            onerror=None)

There is no directory paired_end. This shutil.rmtree() call should be expanded by the possibility that the directory does not exist or something similar. Similarly for the directory single_end.

added Bug label

I think this refers to the old repo structure, in the new one these directories change (and also we do not need prepare_files_for_report)

But nevertheless, Me and Anastasia did not notice this issue while running muscle data. It is highly possible to arise due to re-running the pipeline from the middle. Could you please try to make sure you are on the new commit on the master branch, clean all output and log files and run from top to bottom?

Well I pulled the latest master commit before running things. My log shows commit ca049a0c as latest.

Or is it not in the master branch yet?

Actually, I have never run the pipeline for single-end samples only.

All the test that we have ever did have samples annotated as both single and paired. Is that correct? In test_integration_test we have a samples table with mixed samples, thats for sure... If I recall correctly muscle data were also mixed(?) Then it is a bug that the rule always assumes samples of both types.

I am currently re-running the previous version before the global re-structuring with one single-end sample.

There is a potential bug in the prepare_files_for_report - the rule does not check if a directory exists but tries to remove it anyway.

Since this rule will be deleted in the next version anyway and the structure will change I propose the following hotfix:

change:

shutil.rmtree(
            os.path.join(
                params.results_dir,
                "paired_end"),
            ignore_errors=False,
            onerror=None)

to:

shutil.rmtree(
            os.path.join(
                params.results_dir,
                "paired_end"),
            ignore_errors=True,
            onerror=None)

I am not sure if I should make a separate branch and introduce this bugfix to the old structure now or should we all wait for the new Rhea version with the major refactoring. @katsanto @kanitz ?

I can include your proposed hot fix and finish running the samples. Once the major refactoring is out, I can re-run all samples.

Yes, it is an error in prepare_files_for_report.

[Tue Apr 14 14:23:27 2020]
localrule prepare_files_for_report:
    input: /scicore/home/zavolan/GROUP/TESTS/rhea_tests/rhea_muscle_samples_test/test_run/results/single_end/9mEDL_WT_rep1/mate1_fastqc, /scicore/home/zavolan/GROUP/TESTS
    output: /scicore/home/zavolan/GROUP/TESTS/rhea_tests/rhea_muscle_samples_test/test_run/results/samples, /scicore/home/zavolan/GROUP/TESTS/rhea_tests/rhea_muscle_sampl
    log: /scicore/home/zavolan/GROUP/TESTS/rhea_tests/rhea_muscle_samples_test/test_run/logs/prepare_files_for_report.stderr.log, /scicore/home/zavolan/GROUP/TESTS/rhea_t
    jobid: 3

ESC[33mJob counts:
        count   jobs
        1       prepare_files_for_report
        1ESC[0m
ESC[32m[Tue Apr 14 14:23:52 2020]ESC[0m
ESC[31mError in rule prepare_files_for_report:ESC[0m
ESC[31m    jobid: 0ESC[0m
ESC[31m    output: /scicore/home/zavolan/GROUP/TESTS/rhea_tests/rhea_muscle_samples_test/test_run/results/samples, /scicore/home/zavolan/GROUP/TESTS/rhea_tests/rhea_muscl
ESC[31m    log: /scicore/home/zavolan/GROUP/TESTS/rhea_tests/rhea_muscle_samples_test/test_run/logs/prepare_files_for_report.stderr.log, /scicore/home/zavolan/GROUP/TESTS
ESC[31mESC[0m
ESC[31mRuleException:
FileNotFoundError in line 1163 of /scicore/home/zavolan/GROUP/TESTS/rhea_tests/rhea_muscle_samples_test/rhea/Snakefile:
[Errno 2] No such file or directory: '/scicore/home/zavolan/GROUP/TESTS/rhea_tests/rhea_muscle_samples_test/test_run/results/paired_end'
  File "/scicore/home/zavolan/GROUP/TESTS/rhea_tests/rhea_muscle_samples_test/rhea/Snakefile", line 1163, in __rule_prepare_files_for_report
  File "/scicore/home/zavolan/boersch/Software/miniconda3/envs/rhea/lib/python3.7/shutil.py", line 485, in rmtree
  File "/scicore/home/zavolan/boersch/Software/miniconda3/envs/rhea/lib/python3.7/shutil.py", line 483, in rmtree
  File "/scicore/home/zavolan/boersch/Software/miniconda3/envs/rhea/lib/python3.7/concurrent/futures/thread.py", line 57, in runESC[0m
ESC[31mExiting because a job execution failed. Look above for error messageESC[0m
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /scicore/home/zavolan/GROUP/TESTS/rhea_tests/rhea_muscle_samples_test/test_run/.snakemake/log/2020-04-14T133908.022162.snakemake.log

The same error occurs on line 1258 for the log directory.

The latest master version does not contain many of the significant changes made in the last weeks.

Perhaps you could pull the branch we havent pushed to the master. The tests are passing currently and I do not think it is worth spending time fixing things that will be removed. My opinion.

Yes, I could also try that but I run now the fix Maciek proposed as it is a very small fix.

Are there tests with only single_end or only paired_end?

Agree with merging the refactoring branch and then trying again. I will take care of that in the next couple of days, got sidetracked with the other hackathon. Let's see what that does about the bug.

As for single_end and paired_end tests: I originally ran a paired-end sample we had first as a paired-end sample and then each strand separately as two single-end samples. But things were changed about that test, so I'm not sure anymore.

Apparently the main integration tests runs one paired-end and one single-end sample, but no tests runs only paired- or only single-end samples: https://git.scicore.unibas.ch/zavolan_group/pipelines/rhea/-/blob/master/tests/input_files/samples.tsv

changed milestone to %v0.2.0 release

assigned to @bakma

unassigned @bakma

This is partly fixed, but the issue remains that some results refer to the mate and some to the sample as a whole. For now these are separate rows in the multiqc. Do you think this is the best we can do @bakma @kanitz @zavolan? Should I close this?

added Discuss label

added To Do label

There may be statistics that should be done at sample or at file (mate) level. Let's revisit this after sorting out the MultiQC report.

added Won't fix label

It is fine as we have this now.

closed