Bedtools not found while testing master branch

I was running the pipeline locally using the master branch and got an error message "tests/test_integration_workflow/test.local.sh: line 59: bedtools: command not found"

These are the steps:

git clone ssh://git@git.scicore.unibas.ch:2222/zavolan_group/pipelines/rhea.git
cd rhea/
conda activate rnaseq_pipeline
ml Singularity/2.6.1
nohup bash tests/test_integration_workflow/test.local.sh > integration_workflow_local.log

This is the execution log: integration_workflow_local.log

changed the description

added Bug label

changed milestone to %v0.1.0 release

assigned to @kanitz

I will have a look at this

I think bedtools is activated with pip install when testing, I am not sure we have it in the conda environment.

When you create the rnaseq_pipeline environment, which option did you use from the README?

conda env create -f install/environment.root.yml or conda env create -f install/environment.yml or conda env update -f install/environment.dev.yml?

it seems only the dev has the bedtools, so you should try creating this one.

Actually, it can be the case. I think, I used conda env create -f install/environment.yml. I will try it. If it is the case, don't we need to provide access to bedtools for all environments?

Currently, it is used only by the tests, so I do not think so, but @kanitz knows best.

@boersch: did you follow the instructions in README.md for installing the pipeline dependencies? I think it explains there what you have to do to (a) run the pipeline and (b) run tests. The CI pipeline seems to run fine anyway, including the test that requires bedtools. So I don't think there's really a bug anywhere, unless the docs are wrong.

I've just double checked and the installation instructions are quite clear:

• core dependencies on non-root computer
• additional dependencies for running tests

You need to do both (hence the update in the second call).

Closing this issue. Please reopen if following the instructions does not solve the problem.

closed

removed Bug label

@kanitz @katsanto I am sorry, it is my bad, I completely forgot about updating the environment to be able to run tests... So I removed the old rnaseq_pipeline environment and executed the following code:

ml Singularity/2.6.1
git clone ssh://git@git.scicore.unibas.ch:2222/zavolan_group/pipelines/rhea.git
cd rhea/
conda env create -f install/environment.yml
conda env update -f install/environment.dev.yml
conda activate rnaseq_pipeline
nohup sh tests/test_integration_workflow/test.local.sh >  nohup.local.txt

Since I don't have administrative rights on the login node, I did not install conda env update -f install/environment.dev.yml, right?

Then I get another error, which is a syntax error:

tests/test_integration_workflow/test.local.sh: command substitution: line 68: syntax error near unexpected token `('

This is the log file of the execution nohup.local.txt

I am sorry if I am annoying, I just want to make sure that the pipeline runs correctly.

I just ran the local test locally and it works..

• Are you sure the branch you are in is synchronised with the master branch?
• I do not understand why you need administrative rights to run conda env update -f install/environment.dev.yml, is it not like conda create? You need to run this to update the environment and have all the packages, otherwise you are left with whatever is in environment.yml

I just ran the following on the login node (no issues whatsoever):

git checkout master & git pull origin master  # equivalent to fresh cloning like in instructions
conda env create -f install/environment.yml --force  # --force overwrites existing environment
conda activate rnaseq_pipeline
conda env update -f install/environment.dev.yml
bash tests/test_integration_workflow/test.local.sh
bash tests/test_integration_workflow/test.slurm.sh

And Maria is right: you should definitely install environment.dev.yml if you want to run tests. Or have a look at what's in there and install it in any other way. Still, that probably shouldn't explain your syntax error - it's a bit weird...

@kanitz @katsanto I think, I finally have a reproducible example for both outcomes: a run that induces an error and a clean run.

Start with the new ssh session.

These are commands that induce an error.

conda remove --name rnaseq_pipeline --all # remove your existing conda environment
ml Singularity/2.6.1
mkdir test
cd test
git clone ssh://git@git.scicore.unibas.ch:2222/zavolan_group/pipelines/rhea.git
cd rhea/
conda env create -f install/environment.yml
conda activate rnaseq_pipeline
conda env update -f install/environment.dev.yml
nohup sh tests/test_integration_workflow/test.local.sh >  nohup.local.error.txt

These are commands that produce a clean run.

conda deactivate
conda env create -f install/environment.yml --force  # --force overwrites existing environment
conda activate rnaseq_pipeline
conda env update -f install/environment.dev.yml
nohup bash tests/test_integration_workflow/test.local.sh >  nohup.local.clean.txt

These are log files for both runs: nohup.local.clean.txt nohup.local.error.txt

It looks like something is wrong with setting up the conda environment in the first case, but I don't know what is exactly wrong. I have conda version 4.7.12. Any ideas, what is wrong with the first part?

Why do you use sh? I have never used it so far and it is the only difference I see..

It looks like it is indeed sh that makes the problem. I replaced sh with bash and the run was clean. I tried to read, what is the difference between sh and bash, but it is not so obvious. May be it is good to keep in mind my experience if some of pipeline users also use sh instead of bash and then report an error. @katsanto Thanks a lot for help!

Also the Singularity version is outdated. Current version in the login nodes is by default : 3.3.0-1.el7.