MultiQC plugins for TIN scores and ALFA

resolved all threads

When testing locally (tests/test_integration_workflow/test.local.sh) I get the following error in the multiqc job.

Error in rule multiqc_report:
    jobid: 1
    output: results/multiqc_summary
    log: logs/multiqc_report.stderr.log, logs/multiqc_report.stdout.log (check log file(s) for error message)
    shell:
        (multiqc         --outdir results/multiqc_summary         --config results/multiqc_config.yaml         results         logs;)         1> logs/multiqc_report.stdout.log 2> logs/multiqc_report.stderr.log
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Full Traceback (most recent call last):
  File "/scicore/home/zavolan/gypas/miniconda/envs/zarp/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2189, in run_wrapper
    edit_notebook,
  File "/scicore/home/zavolan/gypas/projects/zarp/Snakefile", line 1887, in __rule_multiqc_report
  File "/scicore/home/zavolan/gypas/miniconda/envs/zarp/lib/python3.7/site-packages/snakemake/shell.py", line 203, in __new__
    raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command ' singularity exec --home /scicore/home/zavolan/gypas/projects/zarp/tests/test_integration_workflow --bind /scicore/home/zavolan/gypas/projects/zarp/tests/test_integration_workflow/../input_files,/scicore/home/zavolan/gypas/projects/zarp/tests/test_integration_workflow/../../images --bind /scicore/home/zavolan/gypas/miniconda/envs/zarp/lib/python3.7/site-packages:/mnt/snakemake /scicore/home/zavolan/gypas/projects/zarp/tests/test_integration_workflow/.snakemake/singularity/725e42e6bc54e349bfb0893cb0d73c6d.simg bash -c 'set -euo pipefail;  (multiqc         --outdir results/multiqc_summary         --config results/multiqc_config.yaml         results         logs;)         1> logs/multiqc_report.stdout.log 2> logs/multiqc_report.stderr.log'' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/scicore/home/zavolan/gypas/miniconda/envs/zarp/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 529, in _callback
    raise ex
  File "/scicore/home/zavolan/gypas/miniconda/envs/zarp/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/scicore/home/zavolan/gypas/miniconda/envs/zarp/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 515, in cached_or_run
    run_func(*args)
  File "/scicore/home/zavolan/gypas/miniconda/envs/zarp/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2201, in run_wrapper
    ex, lineno, linemaps=linemaps, snakefile=file, show_traceback=True
snakemake.exceptions.RuleException: CalledProcessError in line 1363 of /scicore/home/zavolan/gypas/projects/zarp/Snakefile:
Command ' singularity exec --home /scicore/home/zavolan/gypas/projects/zarp/tests/test_integration_workflow --bind /scicore/home/zavolan/gypas/projects/zarp/tests/test_integration_workflow/../input_files,/scicore/home/zavolan/gypas/projects/zarp/tests/test_integration_workflow/../../images --bind /scicore/home/zavolan/gypas/miniconda/envs/zarp/lib/python3.7/site-packages:/mnt/snakemake /scicore/home/zavolan/gypas/projects/zarp/tests/test_integration_workflow/.snakemake/singularity/725e42e6bc54e349bfb0893cb0d73c6d.simg bash -c 'set -euo pipefail;  (multiqc         --outdir results/multiqc_summary         --config results/multiqc_config.yaml         results         logs;)         1> logs/multiqc_report.stdout.log 2> logs/multiqc_report.stderr.log'' returned non-zero exit status 1.
  File "/scicore/home/zavolan/gypas/miniconda/envs/zarp/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2189, in run_wrapper
  File "/scicore/home/zavolan/gypas/projects/zarp/Snakefile", line 1363, in __rule_multiqc_report

RuleException:
CalledProcessError in line 1363 of /scicore/home/zavolan/gypas/projects/zarp/Snakefile:
Command ' singularity exec --home /scicore/home/zavolan/gypas/projects/zarp/tests/test_integration_workflow --bind /scicore/home/zavolan/gypas/projects/zarp/tests/test_integration_workflow/../input_files,/scicore/home/zavolan/gypas/projects/zarp/tests/test_integration_workflow/../../images --bind /scicore/home/zavolan/gypas/miniconda/envs/zarp/lib/python3.7/site-packages:/mnt/snakemake /scicore/home/zavolan/gypas/projects/zarp/tests/test_integration_workflow/.snakemake/singularity/725e42e6bc54e349bfb0893cb0d73c6d.simg bash -c 'set -euo pipefail;  (multiqc         --outdir results/multiqc_summary         --config results/multiqc_config.yaml         results         logs;)         1> logs/multiqc_report.stdout.log 2> logs/multiqc_report.stderr.log'' returned non-zero exit status 1.
  File "/scicore/home/zavolan/gypas/miniconda/envs/zarp/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2189, in run_wrapper
  File "/scicore/home/zavolan/gypas/projects/zarp/Snakefile", line 1363, in __rule_multiqc_report
  File "/scicore/home/zavolan/gypas/miniconda/envs/zarp/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 529, in _callback
  File "/scicore/home/zavolan/gypas/miniconda/envs/zarp/lib/python3.7/concurrent/futures/thread.py", line 57, in run
  File "/scicore/home/zavolan/gypas/miniconda/envs/zarp/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 515, in cached_or_run
  File "/scicore/home/zavolan/gypas/miniconda/envs/zarp/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2201, in run_wrapper

The log message (stderr) of the job is the following:

Traceback (most recent call last):
  File "/usr/local/bin/multiqc", line 11, in <module>
    load_entry_point('multiqc==1.9', 'console_scripts', 'multiqc')()
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 484, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2714, in load_entry_point
    return ep.load()
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2332, in load
    return self.resolve()
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2338, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python3.6/site-packages/multiqc/__main__.py", line 44, in <module>
    multiqc.run_cli(prog_name='multiqc')
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 760, in main
    _verify_python3_env()
  File "/usr/local/lib/python3.6/site-packages/click/_unicodefun.py", line 130, in _verify_python3_env
    " mitigation steps.{}".format(extra)
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Consult https://click.palletsprojects.com/python3/ for mitigation steps.

This system supports the C.UTF-8 locale which is recommended. You might be able to resolve your issue by exporting the following environment variables:

    export LC_ALL=C.UTF-8
    export LANG=C.UTF-8

Click discovered that you exported a UTF-8 locale but the locale system could not pick up from it because it does not exist. The exported locale is 'en_GB.utf8' but it is not supported

Any ideas? To me, it seems that it is related to the new docker image (https://github.com/zavolanlab/Dockerfiles/blob/master/MultiQC/1.9/Dockerfile), but I am not sure.

Can someone else try to run the test locally?

This looks like an issue specific to the settings of our cluster?
These env variables should have been set properly, take a look here:
https://stackoverflow.com/questions/36651680/click-will-abort-further-execution-because-python-3-was-configured-to-use-ascii

Could you try to substitute the bash command to (for testing):

export LC_ALL=C.UTF-8 && export LANG=C.UTF-8 && multiqc \
.
.
.

But how did it work in the CI, though...

So, this seems to work. What I don't understand is (1) why we get it only for the following rule and (2) why we don't get it on the CI. @kanitz Any opinion on this?

Maybe the locale is set on the VM running the tests. You could add echo $LC_ALL and echo $LANG to the CI config and see what it prints. And you could do the same in a new shell on your computer. If there's a difference, it's your computer's setup. I don't think this has anything to do with the cluster? (but not sure)

Thanks a lot @kanitz. So here are the settings for the VM:

$LC_ALL is C.UTF-8
$LANG is C.UTF-8

When I set my ~/.bashrc to the following:

export LC_ALL=C.UTF-8
export LANG=C.UTF-8

The pipeline works fine, but I still get the following warning (not error anymore):

/usr/bin/bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)

@bakma Can you please check if you also get the same warning when you run the pipeline locally? Thank you in advance.

@gypas
I am working on our login node; In my case when I added

export LC_ALL=C.UTF-8
export LANG=C.UTF-8

to my bashrc and sourced it afterwards I got an error:

-bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8): No such file or directory
-bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)

Update on that: it is the first command that causes problems, so I have just executed export LANG=C.UTF-8 in the terminal and then called the pipeline on our node. Finished with no error.
However, during the pipeline execution, in the middle of snakemake logs I also got:

/scicore/home/zavolan/bakma/miniconda3/envs/mapp/bin/bash: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)

Ok, so we have the same issue. Should we mention this to scicore or is it related to our setup? I am a bit confused. Anyone else having such an issue in other snakemake pipelines?

Please take a look at the link I posted in my first reply here - the issue raises whenever we use specific Python3 libraries in an environment where the interpreter thinks that Python is restricted to ASCII (thus it shouts to us for UTF-8 support).

I think this is a problem specific to our cluster, but can also happen for other scientists on other clusters - who knows<?>

What I would propose is to add export LANG=C.UTF-8 in the Dockerfile for our multiqc+plugins such that we make sure the shell variable is always set properly, regardless of the user's machine.

Ok, so I propose the following:

@bakma merges https://github.com/zavolanlab/multiqc-plugins/pull/14
@gypas opens a new PR with the updated Dockerfile in zavolanlab/multiqc-plugins and @bakma merges it
@bakma updates the Dockerfile in the current pipeline and @gypas tests it
@gypas merges the current PR

OK, then at the very first step @gypas has to respond to the PR comment.

I cannot tick the boxes in your post, but (1) is done now.

Thanks. I selected it for you.

added Blocked label

mentioned in merge request !78 (merged)

added 1 commit

e0553269 - add new docker image

Compare with previous version

added 1 commit

3368b0e3 - ci: turn off labkey test

Compare with previous version

removed Blocked label

approved this merge request

merged

mentioned in commit 736d1a3d

mentioned in issue #138 (closed)

MultiQC plugins for TIN scores and ALFA

Merge request reports

Activity