Skip to content
Snippets Groups Projects

MultiQC report: custom plugin for zpca

Merged MultiQC report: custom plugin for zpca
zpca-multiqc into dev
Merged BIOPZ-Bak Maciej requested to merge zpca-multiqc into dev

Addressing #146 (closed).

  • bumped-up multiqc-plugins container version in the Snakefile
  • specified parsing rules in the MultiQC configfile generating script

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • BIOPZ-Bak Maciej changed milestone to %downstream

    changed milestone to %downstream

  • BIOPZ-Bak Maciej added Doing downstream labels

    added Doing downstream labels

  • BIOPZ-Bak Maciej
    BIOPZ-Bak Maciej @bakma ·
    Author Maintainer

    It seems that the zpca plugin crashed on the integration test dataset (but it passed on GCN4). I will have to investigate deeper.

    Crash logs:

    [WARNING]         multiqc : MultiQC Version v1.10 now available!
[INFO   ]         multiqc : This is MultiQC v1.9
[INFO   ]         multiqc : Template    : default
[INFO   ]         multiqc : Report title: ZARP
[INFO   ]         multiqc : Searching   : /scicore/home/zavolan/bakma/zarp-dev/zarp/tests/test_integration_workflow/results
[INFO   ]         multiqc : Searching   : /scicore/home/zavolan/bakma/zarp-dev/zarp/tests/test_integration_workflow/logs
[INFO   ]          fastqc : Found 3 reports
[INFO   ]        cutadapt : Found 2 reports
[INFO   ]        cutadapt : Found 2 reports
[INFO   ]            star : Found 2 reports
[INFO   ]          salmon : Found 2 meta reports
[INFO   ]          salmon : Found 2 fragment length distributions
[INFO   ]        kallisto : Found 2 reports
[ERROR  ]         multiqc : Oops! The 'zpca' MultiQC module broke... 
  Please copy the following traceback and report it at https://github.com/ewels/MultiQC/issues 
  If possible, please include a log file that triggers the error - the last file found was:
    results/zpca/pca_salmon_genes/scree.tsv
============================================================
Module zpca raised an exception: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/multiqc/multiqc.py", line 569, in run
    output = mod()
  File "/usr/local/lib/python3.8/site-packages/multiqc_plugins-1.1.0-py3.8.egg/modules/zpca/zpca.py", line 45, in __init__
    self.findLogs()
  File "/usr/local/lib/python3.8/site-packages/multiqc_plugins-1.1.0-py3.8.egg/modules/zpca/zpca.py", line 54, in findLogs
    data_scree = self.parse_scree_logs(f)
  File "/usr/local/lib/python3.8/site-packages/multiqc_plugins-1.1.0-py3.8.egg/modules/zpca/zpca.py", line 124, in parse_scree_logs
    "PC1": {exp_car_str: float(listToStr[-3])},
ValueError: could not convert string to float: 'Percentage of Explained Variance'
============================================================
[ERROR  ]         multiqc : Oops! The 'zpca' MultiQC module broke... 
  Please copy the following traceback and report it at https://github.com/ewels/MultiQC/issues 
  If possible, please include a log file that triggers the error - the last file found was:
    results/zpca/pca_salmon_transcripts/scree.tsv
============================================================
Module zpca raised an exception: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/multiqc/multiqc.py", line 569, in run
    output = mod()
  File "/usr/local/lib/python3.8/site-packages/multiqc_plugins-1.1.0-py3.8.egg/modules/zpca/zpca.py", line 45, in __init__
    self.findLogs()
  File "/usr/local/lib/python3.8/site-packages/multiqc_plugins-1.1.0-py3.8.egg/modules/zpca/zpca.py", line 54, in findLogs
    data_scree = self.parse_scree_logs(f)
  File "/usr/local/lib/python3.8/site-packages/multiqc_plugins-1.1.0-py3.8.egg/modules/zpca/zpca.py", line 124, in parse_scree_logs
    "PC1": {exp_car_str: float(listToStr[-3])},
ValueError: could not convert string to float: 'Percentage of Explained Variance'
============================================================
[ERROR  ]         multiqc : Oops! The 'zpca' MultiQC module broke... 
  Please copy the following traceback and report it at https://github.com/ewels/MultiQC/issues 
  If possible, please include a log file that triggers the error - the last file found was:
    results/zpca/pca_kallisto_genes/scree.tsv
============================================================
Module zpca raised an exception: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/multiqc/multiqc.py", line 569, in run
    output = mod()
  File "/usr/local/lib/python3.8/site-packages/multiqc_plugins-1.1.0-py3.8.egg/modules/zpca/zpca.py", line 45, in __init__
    self.findLogs()
  File "/usr/local/lib/python3.8/site-packages/multiqc_plugins-1.1.0-py3.8.egg/modules/zpca/zpca.py", line 54, in findLogs
    data_scree = self.parse_scree_logs(f)
  File "/usr/local/lib/python3.8/site-packages/multiqc_plugins-1.1.0-py3.8.egg/modules/zpca/zpca.py", line 124, in parse_scree_logs
    "PC1": {exp_car_str: float(listToStr[-3])},
ValueError: could not convert string to float: 'Percentage of Explained Variance'
============================================================
[ERROR  ]         multiqc : Oops! The 'zpca' MultiQC module broke... 
  Please copy the following traceback and report it at https://github.com/ewels/MultiQC/issues 
  If possible, please include a log file that triggers the error - the last file found was:
    results/zpca/pca_kallisto_transcripts/scree.tsv
============================================================
Module zpca raised an exception: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/multiqc/multiqc.py", line 569, in run
    output = mod()
  File "/usr/local/lib/python3.8/site-packages/multiqc_plugins-1.1.0-py3.8.egg/modules/zpca/zpca.py", line 45, in __init__
    self.findLogs()
  File "/usr/local/lib/python3.8/site-packages/multiqc_plugins-1.1.0-py3.8.egg/modules/zpca/zpca.py", line 54, in findLogs
    data_scree = self.parse_scree_logs(f)
  File "/usr/local/lib/python3.8/site-packages/multiqc_plugins-1.1.0-py3.8.egg/modules/zpca/zpca.py", line 124, in parse_scree_logs
    "PC1": {exp_car_str: float(listToStr[-3])},
ValueError: could not convert string to float: 'Percentage of Explained Variance'
============================================================
[INFO   ]         multiqc : Compressing plot data
[INFO   ]         multiqc : Report      : results/multiqc_summary/multiqc_report.html
[INFO   ]         multiqc : Data        : results/multiqc_summary/multiqc_data
[INFO   ]         multiqc : MultiQC complete
  • BIOPZ-Bak Maciej
    BIOPZ-Bak Maciej @bakma ·
    Author Maintainer

    OK, the difference is in the scree.tsv.

    For the integration test we have:

    	PC1	PC2
Percentage of Explained Variance	100.0	0.0

    For the GCN4 we have:

    	PC1	PC2	PC3
Percentage of Explained Variance	79.7	12.1	8.2

    The error is that Krish hardcoded float(listToStr[-3]), which can be easily changed to float(listToStr[0]).

  • BIOPZ-Bak Maciej
    BIOPZ-Bak Maciej @bakma ·
    Author Maintainer

    Actually, as I look into our plugins repo the parsing script is constructed such that it always plots 1x2, 2x3, 1x3. The logic of first 3 principal components is hard-coded into the whole parsing and plotting script.

    @gypas
    Is there a reason why does our integration test returns only 2 principal components with variances: 100%, 0%?
    To me this seems like an artificial special case which breaks everything downstream.

  • BIOPZ-Bak Maciej
    BIOPZ-Bak Maciej @bakma ·
    Author Maintainer

    OK, digging even deeper into the core script in zpca I see that there is a function which determines the number of PCs based on the number of samples. The observations make sense now as in the integration test there are 2 samples going into the pipeline, so we get PC1 and PC2. The 100-0 split makes sense too..

    However, the problem is that this behaviour is cryptic, right? I am assuming that the official zpca repo does not test such case and therefore Krish assumed that it's always gonna be first 3 PCs. @gypas Could you please add proper tests to your repo? Then we can re-open the issue at multiqc-plugins and ask Krish for a fix. Having that I will plug it into the Snakefile here and push once again. Then it should be fine.

  • BIOPZ-Gypas Foivos mentioned in merge request !81 (merged)

    mentioned in merge request !81 (merged)

  • BIOPZ-Gypas Foivos
    BIOPZ-Gypas Foivos @gypas ·
    Maintainer

    Sure, I will have a look at this and come back to you.

  • BIOPZ-Bak Maciej added 4 commits

    added 4 commits

    Compare with previous version

  • BIOPZ-Bak Maciej added 1 commit

    added 1 commit

    Compare with previous version

  • BIOPZ-Bak Maciej
    BIOPZ-Bak Maciej @bakma ·
    Author Maintainer

    OK, It seems that all the errors are fixed;
    @gypas, before merging - could you please test locally and see if the zpca section in the report looks OK?

    Edited by BIOPZ-Bak Maciej
  • BIOPZ-Bak Maciej
    BIOPZ-Bak Maciej @bakma ·
    Author Maintainer

    The CI passes here but strangely enough the integration test fails on my machine. It seems that zpca created wrong scree.tsv for salmon-based quantification in my run.

    For kallisto I have:

    	PC1	PC2
Percentage of Explained Variance	100.0	0.0

    For salmon I have:

    	PC1	PC2
Percentage of Explained Variance

    CC @gypas

  • BIOPZ-Gypas Foivos added 1 commit

    added 1 commit

    • 27811448 - Bump zpca version in Snakefile

    Compare with previous version

  • BIOPZ-Gypas Foivos added 1 commit

    added 1 commit

    • 83cd7d3b - Bump zpca version in Snakefile

    Compare with previous version

  • BIOPZ-Bak Maciej mentioned in issue #159 (closed)

    mentioned in issue #159 (closed)

  • BIOPZ-Gypas Foivos added 3 commits

    added 3 commits

    Compare with previous version

  • BIOPZ-Gypas Foivos
    BIOPZ-Gypas Foivos @gypas ·
    Maintainer

    Hi @bakma. After merging with the latest changes from dev I cannot reproduce the issue anymore (neither in the CI nor locally). Can you please try as well after pulling the changes of the current branch?

    Edited by BIOPZ-Gypas Foivos
  • BIOPZ-Gypas Foivos requested review from @bakma

    requested review from @bakma

  • BIOPZ-Gypas Foivos removed review request for @bakma

    removed review request for @bakma

  • BIOPZ-Bak Maciej
    BIOPZ-Bak Maciej @bakma ·
    Author Maintainer

    Hi @gypas,
    I run bash tests/test_integration_workflow/test.local.sh locally and confirm that the problem is gone.
    Go ahead and merge away! :)

  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
Please register or sign in to reply