Fix a couple of bottlenecks in the pipeline #159

nvnieuwk · 2025-01-13T12:56:53Z

This PR fixes a couple of bottlenecks I noticed in this pipeline

nf-core-bot · 2025-01-13T12:57:32Z

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 3.0.2.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation.

github-actions · 2025-01-13T12:59:19Z

`nf-core pipelines lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 4035a18

+| ✅ 225 tests passed       |+
!| ❗   4 tests had warnings |!

Details

❗ Test warnings:

pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
pipeline_todos - TODO string in base.config: Check the defaults for all processes

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-rnavar_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-rnavar_logo_light.png
files_exist - File found: docs/images/nf-core-rnavar_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: conf/igenomes_ignored.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-rnavar_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowRnavar.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Found nf-schema plugin
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: validation.help.enabled
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable found: validation.help.beforeText
nextflow_config - Config variable found: validation.help.afterText
nextflow_config - Config variable found: validation.help.command
nextflow_config - Config variable found: validation.summary.beforeText
nextflow_config - Config variable found: validation.summary.afterText
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config variable (correctly) not found: params.max_cpus
nextflow_config - Config variable (correctly) not found: params.max_memory
nextflow_config - Config variable (correctly) not found: params.max_time
nextflow_config - Config variable (correctly) not found: params.validationFailUnrecognisedParams
nextflow_config - Config variable (correctly) not found: params.validationLenientMode
nextflow_config - Config variable (correctly) not found: params.validationSchemaIgnoreParams
nextflow_config - Config variable (correctly) not found: params.validationShowHiddenParams
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 2.0.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.read_length= 150.0
nextflow_config - Config default value correct: params.igenomes_base= s3://ngi-igenomes/igenomes/
nextflow_config - Config default value correct: params.aligner= star
nextflow_config - Config default value correct: params.star_max_memory_bamsort= 0
nextflow_config - Config default value correct: params.star_bins_bamsort= 50
nextflow_config - Config default value correct: params.star_max_collapsed_junc= 1000000
nextflow_config - Config default value correct: params.seq_platform= illumina
nextflow_config - Config default value correct: params.gatk_hc_call_conf= 20.0
nextflow_config - Config default value correct: params.vep_cache= s3://annotation-cache/vep_cache/
nextflow_config - Config default value correct: params.snpeff_cache= s3://annotation-cache/snpeff_cache/
nextflow_config - Config default value correct: params.dbnsfp_fields= rs_dbSNP,HGVSc_VEP,HGVSp_VEP,1000Gp3_EAS_AF,1000Gp3_AMR_AF,LRT_score,GERP++_RS,gnomAD_exomes_AF
nextflow_config - Config default value correct: params.vep_custom_args= --everything --filter_common --per_gene --total_length --offline --format vcf
nextflow_config - Config default value correct: params.vep_out_format= vcf
nextflow_config - Config default value correct: params.gatk_interval_scatter_count= 25
nextflow_config - Config default value correct: params.gatk_vf_window_size= 35
nextflow_config - Config default value correct: params.gatk_vf_cluster_size= 3
nextflow_config - Config default value correct: params.gatk_vf_fs_filter= 30.0
nextflow_config - Config default value correct: params.gatk_vf_qd_filter= 2.0
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
nextflow_config - Config default value correct: params.modules_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/modules/data
nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/modules/data
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-rnavar_logo_light.png matches the template
files_unchanged - docs/images/nf-core-rnavar_logo_light.png matches the template
files_unchanged - docs/images/nf-core-rnavar_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 24.04.2, Config: 24.04.2
readme - README Zenodo placeholder was replaced with DOI.
plugin_includes - No wrong validation plugin imports have been found
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (0 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: template_version_comment.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: ci.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains a matching 'report_comment'.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
modules_config - conf/modules.config found and not ignored.
modules_config - CAT_FASTQ found in conf/modules.config and Nextflow scripts.
modules_config - STAR_GENOMEGENERATE found in conf/modules.config and Nextflow scripts.
modules_config - STAR_GENOMEGENERATE found in conf/modules.config and Nextflow scripts.
modules_config - GFFREAD found in conf/modules.config and Nextflow scripts.
modules_config - GTF2BED found in conf/modules.config and Nextflow scripts.
modules_config - SAMTOOLS_FAIDX found in conf/modules.config and Nextflow scripts.
modules_config - GATK4_CREATESEQUENCEDICTIONARY found in conf/modules.config and Nextflow scripts.
modules_config - MULTIQC found in conf/modules.config and Nextflow scripts.
modules_config - FASTQC found in conf/modules.config and Nextflow scripts.
modules_config - STAR_ALIGN found in conf/modules.config and Nextflow scripts.
modules_config - GATK4_BEDTOINTERVALLIST found in conf/modules.config and Nextflow scripts.
modules_config - GATK4_INTERVALLISTTOOLS found in conf/modules.config and Nextflow scripts.
modules_config - PICARD_MARKDUPLICATES found in conf/modules.config and Nextflow scripts.
modules_config - SAMTOOLS_STATS found in conf/modules.config and Nextflow scripts.
modules_config - GATK4_BASERECALIBRATOR found in conf/modules.config and Nextflow scripts.
modules_config - GATK4_HAPLOTYPECALLER found in conf/modules.config and Nextflow scripts.
modules_config - GATK4_MERGEVCFS found in conf/modules.config and Nextflow scripts.
modules_config - GATK4_COMBINEGVCFS found in conf/modules.config and Nextflow scripts.
modules_config - TABIX_TABIX found in conf/modules.config and Nextflow scripts.
modules_config - GATK4_VARIANTFILTRATION found in conf/modules.config and Nextflow scripts.
modules_config - NFCORE_RNAVAR found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 3.0.2

Run details

nf-core/tools version 3.0.2
Run at 2025-01-15 08:23:26

subworkflows/local/splitncigar/main.nf

Co-authored-by: Maxime U Garcia <[email protected]>

subworkflows/local/splitncigar/main.nf

workflows/rnavar/main.nf

Co-authored-by: Maxime U Garcia <[email protected]>

nvnieuwk added 3 commits January 13, 2025 11:40

fix join bottlenecks

3addcd3

fix a groupTuple bottleneck

ec57401

fix splitcigarreads groupTuple bottleneck

ad1f1bc

nvnieuwk added 2 commits January 13, 2025 13:57

update changelog

8037ce7

make editorconfig happy

0d2bb33

nvnieuwk mentioned this pull request Jan 13, 2025

SplitNCigarReads: sample cross-contamination #119

Closed

maxulysse reviewed Jan 13, 2025

View reviewed changes

subworkflows/local/splitncigar/main.nf Outdated Show resolved Hide resolved

maxulysse reviewed Jan 13, 2025

View reviewed changes

subworkflows/local/splitncigar/main.nf Outdated Show resolved Hide resolved

nvnieuwk and others added 2 commits January 14, 2025 16:35

Update subworkflows/local/splitncigar/main.nf

8ef0e65

Co-authored-by: Maxime U Garcia <[email protected]>

Update subworkflows/local/splitncigar/main.nf

d477c21

Co-authored-by: Maxime U Garcia <[email protected]>

maxulysse reviewed Jan 14, 2025

View reviewed changes

subworkflows/local/splitncigar/main.nf Outdated Show resolved Hide resolved

maxulysse reviewed Jan 14, 2025

View reviewed changes

workflows/rnavar/main.nf Outdated Show resolved Hide resolved

maxulysse approved these changes Jan 14, 2025

View reviewed changes

nvnieuwk and others added 2 commits January 15, 2025 09:19

Update workflows/rnavar/main.nf

0c18809

Co-authored-by: Maxime U Garcia <[email protected]>

Update subworkflows/local/splitncigar/main.nf

4035a18

Co-authored-by: Maxime U Garcia <[email protected]>

nvnieuwk merged commit bfd3665 into nf-core:dev Jan 15, 2025
11 checks passed

nvnieuwk deleted the fix/bottlenecks branch January 15, 2025 09:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix a couple of bottlenecks in the pipeline #159

Fix a couple of bottlenecks in the pipeline #159

Uh oh!

nvnieuwk commented Jan 13, 2025

Uh oh!

nf-core-bot commented Jan 13, 2025

Uh oh!

github-actions bot commented Jan 13, 2025 •

edited

Loading

❗ Test warnings:

✅ Tests passed:

Run details

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Fix a couple of bottlenecks in the pipeline #159

Fix a couple of bottlenecks in the pipeline #159

Uh oh!

Conversation

nvnieuwk commented Jan 13, 2025

Uh oh!

nf-core-bot commented Jan 13, 2025

Uh oh!

github-actions bot commented Jan 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

nf-core pipelines lint overall result: Passed ✅ ⚠️

❗ Test warnings:

✅ Tests passed:

Run details

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions bot commented Jan 13, 2025 •

edited

Loading

`nf-core pipelines lint` overall result: Passed ✅ ⚠️