Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Add gtdbtk/classifywf module #765

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
drpatelh merged 28 commits into from
Oct 5, 2021
Merged

Add gtdbtk/classifywf module #765

drpatelh merged 28 commits into from
Oct 5, 2021

Conversation

@abhi18av
Copy link
Member

@abhi18av abhi18av commented Sep 29, 2021 •
edited
Loading

PR checklist

Closes #XXX

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the module conventions in the contribution docs
  • If necessary, include test data in your PR.
  • Remove all TODO statements.
  • Emit the versions.yml file.
  • Follow the naming conventions.
  • Follow the parameters requirements.
  • Follow the input/output options guidelines.
  • Add a resource label
  • Use BioConda and BioContainers if possible to fulfil software requirements.
  • Ensure that the test works with either Docker / Singularity. Conda CI tests can be quite flaky:
    • PROFILE=docker pytest --tag <MODULE> --symlink --keep-workflow-wd
    • PROFILE=singularity pytest --tag <MODULE> --symlink --keep-workflow-wd
    • PROFILE=conda pytest --tag <MODULE> --symlink --keep-workflow-wd

@abhi18av abhi18av self-assigned this Sep 29, 2021
@abhi18av abhi18av mentioned this pull request Sep 30, 2021
Closed
80 tasks
@abhi18av abhi18av marked this pull request as ready for review October 1, 2021 06:43
@abhi18av abhi18av requested review from rpetit3 and skrakau October 1, 2021 06:43
abhi18av
abhi18av commented Oct 1, 2021
View reviewed changes
Copy link
Member Author

@abhi18av abhi18av left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added my comments for the reviewer - thanks for your time 🙏

For some context: This module is targeted towards Bactopia v2 and hopefully it could be consumed by nf-core/mag as well.

@abhi18av abhi18av added the new module Adding a new module label Oct 1, 2021
@abhi18av
Copy link
Member Author

abhi18av commented Oct 2, 2021 •
edited
Loading

I've added the stubs for generating a database and the core test is passing for docker/conda/singularity ✅ , but there is one failing test with an odd error 🤔

I am not sure which format exactly the ci-tests are expecting for tags in the test.yml file.

    def test_ensure_valid_version_yml(workflow_dir):
        workflow_dir = Path(workflow_dir)
        software_name = workflow_dir.name.split("_")[0].lower()
        try:
            versions_yml_file = workflow_dir / f"output/{software_name}/versions.yml"
            versions_yml = versions_yml_file.read_text()
        except FileNotFoundError:
>           raise AssertionError(
                dedent(
                    f"""\
                    `versions.yml` not found in the output directory.
                    Expected path: `{versions_yml_file}`
    
                    This can have multiple reasons:
                    * The test-workflow failed before a `versions.yml` could be generated.
                    * The workflow name in `test.yml` does not start with the tool name.
                    """
                )
            )
E           AssertionError: `versions.yml` not found in the output directory.
E           Expected path: `/home/runner/pytest_workflow_f71wgx8f/gtdbtk_classifywf/output/gtdbtk/versions.yml`
E           
E           This can have multiple reasons:
E           * The test-workflow failed before a `versions.yml` could be generated.
E           * The workflow name in `test.yml` does not start with the tool name.

tests/test_versions_yml.py:29: AssertionError

@abhi18av abhi18av mentioned this pull request Oct 2, 2021
Merged
14 tasks
@rpetit3
Copy link
Contributor

rpetit3 commented Oct 2, 2021

SUper close, it'll need to be updated for the new changes in software versions

https://github.com/nf-core/modules/blob/master/modules/abacas/main.nf#L42-L45

@abhi18av
Copy link
Member Author

abhi18av commented Oct 2, 2021 •
edited
Loading

Hmm, this is an interesting change - thanks @rpetit3 , I need to hang around the #modules channel more :)

However, for gtdbtk this is a bit problematic, since gtdktk --version doesn't work without the presence of the database and throws the following error

GTDBTK_CLASSIFYWF:
    gtdbtk: ================================================================================ ERROR ________________________________________________________________________________ The 'GTDBTK_DATA_PATH' environment variable is not defined. Please set this variable to your reference data package. https://github.com/Ecogenomics/GTDBTk#installation ================================================================================

... which not being a formatted YAML, raises issue with the yaml file validation 😶

E           AssertionError: Version number for gtdbtk must start with a number. 
E           assert None
E            +  where None = <function match at 0x7f86fb3bfa60>('^\\d+.*', '================================================================================ ERROR ______________________________...b.com/Ecogenomics/GTDBTk#installation ================================================================================')
E            +    where <function match at 0x7f86fb3bfa60> = re.match
E            +    and   '================================================================================ ERROR ______________________________...b.com/Ecogenomics/GTDBTk#installation ================================================================================' = str('================================================================================ ERROR ______________________________...b.com/Ecogenomics/GTDBTk#installation ================================================================================')

The plot thickens 🕵️

@abhi18av
Copy link
Member Author

abhi18av commented Oct 4, 2021

Hmm, I'm a bit confused here. Now that I've fixed the problem while printing the version for gtdb in the stubs, the nf-core linting throws the following error and the conda build is failing

I'm using nf-core, version 2.1 - should I switch to dev based package to keep these from failing?

╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ [!] 2 Tests Failed                                                                                                                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭──────────────────────────────────────────┬────────────────────────────────────────┬──────────────────────────────────────────────────╮
│ Module name                              │ File path                              │ Test message                                     │
├──────────────────────────────────────────┼────────────────────────────────────────┼──────────────────────────────────────────────────┤
│ gtdbtk/classifywf                        │ modules/gtdbtk/classifywf/main.nf      │ 'publish_by_meta:['id']' unspecified in saveAs   │
│                                          │                                        │ function                                         │
│ gtdbtk/classifywf                        │ modules/gtdbtk/classifywf/functions.nf │ New version of functions.nf available            │
╰──────────────────────────────────────────┴────────────────────────────────────────┴──────────────────────────────────────────────────╯

drpatelh
drpatelh reviewed Oct 4, 2021
View reviewed changes
@drpatelh
Copy link
Member

drpatelh commented Oct 4, 2021 •
edited
Loading

Weird about the functions.nf 🤔 Looks ok but maybe there is a diff there I am not seeing.

Could you try to replace the one on the PR with this one.

And apply the suggested change above?

And yep. Defo worth re-installing the dev version of tools again.

@abhi18av
Copy link
Member Author

abhi18av commented Oct 4, 2021

Could you try to replace the one on the PR with this one.

Done 👍

And apply the suggested change above?

Done 👍

The problem with conda still persists 🤔

rpetit3
rpetit3 requested changes Oct 4, 2021
View reviewed changes
@abhi18av abhi18av removed the help wanted Extra attention is needed label Oct 5, 2021
@abhi18av abhi18av requested a review from drpatelh October 5, 2021 07:42
drpatelh
drpatelh reviewed Oct 5, 2021
View reviewed changes
drpatelh
drpatelh reviewed Oct 5, 2021
View reviewed changes
drpatelh
drpatelh reviewed Oct 5, 2021
View reviewed changes
drpatelh
drpatelh reviewed Oct 5, 2021
View reviewed changes
drpatelh
drpatelh reviewed Oct 5, 2021
View reviewed changes
drpatelh
drpatelh reviewed Oct 5, 2021
View reviewed changes
drpatelh
drpatelh reviewed Oct 5, 2021
View reviewed changes
@drpatelh
Copy link
Member

drpatelh commented Oct 5, 2021

Thanks guys! This is a good start and can be amended later if required.

drpatelh
drpatelh reviewed Oct 5, 2021
View reviewed changes
@drpatelh drpatelh merged commit 3868c3a into nf-core:master Oct 5, 2021
@abhi18av abhi18av deleted the abhinav/gtdbtk-classifywf branch October 6, 2021 10:03
@abhi18av
Copy link
Member Author

abhi18av commented Oct 6, 2021

Awesome - thanks for your time and guidance guys! 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@drpatelh drpatelh drpatelh left review comments

@rpetit3 rpetit3 rpetit3 approved these changes

@skrakau skrakau Awaiting requested review from skrakau

Assignees

@abhi18av abhi18av

Labels

new module Adding a new module Ready for Review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@abhi18av @rpetit3 @drpatelh