Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@ewels
Copy link
Member

@ewels ewels commented Jun 17, 2020

Updates to nf-core launch to work with new pipeline JSON schema.

PR checklist

  • This comment contains a description of changes (with reason)
  • If you've fixed a bug or added code that should be tested, add tests!
  • Documentation in docs is updated
  • CHANGELOG.md is updated
  • README.md is updated

Maybe also want to wrap the send-to-website-GUI functionality into this PR too, even if the website side is not yet written.

ewels added 4 commits June 17, 2020 16:33
* Shorten cli help text strings
* Update input paramters instead of clobbering
Start with JSON schema defaults for all parameters, then overwrite these with whatever inputs and prompts are supplied. Then validate this full set of inputs - means that default values for required params no longer fail. Default values then removed again before printing to a file.
@ewels ewels marked this pull request as draft June 17, 2020 22:23
@ewels ewels added the WIP Work in progress label Jun 17, 2020
@codecov
Copy link

codecov bot commented Jun 17, 2020

Codecov Report

Merging #626 into dev will increase coverage by 7.35%.
The diff coverage is 72.11%.

Impacted file tree graph

@@            Coverage Diff             @@
##              dev     #626      +/-   ##
==========================================
+ Coverage   68.09%   75.45%   +7.35%     
==========================================
  Files          11       11              
  Lines        1984     2159     +175     
==========================================
+ Hits         1351     1629     +278     
+ Misses        633      530     -103     
Impacted Files Coverage Δ
nf_core/sync.py 0.00% <ø> (ø)
nf_core/list.py 70.63% <38.29%> (-5.95%) ⬇️
nf_core/launch.py 64.06% <72.84%> (+64.06%) ⬆️
nf_core/schema.py 88.28% <82.55%> (+14.62%) ⬆️
nf_core/utils.py 91.89% <92.00%> (+18.81%) ⬆️
nf_core/lint.py 91.88% <100.00%> (ø)
nf_core/create.py 92.95% <0.00%> (+0.31%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 129bc8f...d89850b. Read the comment docs.

@ewels ewels added this to the 1.10 milestone Jun 17, 2020
@ewels ewels removed the WIP Work in progress label Jun 18, 2020
@ewels ewels marked this pull request as ready for review June 18, 2020 15:19
@ewels ewels requested a review from a team June 18, 2020 15:19
Copy link
Member

@maxulysse maxulysse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, but I'm not sure about the pythony part.
I should really iron out my python skills.

@maxulysse
Copy link
Member

OK, so when I create a group, name it, and then delete it, it doesn't disappear from the builder, and I can't access options (with the cog).
I can delete a group that I haven't named, but not one that is named.

Deleted group that doesn't disappear from the builder is actually not saved in the schema, but still disturbing to see it there.

@maxulysse
Copy link
Member

I like the possibility to add an icon to a params, but do we really need all brands from font-awesome?

Given the huge amount of icons, is there any guidelines on which icons to use for some common params?
Or should we already hardcode the basics common params (--input, --help, --genome...)
Like should I get info, info-circle, hands-helping, question or question-circle for --help.

I am allowed to try to change the icon for a group, but it is actually not working.
Group does have the possibility to have an help message which is disturbing as it's naming it params..

@maxulysse
Copy link
Member

I understood that nf-core schema build . --url ... is to change the url of the builder, which I guess was useful when developing.

I was hoping you actually could save your progress and access it again, which also explained the 24 hours expiration period.

I'm now guessing that if you want to continue working on a schema, you need to save it first, and then continue with the json.

Could be good to detail that in the docs.

@maxulysse
Copy link
Member

maxulysse commented Jun 20, 2020

Keyboards shortcuts for power users is a good idea.

But with a long enough list of params, if you're on the bottom and use shift + enter to go up, at some point the focus goes too up and you actually don't see it anymore.

EDIT: and similar behaviour with shift + ctrl +

Also, it's shift + enter to go up, not tab + enter

@ewels
Copy link
Member Author

ewels commented Jun 20, 2020

If you’re able to make a gif of this kind of behaviour to explain what you mean, that would be fab 👍🏻 Screenshots also fine.

@apeltzer
Copy link
Member

I agree that having defaults for --input for example and --help would be cool, as they will be very similar across pipelines and should be using the same font awesome icons as Maxime said :-)

@ewels
Copy link
Member Author

ewels commented Jun 21, 2020

Mentioned on slack - my idea for this is that if no schema is found it will auto generate one using the pipeline schema, then update this against the pipeline.

Should be relatively little code to add and will mean you start off with a nice boilerplate schema with all of the common options already done for you.

@apeltzer
Copy link
Member

Sounds good to me :-)

@ewels
Copy link
Member Author

ewels commented Jun 22, 2020

#630

ewels added 2 commits June 22, 2020 09:37
Instead of recurisvely calling the check status function, use a loop. Avoids recursion depth limits and means that we can effectively wait forever without timing out.
@ewels
Copy link
Member Author

ewels commented Jun 22, 2020

Ok, rewrote how the schema builder waits for a response from the website so that it no longer times out. It should wait forever now.

@ewels ewels marked this pull request as draft June 23, 2020 19:19
@ewels
Copy link
Member Author

ewels commented Jun 23, 2020

TODO:

  • Convert answer variable types after prompt (number, integer, range)
    • Avoid ERROR: [✗] Input parameters are invalid: '5' is not of type 'integer'
  • Give selection prompt for parameters with enum?
    • Only if number of options is small?

@ggabernet
Copy link
Member

ggabernet commented Jun 23, 2020

The json schema web interface looks amazing and works super smoothly and made me realize of a couple of params in Bcellmagic missing in the nextflow.config, so great stuff!

I have just one minor comment:

  • Dragging a parameter into a new parameter group works, but it took me a while to make it work. Maybe making the box where the single parameters fit a bit bigger when it is empty makes it easier.

For the nf-core launch cli:

  • When a parameter is required but not provided, there is no note for that but only an Error on the very end: ERROR: [✗] Input parameters are invalid: 'metadata' is a required property. A (required) note could be shown next to the parameter, to avoid having to repeat the process for all required parameters.

@ewels ewels marked this pull request as ready for review June 24, 2020 07:02
@ewels
Copy link
Member Author

ewels commented Jun 24, 2020

Thanks for the testing and feedback @ggabernet!

Regarding the last point, for me this already happens:

image

Could you give a bit more info about when this happens for you so that I can try to replicate it?

ewels added a commit to ewels/nf-core-website that referenced this pull request Jun 24, 2020
Set a minimum height for the content area of a group. This makes it easier to drop params into empty groups.

Suggested by @ggabernet in nf-core/tools#626
@ggabernet
Copy link
Member

ggabernet commented Jun 24, 2020

With the JSON schema builder I set metadata as required:

Screenshot 2020-06-24 at 09 43 07

The JSON description for metadata in the builder is then the following:

                "metadata": {
                    "type": "string",
                    "description": "Input metadata sheet",
                    "fa_icon": "far fa-file"
                },

Edit: found that this is listed here:

            "fa_icon": "fas fa-terminal",
            "required": [
                "metadata",
                "cprimers",
                "vprimers"
            ],

Then I launch the pipeline with nf-core launch, it finds the schema and groups the parameters in the groups I defined in the builder, however I can press Continue through this group of params and it does not prompt me an error that metadata is required. Just in the end:

Screenshot 2020-06-24 at 09 51 56

It does prompt me an error if I try to edit it and leave it empty, though, the same as you described:

Screenshot 2020-06-24 at 09 54 56

@ewels
Copy link
Member Author

ewels commented Jun 24, 2020

Aha, ok - so it needs to validate at the group level as well as the parameter level 👍 Thanks, yes I should be able to do this..

@ewels
Copy link
Member Author

ewels commented Jun 24, 2020

And yes, in the template we have a default value set for reads but maybe it's a better idea to leave this without a default (in the schema at least).

@ggabernet
Copy link
Member

ggabernet commented Jun 24, 2020

One other small thing: when I run nf-core launch . in a directory where there's no pipeline, it still creates an nextflow_schema.json.

In the end it complains that it does not find the main.nf:

Screenshot 2020-06-24 at 10 11 15

Maybe it could check for the main.nf already when creating the schema, or if nextflow.config is not there, just stop.

@ewels
Copy link
Member Author

ewels commented Jun 24, 2020

Hah, excellent! Apparently nextflow config works fine with an empty directory too. Ok I'll put in a check for that, good spot 👍

ewels added 4 commits June 24, 2020 10:42
Changed order of groups - moved Generic options up.
Added some descriptions and help text to groups.
Also remove default value from template schema for --reads
@ewels
Copy link
Member Author

ewels commented Jun 24, 2020

Ok, I think that all of your points are now addressed @ggabernet 👍

ewels added 3 commits June 24, 2020 16:57
Also update caching for fetch_wf_config() so that it works purely on filenames / file hashes.
Fix bug with parsing params from main.nf
@ewels ewels merged commit 81ca05b into nf-core:dev Jun 24, 2020
@ewels ewels deleted the schema_launch_updates branch June 24, 2020 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Schema builder: Use template as default Refactor nf-core launch for new schema Launch should allow to choose between pipeline releases

4 participants