Thanks to visit codestin.com
Credit goes to github.com

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
cb06dc9
Added questionary select list for releases
ErikDanielsson Apr 15, 2021
519dd18
Added confirmation prompt for image download and some docs
ErikDanielsson Apr 15, 2021
77808a4
Added prompt for singularity caching (and 'negated' prompt for image …
ErikDanielsson Apr 15, 2021
f53b942
Added prompt for compression type
ErikDanielsson Apr 15, 2021
958e95b
Added checking for 'export NXF_SINGULARITY_CACHEDIR' in bashrc
ErikDanielsson Apr 15, 2021
897bae3
Added version selection list for launch
ErikDanielsson Apr 16, 2021
89ff0ad
Added path prompt
ErikDanielsson Apr 16, 2021
bf4b64a
Changed to only prompt when options are not specified
ErikDanielsson Apr 16, 2021
8a945e7
Clean up some unnecessary changes
ErikDanielsson Apr 16, 2021
d39e4eb
Merge branch 'master' into prompts
ewels Apr 28, 2021
ec7d486
Fix syntax error
ewels Apr 28, 2021
115c112
Testing and refactoring
ewels Apr 28, 2021
935b64e
Fix behaviour when pipeline name doesn't exist
ewels Apr 28, 2021
b41337d
Better cli styling
ewels Apr 28, 2021
48f9146
Fast fail for non-existant repos
ewels Apr 28, 2021
0fc0477
More testing - bugfix
ewels Apr 28, 2021
59e0654
Clean up + finish refactor for --container instead of --singularity
ewels Apr 28, 2021
bb6b6d2
Fix Singularity installation check
ewels Apr 29, 2021
d9c7207
Fix / rewrite code for singularity cachedir prompts + bashrc addition
ewels Apr 29, 2021
1c921d6
Restructure and reorganise prompts code
ewels Apr 29, 2021
9f0fe81
Tweaks to log messages
ewels Apr 29, 2021
24adc00
Use new rich console stderr argument, streamline imports
ewels Apr 29, 2021
fe77d55
Unsafe ask, colour select for releases / branches
ewels Apr 29, 2021
3c80067
Write some help text for prompts
ewels Apr 29, 2021
f9c9fdc
Refactored code that gets branches and releases and hashes. Rewired s…
ewels Apr 30, 2021
60c3730
Pipeline name should be full_name
ewels Apr 30, 2021
9c17495
Update nf_core/launch.py
ewels Apr 30, 2021
9796e69
Merge branch 'dev' of github.com:nf-core/tools into prompts
ewels Apr 30, 2021
3d0d415
Just bold, not white on bold.
ewels Apr 30, 2021
10ddb01
Removed more white bold, replaced with bold.
ewels Apr 30, 2021
06265eb
Awesome questionary path auto-completion for singularity cachedir path
ewels Apr 30, 2021
c63487c
Refine cli flags, -c now for container instead of compress
ewels Apr 30, 2021
0f2b8f4
Launch - sort releases by release date
ewels Apr 30, 2021
f7f3754
Merge branch 'master' into prompts
ewels Apr 30, 2021
f1999d4
Check if terminal is interactive before prompting for setting
ewels Apr 30, 2021
683242d
Screen container names for dynamic {squiggly_brackets}
ewels Apr 30, 2021
7b745b3
Push up minimum version of rich, louder colours
ewels Apr 30, 2021
b66961c
Bump rich minimum version again, as v1.10.0 fixed table style issue
ewels Apr 30, 2021
a0bc438
Fix pytests for download code
ewels Apr 30, 2021
bd8152b
Don't force interactive terminal on GitHub Actions for pytest
ewels Apr 30, 2021
9bbdbe3
Different approach to try to avoid interactive prompt in tests
ewels Apr 30, 2021
67ec08b
Refactor: Move bunch of prompt / lookup code into utils for reuse
ewels May 1, 2021
ed9dd62
Launch: Prompt for pipeline and release
ewels May 1, 2021
ea32219
Pull in updates, fix merge conflict
ewels May 1, 2021
8d4fc0e
nf_core.utils.get_repo_releases_branches() - return pipeline name.
ewels May 6, 2021
5478af8
Update pytests
ewels May 6, 2021
73fe7f4
Download - update readme docs
ewels May 6, 2021
205340b
Download docs - didn't compress
ewels May 6, 2021
70eca94
Merge branch 'master' into prompts
ewels May 6, 2021
19f2d5c
Address review comments for singularty cachedir prompt
ewels May 7, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 41 additions & 27 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -344,49 +344,58 @@ Sometimes you may need to run an nf-core pipeline on a server or HPC system that
In this case you will need to fetch the pipeline files first, then manually transfer them to your system.

To make this process easier and ensure accurate retrieval of correctly versioned code and software containers, we have written a download helper tool.
Simply specify the name of the nf-core pipeline and it will be downloaded to your current working directory.

By default, the pipeline will download the pipeline code and the [institutional nf-core/configs](https://github.com/nf-core/configs) files.
If you specify the flag `--singularity`, it will also download any singularity image files that are required.
The `nf-core download` command will download both the pipeline code and the [institutional nf-core/configs](https://github.com/nf-core/configs) files. It can also optionally download any singularity image files that are required.

Use `-r`/`--release` to download a specific release of the pipeline. If not specified, the tool will automatically fetch the latest release.
If run without any arguments, the download tool will interactively prompt you for the required information.
Each option has a flag, if all are supplied then it will run without any user input needed.

```console
$ nf-core download rnaseq -r 3.0 --singularity
$ nf-core download

,--./,-.
___ __ __ __ ___ /,-._.--~\
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'

nf-core/tools version 1.13
nf-core/tools version 1.14


Specify the name of a nf-core pipeline or a GitHub repository name (user/repo).
? Pipeline name: rnaseq
? Select release / branch: 3.0 [release]

In addition to the pipeline code, this tool can download software containers.
? Download software container images: singularity

Nextflow and nf-core can use an environment variable called $NXF_SINGULARITY_CACHEDIR that is a path to a directory where remote Singularity
images are stored. This allows downloaded images to be cached in a central location.
? Define $NXF_SINGULARITY_CACHEDIR for a shared Singularity image download folder? [y/n]: y
? Specify the path: cachedir/

INFO Saving rnaseq
So that $NXF_SINGULARITY_CACHEDIR is always defined, you can add it to your ~/.bashrc file. This will then be autmoatically set every time you open a new terminal. We can add the following line to this file for you:
export NXF_SINGULARITY_CACHEDIR="/path/to/demo/cachedir"
? Add to ~/.bashrc ? [y/n]: n

If transferring the downloaded files to another system, it can be convenient to have everything compressed in a single file.
This is not recommended when downloading Singularity images, as it can take a long time and saves very little space.
? Choose compression type: none
INFO Saving 'nf-core/rnaseq
Pipeline release: '3.0'
Pull singularity containers: 'Yes'
Output file: 'nf-core-rnaseq-3.0.tar.gz'
Pull containers: 'singularity'
Using $NXF_SINGULARITY_CACHEDIR': /path/to/demo/cachedir
Output directory: 'nf-core-rnaseq-3.0'
INFO Downloading workflow files from GitHub
INFO Downloading centralised configs from GitHub
INFO Fetching container names for workflow
INFO Found 29 containers
INFO Tip: Set env var $NXF_SINGULARITY_CACHEDIR to use a central cache for container downloads
Downloading singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% • 29/29 completed
INFO Compressing download..
INFO Command to extract files: tar -xzf nf-core-rnaseq-3.0.tar.gz
INFO MD5 checksum for nf-core-rnaseq-3.0.tar.gz: 9789a9e0bda50f444ab0ee69cc8a95ce
Downloading singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% • 29/29 completed
```

The tool automatically compresses all of the resulting file in to a `.tar.gz` archive.
You can choose other formats (`.tar.bz2`, `zip`) or to not compress (`none`) with the `-c`/`--compress` flag.
The console output provides the command you need to extract the files.

Once uncompressed, you will see something like the following file structure for the downloaded pipeline:
Once downloaded, you will see something like the following file structure for the downloaded pipeline:

```console
$ tree -L 2 nf-core-methylseq-1.4/
$ tree -L 2 nf-core-rnaseq-3.0/

nf-core-rnaseq-3.0
├── configs
Expand All @@ -404,7 +413,11 @@ nf-core-rnaseq-3.0
└── main.nf
```

You can run the pipeline by simply providing the directory path for the `workflow` folder to your `nextflow run` command.
You can run the pipeline by simply providing the directory path for the `workflow` folder to your `nextflow run` command:

```bash
nextflow run /path/to/download/nf-core-rnaseq-3.0/workflow/ --input mydata.csv # usual parameters here
```

### Downloaded nf-core configs

Expand All @@ -414,7 +427,7 @@ So using `-profile <NAME>` should work if available within [nf-core/configs](htt
### Downloading singularity containers

If you're using Singularity, the `nf-core download` command can also fetch the required Singularity container images for you.
To do this, specify the `--singularity` option.
To do this, select `singularity` in the prompt or specify `--container singularity` in the command.
Your archive / target output directory will then include three folders: `workflow`, `configs` and also `singularity-containers`.

The downloaded workflow files are again edited to add the following line to the end of the pipeline's `nextflow.config` file:
Expand All @@ -433,7 +446,8 @@ We highly recommend setting the `$NXF_SINGULARITY_CACHEDIR` environment variable
If found, the tool will fetch the Singularity images to this directory first before copying to the target output archive / directory.
Any images previously fetched will be found there and copied directly - this includes images that may be shared with other pipelines or previous pipeline version downloads or download attempts.

If you are running the download on the same system where you will be running the pipeline (eg. a shared filesystem where Nextflow won't have an internet connection at a later date), you can choose specify `--singularity-cache`.
If you are running the download on the same system where you will be running the pipeline (eg. a shared filesystem where Nextflow won't have an internet connection at a later date), you can choose to _only_ use the cache via a prompt or cli options `--singularity-cache-only` / `--singularity-cache-copy`.

This instructs `nf-core download` to fetch all Singularity images to the `$NXF_SINGULARITY_CACHEDIR` directory but does _not_ copy them to the workflow archive / directory.
The workflow config file is _not_ edited. This means that when you later run the workflow, Nextflow will just use the cache folder directly.

Expand All @@ -451,15 +465,15 @@ Where both are found, the download URL is preferred.

Once a full list of containers is found, they are processed in the following order:

1. If the target image already exists, nothing is done (eg. with `$NXF_SINGULARITY_CACHEDIR` and `--singularity-cache` specified)
2. If found in `$NXF_SINGULARITY_CACHEDIR` and `--singularity-cache` is _not_ specified, they are copied to the output directory
1. If the target image already exists, nothing is done (eg. with `$NXF_SINGULARITY_CACHEDIR` and `--singularity-cache-only` specified)
2. If found in `$NXF_SINGULARITY_CACHEDIR` and `--singularity-cache-only` is _not_ specified, they are copied to the output directory
3. If they start with `http` they are downloaded directly within Python (default 4 at a time, you can customise this with `--parallel-downloads`)
4. If they look like a Docker image name, they are fetched using a `singularity pull` command
* This requires Singularity to be installed on the system and is substantially slower

Note that compressing many GBs of binary files can be slow, so specifying `--compress none` is recommended when downloading Singularity images.

If you really like hammering your internet connection, you can set `--parallel-downloads` to a large number to download loads of images at once.
If the download speeds are much slower than your internet connection is capable of, you can set `--parallel-downloads` to a large number to download loads of images at once.

## Pipeline software licences

Expand Down
30 changes: 14 additions & 16 deletions nf_core/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@

from click.types import File
from rich import print
from rich.prompt import Confirm
import click
import logging
import os
Expand Down Expand Up @@ -35,7 +36,7 @@ def run_nf_core():
rich.traceback.install(width=200, word_wrap=True, extra_lines=1)

# Print nf-core header to STDERR
stderr = rich.console.Console(file=sys.stderr, force_terminal=nf_core.utils.rich_force_colors())
stderr = rich.console.Console(stderr=True, force_terminal=nf_core.utils.rich_force_colors())
stderr.print("\n[green]{},--.[grey39]/[green],-.".format(" " * 42), highlight=False)
stderr.print("[blue] ___ __ __ __ ___ [green]/,-._.--~\\", highlight=False)
stderr.print("[blue] |\ | |__ __ / ` / \ |__) |__ [yellow] } {", highlight=False)
Expand Down Expand Up @@ -115,7 +116,7 @@ def nf_core_cli(verbose, log_file):
log.addHandler(
rich.logging.RichHandler(
level=logging.DEBUG if verbose else logging.INFO,
console=rich.console.Console(file=sys.stderr, force_terminal=nf_core.utils.rich_force_colors()),
console=rich.console.Console(stderr=True, force_terminal=nf_core.utils.rich_force_colors()),
show_time=False,
markup=True,
)
Expand Down Expand Up @@ -200,36 +201,33 @@ def launch(pipeline, id, revision, command_only, params_in, params_out, save_all


# nf-core download


@nf_core_cli.command(help_priority=3)
@click.argument("pipeline", metavar="<pipeline name>")
@click.argument("pipeline", required=False, metavar="<pipeline name>")
@click.option("-r", "--release", type=str, help="Pipeline release")
@click.option("-o", "--outdir", type=str, help="Output directory")
@click.option(
"-c",
"--compress",
type=click.Choice(["tar.gz", "tar.bz2", "zip", "none"]),
default="tar.gz",
help="Archive compression type",
"-x", "--compress", type=click.Choice(["tar.gz", "tar.bz2", "zip", "none"]), help="Archive compression type"
)
@click.option("-f", "--force", is_flag=True, default=False, help="Overwrite existing files")
@click.option("-s", "--singularity", is_flag=True, default=False, help="Download singularity images")
@click.option(
"-c",
"--singularity-cache",
is_flag=True,
default=False,
help="Don't copy images to the output directory, don't set 'singularity.cacheDir' in workflow",
"-c", "--container", type=click.Choice(["none", "singularity"]), help="Download software container images"
)
Comment on lines 214 to +216
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unless we are planning to extend this choice list at some point, this could also just be flag instead of a prompt I think.
But also fine leaving it as a prompt.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My preference would be to leave it as a prompt. One could make a semi-reasonable case for offering download to pull e.g. docker images to the local instance.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I went back on forth on this. At the moment it is just a --singularity flag. However, with the interactive prompts I would have needed to make it an on/off flag (--singularity --no-singularity). I don't love littering the cli with these, and figured that --container [str] was kind of more concise and had the added benefit of future-proofing.

@phue - as the only CharlieCloud user I know, does that have a similar concept of pulling image files? Would it ever be useful to add CharlieCloud image downloads to the nf-core download command for example?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And yes I also wondered about docker too @pontus 👍🏻

@click.option(
"--singularity-cache-only/--singularity-cache-copy",
help="Don't / do copy images to the output directory and set 'singularity.cacheDir' in workflow",
)
@click.option("-p", "--parallel-downloads", type=int, default=4, help="Number of parallel image downloads")
def download(pipeline, release, outdir, compress, force, singularity, singularity_cache, parallel_downloads):
def download(pipeline, release, outdir, compress, force, container, singularity_cache_only, parallel_downloads):
"""
Download a pipeline, nf-core/configs and pipeline singularity images.

Collects all files in a single archive and configures the downloaded
workflow to use relative paths to the configs and singularity images.
"""
dl = nf_core.download.DownloadWorkflow(
pipeline, release, outdir, compress, force, singularity, singularity_cache, parallel_downloads
pipeline, release, outdir, compress, force, container, singularity_cache_only, parallel_downloads
)
dl.download_workflow()

Expand Down
3 changes: 1 addition & 2 deletions nf_core/bump_version.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
a nf-core pipeline.
"""

import click
import logging
import os
import re
Expand All @@ -12,7 +11,7 @@
import nf_core.utils

log = logging.getLogger(__name__)
stderr = rich.console.Console(file=sys.stderr, force_terminal=nf_core.utils.rich_force_colors())
stderr = rich.console.Console(stderr=True, force_terminal=nf_core.utils.rich_force_colors())


def bump_pipeline_version(pipeline_obj, new_version):
Expand Down
Loading