Welcome to the nf-scil project ! A Nextflow modules and workflows repository for neuroimaging maintained by the SCIL team. The primary focus of the library is to provide pre-built processes and processing sequences for diffusion Magnetic Resonance Imaging, optimized for Nextflow DSL2, based on open-source technologies and made easily available to pipeline's developers through the nf-core framework.

• Using modules from nf-scil
• Developing in nf-scil
  • Manual configuration
    • Dependencies
    • Python environment
    • Loading the project's environment
    • Working with VS Code
  • Configuration via the devcontainer
  • Contributing to the nf-scil project
    • Adding a new module to nf-scil
      • Generate the template
      • Edit the template
        • Editing the module's main.nf
        • Editing the module's meta.yml
        • Editing the module's environment.yml
        • Editing the tests' main.nf.test
        • Editing the tests' nextflow.config
      • Run the tests to generate the test metadata file
      • Lint your code
      • Last safety test
      • Submit your PR
    • Defining processes optional parameters
    • Test data infrastructure
      • Using the .test_data directory
      • Using Scilpy Fetcher
    • Adding a new subworkflow to nf-scil
      • Generate the template
      • Generate the template
        • Edit the subworkflow's main.nf
          • Define your Subworkflow inputs.
          • Fill the main: section.
          • Define your Workflow outputs.
        • Edit the subworkflow's meta.yml
      • Lint your code
      • Submit your PR
  • Running tests
  • Configuring Docker for easy usage
  • Installing Prettier

To import modules from nf-scil, you first need to install nf-core on your system (can be done simply using pip install nf-core). Once done, nf-scil modules are imported using this command :

nf-core modules \
  --git-remote https://github.com/scilus/nf-scil.git \
  install <category>/<tool>

where you input the <tool> you want to import from the desired <category>. To get a list of the available modules, run :

nf-core modules \
  --git-remote https://github.com/scilus/nf-scil.git \
  list remote

The same can be done for subworkflows, replacing modules in the nf-core command by `subworkflows, e.g. :

nf-core subworkflows \
  --git-remote https://github.com/scilus/nf-scil.git \
  install <category>/<subworkflow>

It can become heavy to always prepend the commands with --git-remote, even so if you need to specify a --branch where to fetch the information. You can instead define the git-remote and branch using Environment Variables :

export NFCORE_MODULES_GIT_REMOTE=https://github.com/scilus/nf-scil.git
export NFCORE_MODULES_GIT_BRANCH=main
export NFCORE_SUBWORKFLOWS_GIT_REMOTE=https://github.com/scilus/nf-scil.git
export NFCORE_SUBWORKFLOWS_GIT_BRANCH=main

and call all commands without specifying the --git-remote and --branch options, while still targeting the nf-scil repository.

The nf-scil project requires some specific tools to be installed on your system so that the development environment runs correctly. You can install them manually, but if you desire to streamline the process and start coding faster, we highly recommend using the VS Code development container to get fully configured in a matter of minutes.

• Python ≥ 3.8, < 3.13
• Docker ≥ 24 (we recommend using Docker Desktop)
• Java Runtime ≥ 11, ≤ 17
  • On Ubuntu, install openjdk-jre-<version> packages
• Nextflow ≥ 21.04.3
• nf-test ≥ 0.9.0-rc1
• Node ≥ 14 and Prettier (see below)

The project uses poetry to manage python dependencies. To install it using pipx, run the following commands :

pip install pipx
pipx ensurepath
pipx install poetry==1.8.*

Once done, install the project with :

poetry install

The project scripts and dependencies can be accessed using :

poetry shell

which will activate the project's python environment in the current shell.

To exit the environment, simply enter the exit command in the shell.

Set the following environment variables in your .bashrc (or whatever is the equivalent for your shell) :

export NFCORE_MODULES_GIT_REMOTE=https://github.com/scilus/nf-scil.git
export NFCORE_MODULES_GIT_BRANCH=main
export NFCORE_SUBWORKFLOWS_GIT_REMOTE=https://github.com/scilus/nf-scil.git
export NFCORE_SUBWORKFLOWS_GIT_BRANCH=main

This will make it so the nf-core commands target the right repository by default. Else, you'll need to add --git-remote and --branch options to pretty much all commands relating to modules and subworkflows.

The nf-scil project curates a bundle of useful extensions for Visual Studio Code, the nf-scil-extensions package. You can find it easily on the extension marketplace.

The devcontainer definition for the project contains all required dependencies and setup steps are automatically executed. To use this installation method, you need to have Docker (refer to this section for configuration requirements or validate your configuration) and Visual Studio Code installed on your system.

Open the cloned repository in VS Code and click on the arrow box in the lower left corner, to get a prompt to Reopen in container. The procedure will start a docker build, wait for a few minutes and enjoy your fully configured development environment.

• Available in the container :

  • nf-scil, nf-core all accessible through the terminal, which is configured to load the poetry environment in shells automatically
  • nf-scil configured as the main repository for all nf-core commands, using NFCORE_* environment variables
  • git, git-lfs, github-cli
  • curl, wget, apt-get
  • nextflow, nf-test, docker, tmux

• Available in the VS Code IDE through extensions :

  • Docker images and containers management
  • Python and C++ linting, building and debugging tools
  • Github Pull Requests management
  • Github flavored markdown previewing

If you want to propose a new module to the repository, follow the guidelines in the module creation documentation. The same goes for subworkflows, using these guidelines instead. We follow standards closely aligned with nf-core, with some exceptions on process atomicity and how test data is handled. Modules that don't abide to them won't be accepted and PR containing them will be closed automatically.

Tests are run through nf-core, using the command :

nf-core modules test <category/tool>

The tool can be omitted to run tests for all modules in a category.

The simplest way of installing everything Docker is to use Docker Desktop. You can also go the engine way and install Docker manually.

Once installed, you need to configure some access rights to the Docker daemon. The easiest way to do this is to add your user to the docker group. This can be done with the following command :

sudo groupadd docker
sudo usermod -aG docker $USER

After running this command, you need to log out and log back in to apply the changes.

To install Prettier for the project, you need to have node and npm installed on your system to at least version 14. On Ubuntu, you can do it using snap :

sudo snap install node --classic

However, if you cannot install snap, or have another OS, refer to the official documentation for the installation procedure.

Under the current configuration for the Development Container, for this project, we use the following procedure, considering ${NODE_MAJOR} is at least 14 for Prettier :

curl -fsSL https://deb.nodesource.com/setup_${NODE_MAJOR}.x | bash - &&\
apt-get install -y nodejs

npm install --save-dev --save-exact prettier

echo "function prettier() { npm exec prettier $@; }" >> ~/.bashrc