✅ Deploy Preview for nextflow-docs-staging ready!

Name	Link
🔨 Latest commit	09c9e96
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs-staging/deploys/6a2ae8fb06f1bf00081bb8b5
😎 Deploy Preview	https://deploy-preview-7213--nextflow-docs-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

Great write up, thanks for this Ben!

As you might expect, I'm most concerned about the params. You characterise it as a one-off cost which is mitigated by LLMs, however that doesn't take into account updates to included pipelines (a core functionality with included modules). The params drift with updates would be dangerous and a constant source of dev work.

I'd still love to look into how we could bulk import nested config and apply it at root level. Even if it is a separate import + apply mechanism (eg. like config profiles in a sense?). I think without it, the use of the meta pipeline functionality is substantially limited.

As you might expect, I'm most concerned about the params.

Agreed. Feel like we need some sort of auto-import of the params of child workflows, so e.g. they appear automatically in Platform, and I could say e.g. meta.rnaseq.pseudoaligner = 'kallisto' in the meta pipeline's nextflow.config to override.

Then some auto-assembly of docs as well.

Basically we need to standardise at the nextflow level where a bunch of the non-nextflow pieces need to live.

As you might expect, I'm most concerned about the params.

Agreed. Feel like we need some sort of auto-import of the params of child workflows, so e.g. they appear automatically in Platform, and I could say e.g. meta.rnaseq.pseudoaligner = 'kallisto' in the meta pipeline's nextflow.config to override.

Then some auto-assembly of docs as well.

Basically we need to standardise at the nextflow level where a bunch of the non-nextflow pieces need to live.

I disagree. Having unpredictable global scope params blocks is just weird and if we were designed Nextflow today we would never include this behaviour. In other languages, globals need to be used with caution and are generally not advised. Having random params.foo.bar.baz with no way of validating or checking is just "something you have to know", instead of being clear to the author.

In a scenario where I update my workflow from v1.1 to v1.2, an update to params should be explicit in the input block, not implicit and I hope it doesn't change too much.

If we really want to make them importable, we could add a dedicated params block to the workflow definition:

workflow THING {
   params:
        foo: Int
        bar: Bool
        baz: String

   take:
   // etc
}

but this doesn't feel very different to:

record ThingParams {
    foo: Int
    bar: Bool
    baz: String
}

workflow THING {
   take:
       params: ThingParams

   // etc
}

My main concern here is versioning of imported workflows. Do we include a lock file or something to ensure consistency or just trust in the files that are copied into the workflow code?

@pditommaso thanks for the review

The part I'd love to see fleshed out is how channels and values get bound into the included workflow's inputs.

There isn't much to say here because it just works like normal. In the appendix example, NFCORE_RNASEQ is just a named workflow. The meta-pipeline calls it the same way that rnaseq would call it. The only difference is that some inputs might come from upstream outputs instead of params.

... how a scalar value vs. a streaming channel is bound at the call site, the value-channel/queue-channel broadcast semantics, and whether a typed take: can accept a bare value type like String/Path.

A workflow take can be a channel, a dataflow value, or a regular value. This is how it has always worked

The document moves between "meta-pipeline" and "remote pipeline", and I think the framing could be sharpened. Terms like workflow modularisation / workflow inclusion / workflow composition might describe what's happening (composing one workflow into another) more directly than introducing a new "meta-pipeline" category.

"Meta-pipeline" is the top-line feature that everyone is after, but the only actual new feature proposed by the ADR is "remote pipeline inclusion" -- how to install a pipeline as a component and keep it in sync with the source. This is why the ADR is titled "Remote pipeline inclusion". Once you have that, everything else is just normal workflow composition and convention.

They are distinct concepts -- the ADR does not treat them as interchangeable.

Once you discard the entry workflow, params, and output block and import only the core workflow, what's left looks a lot like a sub-workflow. It'd be helpful to clarify how this differs from including a remote sub-workflow, and what the main benefit is that justifies a separate mechanism (separate storage layout, a new nextflow_spec.json, a separate CLI, etc.).

The core workflow looks like a subworkflow because it is a subworkflow 😄

The only new thing that we introduce here is installing a pipeline into a project as a component and keeping it in sync with the remote source (either from Git or the registry). For that you likely need a pipeline spec (version, checksum) and a CLI (installing, updating). I just haven't spelled all that out yet because the bigger question right now is how to minimize developer overhead

I'd lean toward framing the next step as enabling remote sub-workflows — the natural progression after remote modules (processes). Module (process) → sub-workflow → composition feels like a clean, incremental story that reuses the conventions we already have, rather than introducing a "pipeline" as a new top-level artifact with its own resolution rules, storage path, and spec file. If we get remote sub-workflow inclusion right, "meta-pipelines" might largely fall out of it as a usage pattern rather than a new concept.

Looks like you arrived at the same place as me. Remote workflows are the real feature, meta-pipelines emerge naturally as a convention on top.

I'm not sure whether it's worth trying to distinguish between pipelines / workflows / subworkflows. They're all basically the same thing. Especially if we add the ability to execute named workflows directly (#7208). The difference boils down to boilerplate, which we want to minimize anyway

This is why I just talk about "remote pipeline inclusion", because when I import a workflow, I don't really care whether that workflow is a "pipeline" like rnaseq or a "subworkflow" like BAM_STATS_SAMTOOLS. Workflow composition works the same way either way.

Happy to rename the ADR to "remote workflow inclusion" to align with the workflow keyword.

My main concern here is versioning of imported workflows. Do we include a lock file or something to ensure consistency or just trust in the files that are copied into the workflow code?

Modules have a .moduleinfo file with a hash to allow checking that stuff wasn't modified. I think I saw something similar mentioned here for pipelines / workflows?

Having unpredictable global scope params blocks is just weird and if we were designed Nextflow today we would never include this behaviour. In other languages, globals need to be used with caution and are generally not advised.

@adamrtalbot agreed, I never said global. I would love it if the pipeline config is imported within a dedicated scope and treated as a baseline default. Then the import-ing pipeline can override anything, but doesn't need to duplicate config that isn't being changed.

Doing this would not be trivial. The only way I can think of is to do something fairly radical like rendering the config at import time and saving that to a locked config file somewhere. Or some other crazy mechanism.

Having unpredictable global scope params blocks is just weird and if we were designed Nextflow today we would never include this behaviour. In other languages, globals need to be used with caution and are generally not advised.

@adamrtalbot agreed, I never said global. I would love it if the pipeline config is imported within a dedicated scope and treated as a baseline default. Then the import-ing pipeline can override anything, but doesn't need to duplicate config that isn't being changed.

Doing this would not be trivial. The only way I can think of is to do something fairly radical like rendering the config at import time and saving that to a locked config file somewhere. Or some other crazy mechanism.

Config or params? In my mind they are very different concepts, I was referring to parameters here.

Happy to rename the ADR to "remote workflow inclusion" to align with the workflow keyword.

I agree with this. They're all workflows*, the only thing that separates a "pipeline" from a subworkflow is perception.

*except the anonymous entry workflow, which is where the sticky point about params and config comes in 😉

Config or params? In my mind they are very different concepts, I was referring to parameters here.

Ideally params, but might need to be config for all the ext stuff..?

Happy to rename the ADR to "remote workflow inclusion" to align with the workflow keyword.

Yeah as it stands I think this basically boils down to the functionality we already have with nf-core subworkflows, right? Which is quite far from what I think of as meta-pipelines. Still good to have and useful..

Yeah as it stands I think this basically boils down to the functionality we already have with nf-core subworkflows, right?

Can the nf-core tooling install a workflow from a pipeline repo? e.g. NFCORE_RNASEQ from nf-core/rnaseq? I think that is the main thing that this ADR adds

From our discussion today:

• define a workflow registry as a distinct concept from modules, but follows similar patterns (spec file, publish/install mechanism)

• workflow registry encapsulates both "core workflows" and "subworkflows"

• pipeline developer should isolate core workflow and publish it to the workflow registry, then other pipelines can import it

• for users who want to daisy-chain pipelines with minimal effort -- just make a pipeline chain (e.g. nf-cascade, platform actions) and sacrifice dataflow concurrency

• define a workflow registry as a distinct concept from modules, but follows similar patterns (spec file, publish/install mechanism)

I'm still convinced that this isn't a distinct concept — it's the same entity at a different granularity, and the framing matters because it drives how much we end up building.

Logical view. A module is just a component with a well-defined input/output interface that hides its internals — it's self-contained. By that definition a process, a subworkflow, and a pipeline's core workflow are all modules; they differ only in granularity, not in kind. We already agreed upthread that "they're all basically the same thing." If the abstraction is identical, introducing a second top-level entity ("workflow") next to "module" creates a distinction the type system and the user's mental model don't actually need.

Tactical view. Making it a distinct concept means re-implementing the entire surface twice — for module and for workflow — for operations that are literally identical:

create      Create a new skeleton
install     Install from the registry
run         Run directly from the registry
list        List installed
remove      Remove an installed entry
search      Search the registry
view        Show info and usage template
publish     Publish to the registry
spec        Generate a meta.yml spec
validate    Validate structure and metadata

That's two CLIs, two sets of registry APIs, two spec formats, two sets of Platform integrations, two docs trees — to express what is, behaviorally, one operation set over one kind of artifact. The cost is recurring (every new capability has to be added in both places and kept in sync), not one-off.

Proposal. Keep a single registry and a single command group over one artifact: the module. If we need to differentiate a coarse-grained "core workflow" from a fine-grained process, do it with a type/kind field in the spec (and resolution conventions), not with a parallel system. The publish/install/spec mechanism is shared by construction; the differences (e.g. a core workflow vendoring its own modules/) become metadata and layout conventions on the same entity rather than a second concept.

This also keeps the door open for #7208 (running named workflows directly) without forking the model — "run a module" and "run a workflow" stay the same command.

Note, my previous comment is based on the assumption is focused on sub-workflow only, not plain pipelines.