Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Change stdout outputs to stderr, to not break piped output#7121

Merged
bentsherman merged 6 commits into
masterfrom
stderr
Jun 8, 2026
Merged

Change stdout outputs to stderr, to not break piped output#7121
bentsherman merged 6 commits into
masterfrom
stderr

Conversation

@ewels

@ewels ewels commented May 8, 2026

Copy link
Copy Markdown
Member

Prevent messages such as Monitor the execution with Seqera Platform using this URL: from breaking the JSON output, eg. from nextflow inspect. Allows piping into jq and other tools.

See #7116

Details

Route progress, banners and log fallbacks to stderr

Some Nextflow commands now emit machine-readable output on stdout (nextflow inspect -o json, nextflow run outputs, etc.). Today, log messages, progress UI, banners and spinners are also written to stdout, which makes that output difficult to pipe into downstream tools (jq, etc.).

This change applies a simple rule: a command's main output stays on stdout; everything else (progress, status, banners, log fallbacks) moves to stderr. Human-readable command output (e.g. --help, --version, nextflow info, nextflow view, nextflow list) is unaffected — it remains on stdout because it is the main output of those commands.

Files changed

  • modules/nextflow/src/main/groovy/nextflow/util/LoggerHelper.groovy — the global logback appender's System.out fallback (used when no AnsiLogObserver is registered, or the observer isn't started yet) now writes to System.err. Highest leverage change: affects log output from every command, including the "Monitor the execution with Seqera Platform" sticky message that breaks nextflow inspect | jq today.
  • modules/nextflow/src/main/groovy/nextflow/trace/AnsiLogObserver.groovy — progress rendering, summary block, and info/warning/error UI panels now use AnsiConsole.err instead of AnsiConsole.out. AnsiConsole.systemInstall() is unchanged.
  • modules/nextflow/src/main/groovy/nextflow/cli/CmdRun.groovy — the N E X T F L O W ~ version … banner and the Launching … [run-name] - revision: … line now use AnsiConsole.err.
  • modules/nextflow/src/main/groovy/nextflow/util/SpinnerUtil.groovy — animated spinners (and the non-ANSI plaintext fallback in stopWithMessage) now write to stderr.
  • modules/nextflow/src/main/groovy/nextflow/trace/AgentLogObserver.groovy — class-level docstring corrected to say "standard output" instead of "standard error". The implementation, the integration tests, and the design intent all agree that the structured [PIPELINE]/[WORKDIR]/[SUCCESS] lines emitted in NXF_AGENT_MODE are the meaningful programmatic output of agent mode and belong on stdout; only the docstring was wrong.

Deliberately unchanged

  • Session.printConsole() stays on stdout. It is the data path used by the view() channel operator, echo true task stdout forwarding, and the workflow output {} DSL block — i.e. exactly the kind of programmatic output stdout is for.
  • AgentLogObserver output streams — agent-mode structured output ([PIPELINE], [WORKDIR], [SUCCESS], [ERROR], [FAILED], etc.) stays on stdout. In NXF_AGENT_MODE these lines are the main output, designed to be parsed by AI tooling.
  • --help, --version, nextflow info, nextflow view, nextflow list, nextflow clean, nextflow clone, nextflow plugin create, nextflow log, nextflow lineage, nextflow inspect, nextflow config, nextflow lint, nextflow fs, nextflow secrets, wave, etc. — these all already write their main output on stdout and are unaffected.
  • nf-tower LaunchCommandImplnextflow launch currently mixes its banner and the workflow ID on stdout. Left for a follow-up that can introduce a JSON output mode.
  • nf-tower AuthCommandImpl — interactive prompts and confirmations stay on stdout (these commands aren't piped).

Net effect

nextflow inspect | jq, nextflow run … | … (and equivalents for other pipeable commands) no longer have stdout polluted by progress UI, banners, spinners, or log fallbacks. The "Monitor the execution with Seqera Platform using this URL: …" sticky message in particular — which is emitted via log.info(STICKY, …) from the Tower observer — now correctly lands on stderr in commands like nextflow inspect because it falls through the no-observer path in LoggerHelper.

Verification

  • ./gradlew :nextflow:compileGroovy passes.
  • agent-output.nf and agent-output-error.nf integration tests pass: [PIPELINE]/[WORKDIR]/[SUCCESS] lines from AgentLogObserver and World received: Hello from agent mode from Session.printConsole() continue to land on stdout, as the tests assert.

@netlify

netlify Bot commented May 8, 2026

Copy link
Copy Markdown

Deploy Preview for nextflow-docs-staging ready!

Name Link
🔨 Latest commit 408b4a0
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/6a2717b2eb9dee000816edce
😎 Deploy Preview https://deploy-preview-7121--nextflow-docs-staging.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@jorgee jorgee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, it looks fine with me.

Some concerns:

  • It could be a visible change for the user; shouldn't we warn about it? Is the merge commit message included in the changelog enough, or is anything else needed in the docs?

  • I think it could also be good to run e2e test just in case they parse one of the changed messages from stdout to avoid strange failures in the future.

  • Missing regression test checking the issue case: A small integration test that runs nextflow inspect -o json on a trivial pipeline and asserts stdout parses as valid JSON would lock this in and prevent regressions on the no-observer log-fallback path (which is the easiest one to accidentally re-break).

Locks in that `nextflow inspect -format json` only writes JSON to
stdout so the output stays pipe-safe (e.g. `| jq`). The no-observer
log fallback in LoggerHelper is the easiest path to accidentally
re-break.

See #7116

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Signed-off-by: Phil Ewels <[email protected]>
@ewels

ewels commented May 26, 2026

Copy link
Copy Markdown
Member Author

Thanks @jorgee!

  • I leave the docs question to @christopher-hakkaart but for me, changelog is enough. This is basically reverting to what I suspect is expected behaviour for most people? But I don't feel strongly about this.
  • Sure - how do we do this on a PR?
  • Pushed in 66667f0

@christopher-hakkaart

Copy link
Copy Markdown
Collaborator

I think Changelog is probably enough here. At least from my understanding, this isn't a breaking change, but rather a visual fix.

@ewels

ewels commented May 27, 2026

Copy link
Copy Markdown
Member Author

It's only breaking if people have automations / tests etc. set up that rely on checking for specific strings within STDOUT Nextflow output (which will now become STDERR). I'm sure that this will affect some people, but I'm hoping not many. I think much more severe breaking changes have gone out in the past with no more than a changelog entry 😅 This is what the changelog is for, after all.

Can always try to highlight it at the top of the changelog if we want?

@ewels

ewels commented May 30, 2026

Copy link
Copy Markdown
Member Author

@jorgee - how do I trigger an e2e test / get this PR moving?

@bentsherman

Copy link
Copy Markdown
Member

Make a commit with [e2e prod]

@ewels

ewels commented May 30, 2026

Copy link
Copy Markdown
Member Author

I think it worked..? :)

@bentsherman

Copy link
Copy Markdown
Member

You have to look up the platform runs to see if they actually succeeded. The CI check only verifies the launch. You can find the runs through the CI logs

@ewels

ewels commented May 30, 2026

Copy link
Copy Markdown
Member Author

It seems that they run under Paolo's user space? https://cloud.seqera.io/user/pditommaso/watch/4IPHkGZiiLv4d2

I don't have access to that.

I think, anyway.. Not entirely clear to me how this works - trying to figure it out now.

@ewels

ewels commented May 30, 2026

Copy link
Copy Markdown
Member Author

I found the GitHub actions logic pretty confusing to follow, so have put in a separate PR to try to clean it up a bit: #7192

Also adds the ability to trigger tests with a comment and a manual dispatch (GitHub UI actions page dropdown), in addition to the existing commit message. Also prints the e2e platform URL and next steps on the CI actions summary.

@bentsherman bentsherman linked an issue Jun 8, 2026 that may be closed by this pull request
@pditommaso

This comment was marked as outdated.

@pditommaso

Copy link
Copy Markdown
Member

LGTM! ✅

@bentsherman bentsherman merged commit dd9c2eb into master Jun 8, 2026
23 checks passed
@bentsherman bentsherman deleted the stderr branch June 8, 2026 19:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nextflow inspect|config -o json doesn't return valid json

5 participants