Change stdout outputs to stderr, to not break piped output#7121
Conversation
✅ Deploy Preview for nextflow-docs-staging ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
See #7116 Signed-off-by: Phil Ewels <[email protected]>
jorgee
left a comment
There was a problem hiding this comment.
Overall, it looks fine with me.
Some concerns:
-
It could be a visible change for the user; shouldn't we warn about it? Is the merge commit message included in the changelog enough, or is anything else needed in the docs?
-
I think it could also be good to run e2e test just in case they parse one of the changed messages from stdout to avoid strange failures in the future.
-
Missing regression test checking the issue case: A small integration test that runs
nextflow inspect -o jsonon a trivial pipeline and asserts stdout parses as valid JSON would lock this in and prevent regressions on the no-observer log-fallback path (which is the easiest one to accidentally re-break).
Locks in that `nextflow inspect -format json` only writes JSON to stdout so the output stays pipe-safe (e.g. `| jq`). The no-observer log fallback in LoggerHelper is the easiest path to accidentally re-break. See #7116 Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> Signed-off-by: Phil Ewels <[email protected]>
|
Thanks @jorgee!
|
|
I think Changelog is probably enough here. At least from my understanding, this isn't a breaking change, but rather a visual fix. |
|
It's only breaking if people have automations / tests etc. set up that rely on checking for specific strings within Can always try to highlight it at the top of the changelog if we want? |
|
@jorgee - how do I trigger an e2e test / get this PR moving? |
|
Make a commit with |
Signed-off-by: Phil Ewels <[email protected]>
|
I think it worked..? :) |
|
You have to look up the platform runs to see if they actually succeeded. The CI check only verifies the launch. You can find the runs through the CI logs |
|
It seems that they run under Paolo's user space? https://cloud.seqera.io/user/pditommaso/watch/4IPHkGZiiLv4d2 I don't have access to that. I think, anyway.. Not entirely clear to me how this works - trying to figure it out now. |
|
I found the GitHub actions logic pretty confusing to follow, so have put in a separate PR to try to clean it up a bit: #7192 Also adds the ability to trigger tests with a comment and a manual dispatch (GitHub UI actions page dropdown), in addition to the existing commit message. Also prints the e2e platform URL and next steps on the CI actions summary. |
This comment was marked as outdated.
This comment was marked as outdated.
|
LGTM! ✅ |
Prevent messages such as
Monitor the execution with Seqera Platform using this URL:from breaking the JSON output, eg. fromnextflow inspect. Allows piping intojqand other tools.See #7116
Details
Route progress, banners and log fallbacks to stderr
Some Nextflow commands now emit machine-readable output on stdout (
nextflow inspect -o json,nextflow runoutputs, etc.). Today, log messages, progress UI, banners and spinners are also written to stdout, which makes that output difficult to pipe into downstream tools (jq, etc.).This change applies a simple rule: a command's main output stays on stdout; everything else (progress, status, banners, log fallbacks) moves to stderr. Human-readable command output (e.g.
--help,--version,nextflow info,nextflow view,nextflow list) is unaffected — it remains on stdout because it is the main output of those commands.Files changed
modules/nextflow/src/main/groovy/nextflow/util/LoggerHelper.groovy— the global logback appender'sSystem.outfallback (used when noAnsiLogObserveris registered, or the observer isn't started yet) now writes toSystem.err. Highest leverage change: affects log output from every command, including the "Monitor the execution with Seqera Platform" sticky message that breaksnextflow inspect | jqtoday.modules/nextflow/src/main/groovy/nextflow/trace/AnsiLogObserver.groovy— progress rendering, summary block, and info/warning/error UI panels now useAnsiConsole.errinstead ofAnsiConsole.out.AnsiConsole.systemInstall()is unchanged.modules/nextflow/src/main/groovy/nextflow/cli/CmdRun.groovy— theN E X T F L O W ~ version …banner and theLaunching … [run-name] - revision: …line now useAnsiConsole.err.modules/nextflow/src/main/groovy/nextflow/util/SpinnerUtil.groovy— animated spinners (and the non-ANSI plaintext fallback instopWithMessage) now write to stderr.modules/nextflow/src/main/groovy/nextflow/trace/AgentLogObserver.groovy— class-level docstring corrected to say "standard output" instead of "standard error". The implementation, the integration tests, and the design intent all agree that the structured[PIPELINE]/[WORKDIR]/[SUCCESS]lines emitted inNXF_AGENT_MODEare the meaningful programmatic output of agent mode and belong on stdout; only the docstring was wrong.Deliberately unchanged
Session.printConsole()stays on stdout. It is the data path used by theview()channel operator,echo truetask stdout forwarding, and the workflowoutput {}DSL block — i.e. exactly the kind of programmatic output stdout is for.AgentLogObserveroutput streams — agent-mode structured output ([PIPELINE],[WORKDIR],[SUCCESS],[ERROR],[FAILED], etc.) stays on stdout. InNXF_AGENT_MODEthese lines are the main output, designed to be parsed by AI tooling.--help,--version,nextflow info,nextflow view,nextflow list,nextflow clean,nextflow clone,nextflow plugin create,nextflow log,nextflow lineage,nextflow inspect,nextflow config,nextflow lint,nextflow fs,nextflow secrets,wave, etc. — these all already write their main output on stdout and are unaffected.nf-towerLaunchCommandImpl—nextflow launchcurrently mixes its banner and the workflow ID on stdout. Left for a follow-up that can introduce a JSON output mode.nf-towerAuthCommandImpl— interactive prompts and confirmations stay on stdout (these commands aren't piped).Net effect
nextflow inspect | jq,nextflow run … | …(and equivalents for other pipeable commands) no longer have stdout polluted by progress UI, banners, spinners, or log fallbacks. The "Monitor the execution with Seqera Platform using this URL: …" sticky message in particular — which is emitted vialog.info(STICKY, …)from the Tower observer — now correctly lands on stderr in commands likenextflow inspectbecause it falls through the no-observer path inLoggerHelper.Verification
./gradlew :nextflow:compileGroovypasses.agent-output.nfandagent-output-error.nfintegration tests pass:[PIPELINE]/[WORKDIR]/[SUCCESS]lines fromAgentLogObserverandWorld received: Hello from agent modefromSession.printConsole()continue to land on stdout, as the tests assert.