Thanks to visit codestin.com
Credit goes to github.com

Skip to content

JUnit formatter reported last skipped test as failure if teardown_suite logs to fd3 #1181

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Peter-Darton-i2 wants to merge 4 commits into
base: master
Choose a base branch
from
Open

JUnit formatter reported last skipped test as failure if teardown_suite logs to fd3 #1181

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
55993f0
Ensure we only output a test-failure when the test failed
Peter-Darton-i2 Dec 3, 2025
6787d2f
Add unit-test suite that demonstrates the issue
Peter-Darton-i2 Dec 10, 2025
03f7f7d
Reduce unnecessary changes
Peter-Darton-i2 Dec 11, 2025
31a4593
Add unit-test to demonstrate the issue
Peter-Darton-i2 Dec 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions libexec/bats-core/bats-format-junit
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,8 +74,13 @@ print_test_case() {
if [[ -n "$_system_out_log" ]]; then
printf " <system-out>%s</system-out>\n" "$(xml_escape "${_system_out_log}")"
fi
if [[ -n "$_buffer_log" || "$test_result_state" == not_ok ]]; then
if [[ "$test_result_state" == not_ok ]]; then # Failed tests need a <failure> element.
# If we have system err output, we use that as the failure text.
printf " <failure type=\"failure\">%s</failure>\n" "$(xml_escape "${_buffer_log}")"
else
if [[ -n "$_buffer_log" ]]; then
printf " <system-err>%s</system-err>\n" "$(xml_escape "${_buffer_log}")"
fi
Comment on lines +77 to +83
Copy link
Contributor Author

@Peter-Darton-i2 Peter-Darton-i2 Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the core of my thinking here: only "not_ok" tests should report a <failure... element.
Everything else should say it's ok while recording as much of the test's output as system-out / system-err as possible.

fi
if [[ "$test_result_state" == skipped ]]; then
printf " <skipped>%s</skipped>\n" "$(xml_escape "$test_skip_message")"
Expand Down Expand Up @@ -127,7 +132,7 @@ flush() {
_buffer=""
}

log() {
log_system_err() {
if [[ -n "$_buffer_log" ]]; then
_buffer_log="${_buffer_log}
$1"
Expand Down Expand Up @@ -157,7 +162,7 @@ $1"
finish_file() {
if [[ "${class}" != JUNIT_FORMATTER_NO_FILE_ENCOUNTERED ]]; then
file_header
printf "%s\n" "${_buffer}"
printf "%s" "${_buffer}"
file_footer
class=''
name=''
Expand Down Expand Up @@ -226,7 +231,7 @@ bats_tap_stream_comment() { # <comment text without leading '# '> <scope>
;;
*)
# everything else is considered error output
log "$1"
log_system_err "$1"
;;
esac
}
Expand All @@ -239,7 +244,8 @@ bats_tap_stream_suite() { # <file name>
}

bats_tap_stream_unknown() { # <full line>
:
local line="$1" scope="$2"
log_system_out "$line"
Comment on lines +247 to +248
Copy link
Contributor Author

@Peter-Darton-i2 Peter-Darton-i2 Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure of the wisdom of these two lines.

}

main() {
Expand Down
27 changes: 27 additions & 0 deletions test/fixtures/junit-formatter/issue1180/referenced_connectors_tests.bats
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#!/usr/bin/env bats
# shellcheck shell=bash

function setup_file() {
skip
}

function teardown_file() {
echo "normal teardown_file stdout"
echo "# Hash teardown_file stdout"
echo "normal teardown_file stderr" >&2
echo "# Hash teardown_file stderr" >&2
echo "normal teardown_file fd3" >&3
echo "# Hash teardown_file fd3" >&3
}

@test "skipped-and-junit-agrees" {
echo "unexpected stdout as this should be skipped"
echo "unexpected stderr as this should be skipped" >&2
echo "unexpected fd3 as this should be skipped" >&3
}

@test "skipped-but-junit-reports-failure" {
echo "unexpected stdout as this should be skipped"
echo "unexpected stderr as this should be skipped" >&2
echo "unexpected fd3 as this should be skipped" >&3
}
14 changes: 14 additions & 0 deletions test/fixtures/junit-formatter/issue1180/setup_suite.bash
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# shellcheck shell=bash

function setup_suite() {
true
}

teardown_suite() {
echo "normal teardown_suite stdout"
echo "# Hash teardown_suite stdout"
echo "normal teardown_suite stderr" >&2
echo "# Hash teardown_suite stderr" >&2
echo "normal teardown_suite fd3" >&3
echo "# Hash teardown_suite fd3" >&3
Comment on lines +12 to +13
Copy link
Contributor Author

@Peter-Darton-i2 Peter-Darton-i2 Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without this PR, this output causes the last test in the suite to be falsely-flagged as a failure by the JUnit formatter.

}
9 changes: 9 additions & 0 deletions test/junit-formatter.bats
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,15 @@ TESTSUITES_REGEX="<testsuites time=\"$FLOAT_REGEX\">"
[[ "${lines[7]}" == '</testsuite>' ]]
}

@test "junit does not mark tests with FD 3 output in teardown_suite as failed (issue #1180)" {
bats_require_minimum_version 1.5.0
local stderr='' # silence shellcheck
name=non-empty reentrant_run -0 --separate-stderr bats --formatter junit "$FIXTURE_ROOT/issue1180"
[ "${stderr}" == "" ] || { echo "stderr should be empty but was: ${stderr}" >&3; return 1; }
[[ "${output}" != *'<failure '* ]]
[[ "${output}" == *'teardown_suite fd3'* ]]
Copy link
Contributor Author

@Peter-Darton-i2 Peter-Darton-i2 Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not the "tightest" of tests, but it seems to do the job.
The important thing is that there should be no failure elements if the bats tests passed.
A nice-to-have is to see the logged output in the report.

}

@test "don't choke on setup_file errors" {
bats_require_minimum_version 1.5.0
local stderr='' # silence shellcheck
Expand Down
Loading