JUnit formatter reported last skipped test as failure if teardown_suite logs to fd3 #1181
Conversation
| echo "normal teardown_suite fd3" >&3 | ||
| echo "# Hash teardown_suite fd3" >&3 |
There was a problem hiding this comment.
Without this PR, this output causes the last test in the suite to be falsely-flagged as a failure by the JUnit formatter.
| name=non-empty reentrant_run -0 --separate-stderr bats --formatter junit "$FIXTURE_ROOT/issue1180" | ||
| [ "${stderr}" == "" ] || { echo "stderr should be empty but was: ${stderr}" >&3; return 1; } | ||
| [[ "${output}" != *'<failure '* ]] | ||
| [[ "${output}" == *'teardown_suite fd3'* ]] |
There was a problem hiding this comment.
This is not the "tightest" of tests, but it seems to do the job.
The important thing is that there should be no failure elements if the bats tests passed.
A nice-to-have is to see the logged output in the report.
| local line="$1" scope="$2" | ||
| log_system_out "$line" |
There was a problem hiding this comment.
I am not sure of the wisdom of these two lines.
| if [[ "$test_result_state" == not_ok ]]; then # Failed tests need a <failure> element. | ||
| # If we have system err output, we use that as the failure text. | ||
| printf " <failure type=\"failure\">%s</failure>\n" "$(xml_escape "${_buffer_log}")" | ||
| else | ||
| if [[ -n "$_buffer_log" ]]; then | ||
| printf " <system-err>%s</system-err>\n" "$(xml_escape "${_buffer_log}")" | ||
| fi |
There was a problem hiding this comment.
This is the core of my thinking here: only "not_ok" tests should report a <failure... element.
Everything else should say it's ok while recording as much of the test's output as system-out / system-err as possible.
Fixes #1180
...but I'd recommend that #1177 also be considered for merge at the same time; these seem related.