The build failure has to be OSX specific, because I compiled and tested 3.2.57(1)-release on Ubuntu and could not reproduce the failure.

Weird. 3.2.57(1)-release on OSX in Travis is now failing with the same errors I saw on 4.2.48(1)-release on Amazon Linux. I'll look at this more tomorrow, when I've had a bit more sleep.

So I have some news, some bad news, and some good news. 😉 If the explanation of the bad news is too boring, you can skip straight to the good news at the very end.

The news is, you can use emit_debug_output (from test/test_helper.bash) instead of echo "$output". Does the same thing, essentially, but is easier to grep for. (Also, you'll need to remove echo "$output" from the final PR.)

The bad news is, you're encountering the phenomenon that led me to abandon the pursuit of an alternative to #24 for detecting [[ ]] failures under 3.2.57(1)-release (macOS) this weekend. Consider:

$ if [[ 'foo' == 'foo' ]]; then printf 'TRUE: %d\n' "$?"; else printf 'FALSE: %d\n' "$?"; fi
TRUE: 0

$ if [[ 'foo' == 'bar' ]]; then printf 'TRUE: %d\n' "$?"; else printf 'FALSE: %d\n' "$?"; fi
FALSE: 1

In other words, while conditional expression failures within an if, elif, while, until, etc. statement won't trigger the ERR trap (per the documentation for set -e in the man page), it will still set $? as "the most recently executed foreground pipeline". (Surprised the hell out of me!)

This screwed up my idea that bats_debug_trap could check "$?" to detect [[ ]] failures that didn't trigger the ERR trap. It's the reason why the first line of bats_error_trap is local status="$?". It's now screwing up the current implementation in this PR that tries call bats_error_trap inside a conditional.

The good news is I've reproduced the problem locally, and the solution is trivial. First, revert to the original code.

First, add BATS_ERROR_STATUS= in the same block in bats_perform_test that wipes other variables, namely:

BATS_TEST_COMPLETED=""
BATS_TEARDOWN_COMPLETED=""
BATS_ERROR_STATUS=

Then update the following line in bash_error_trap from:

BATS_ERROR_STATUS="$status"

to:

BATS_ERROR_STATUS="${BATS_ERROR_STATUS:-$status}"

Setting BATS_ERROR_STATUS= in bats_perform_test avoids having this line trigger unbound variable errors, and also ensures that the BATS_ERROR_STATUS from a previous test doesn't accidentally carry over into the current one.

Extra notes explaining why we shouldn't short-circuit on BATS_ERROR_STATUS

Before posting this, I also tried updating the conditional in bats_error_trap immediately above this line from:

if [[ -z "$BATS_TEST_COMPLETED" ]]; then

to:

if [[ -z "$BATS_TEST_COMPLETED" && -z "$BATS_ERROR_STATUS" ]]; then

By itself this caused "referencing unset parameter in setup produces error output" and "referencing unset parameter in test produces error output" to fail, because BATS_ERROR_TRAP wasn't properly wiped by bats_perform_test. So to fix that, I added the BATS_ERROR_TRAP= line to bats_perform_test.

All the Bats tests passed, but then when I ran my mbland/go-script-bash tests, a bunch of assertions failed. Short version: My assertion library was broken, because the -z "$BATS_ERROR_STATUS" made it so that I couldn't compose assertions from other assertions, since bats_error_trap would then only ever fire once.

The back story on the assertion library: I figured out how to clean up BATS_ERROR_STACK_TRACE such that it would always point directly to the line in the test that failed, not the line in the assertion. But when composing assertions from other assertions, having bats_error_trap fire only once meant that the stack trace wouldn't get cleaned up properly. (I'm hoping to make the helpers for writing such assertions either a standalone package or part of bats-core.)

The bottom line is, there's a real use case for enabling bats_error_trap to fire multiple times. Setting BATS_ERROR_STATUS as proposed above both fixes the problem at hand, while ensuring composable assertions (and composable, efficient Bats helper functions in general, really) remain possible.

Thanks for the detailed write-up! That's definitely what I was trying for, but I was doing it on fumes after my kid had a really rough meltdown from lack of sleep.

And thanks for the follow-up regarding the need for multiple firings of the error trap. I don't have enough test suites locally to have encountered that (none of mine goes deep enough to have assertions that rely on other assertions). The changes you suggested do indeed resolve the issue on Amazon Linux.

I've been giving some thought to how to test multiple Bash versions, and will see about putting something together for improved test coverage in the near future.

Also, I'm going to try setting up Travis and AppVeyor to work on my fork of the repo so I can stop cluttering the project's commit history with my monkeying around.

Before, I merge, one last comment: I just ran the current test suite again under 4.1.17, 4.2, 4.2.46, 4.2.53, and 4.3. Only the 4.2 versions triggered the "failure with significant status" failure.

I also instrumented the Bats code to observe the value of $? upon entering bats_error_trap (a.k.a. $status). In the non-4.2 versions, $? was set to 2 both times (when it fired directly as an ERR trap, and when it was fired by bats_teardown_trap). In the 4.2 versions, it was set to 2 the first time, and 0 the second.

It's not obvious to me from https://tiswww.case.edu/php/chet/bash/CHANGES or from diffing the sources of 4.2.53 and 4.3 what might've been the root cause and fix. Were the bug fixed at 4.2.53 or earlier, it'd be easy to narrow down the patch file from https://ftp.gnu.org/gnu/bash/. Sadly, there are no individual patch files between the last patch level and the next minor version, and neither does https://git.savannah.gnu.org/cgit/bash.git appear to show the changes, either.

Just wanted that on the record first.