Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@mbland
Copy link
Contributor

@mbland mbland commented May 14, 2018

Closes #72 and #81. Both of those issues deal with the bats_error_trap executing with an incorrect value for $?. Bats exited with an error, but provided no stack trace information to pinpoint its source.

With credit to sstephenson, Bats was smart enough to know and to report that an error occurred, even when it couldn't tell exactly where the error came from.

Building on that existing mechanism, this change produces output for these failure cases even when bats_error_trap isn't called with the correct value for $?. It passes all the existing tests under Bash 3.2.57(1)-release and 4.4.19(1)-release. I also timed the original suites with and without the change to ensure the runtime cost was negligible.

A couple more notes:

  • This change obsoletes the suggestion in silent failure when accessing unbound variable with set -u #81 that a test case to validate testing code running under set -u is necessary. run() disables set -e for the code under test to begin with, sidestepping the problem with set -eu interactions in Bash versions prior to 4.1.0.

  • I've noticed that this mechanism does not work when the problematic behavior is in teardown(). The change preserves existing behavior, the remedy for that issue requires further thinking (and a new issue).

cc: @cpmills1975 @tterranigma @eedwards-sk

Closes #72 and #81. Both of those issues deal with the `bats_error_trap`
executing with an incorrect value for `$?`. Bats exited with an error,
but provided no stack trace information to pinpoint its source.

* #72 deals with the inconsistent failure-mode behavior of the `source`
  builtin between Bash versions. I believe it's feasible to update
  `load` with robust error checking around `source` and strongly
  recommend its use in Bats test files instead of using `source`
  directly. I've already opened #80 to track this.

* #81 deals with inconsistent failure-mode behavior of `set -u`,
  particularly when code within a Bats test file itself references an
  unbound variable.

With credit to sstephenson, Bats was smart enough to know and to report
that an error occurred, even when it couldn't tell exactly where the
error came from.

Building on that existing mechanism, this change produces output for
these failure cases even when `bats_error_trap` isn't called with the
correct value for `$?`. It passes all the existing tests under Bash
3.2.57(1)-release and 4.4.19(1)-release. I also timed the original
suites with and without the change to ensure the runtime cost was
negligible.

A couple more notes:

* This change obsoletes the suggestion in #81 that a test case to
  validate testing code running under `set -u` is necessary. `run()`
  disables `set -e` for the code under test to begin with, sidestepping
  the problem with `set -eu` interactions in Bash versions prior to
  4.1.0.

* I've noticed that this mechanism does _not_ work when the problematic
  behavior is in `teardown()`. The change preserves existing behavior,
  the remedy for that issue requires further thinking (and a new issue).
@mbland mbland self-assigned this May 14, 2018
@mbland mbland requested a review from a team May 14, 2018 04:17
@ghost ghost added the review label May 14, 2018
This enables Bats to report errors in `teardown()` due to one of its
commands triggering the EXIT trap without setting `$?` properly. This is
accomplished by calling `bats_teardown_trap` manually whenever the test
case (including `setup()`) runs to completion, i.e. whenever the test
case doesn't trigger an error that triggers the EXIT trap.

This will break down when both the test case (including `setup()`) _and_
`teardown()` trigger errors that both trigger the EXIT trap, because a
running EXIT trap cannot trigger another EXIT trap by design. In this
case, `bats_exit_trap` will not get called, truncating the test output.
@mbland
Copy link
Contributor Author

mbland commented May 14, 2018

Just pushed a new commit that will report similar errors in teardown(), provided that neither setup() nor the test case itself previously triggered an error that triggered the EXIT trap. See the comments for commit 1b77683 for further details.

This is actually a more reliable way to check for an error than checking
the value of `$?`, since `$?` can sometimes be zero even after an error
occurred.

The updated test cases failed before the code change was applied, and
pass after.

Also updated the comment to try to be clearer about the fact that the
test will generate no failure output when an EXIT trigger happens while
`bats_teardown_trap` executes as the EXIT trap.
@mbland mbland force-pushed the handle-exit-with-err branch from 6132424 to 04ac77e Compare May 14, 2018 19:44
Using `"${status:-1}"` was left over from before I realized that
`$status` is always defined, even when it's zero, in which case the
expression would resolve to zero.
@mbland
Copy link
Contributor Author

mbland commented May 15, 2018

@jasonkarns @sublimino and @btamayo Life pulled me away from this and other projects for a while, but it seems I've regained the bandwidth to start incrementally moving things forward again.

If at least one of y'all could review this, I'd appreciate it. I think it can safely sneak in before our "release"—which I'd also like to finally get across the finish line soon.

@mbland
Copy link
Contributor Author

mbland commented May 22, 2018

Gang, I'm gonna go ahead and merge this. If there's any issues after-the-fact, I'm happy to file a new issue and address them later.

@mbland mbland merged commit 6cf4be3 into master May 22, 2018
@mbland mbland deleted the handle-exit-with-err branch May 22, 2018 14:31
@ghost ghost removed the review label May 22, 2018
@mbland mbland mentioned this pull request Jun 3, 2018
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants