Verbose output for faster troubleshooting of test problems #488

NorseGaud · 2021-08-20T20:19:50Z

I have reviewed the Contributor Guidelines.
I have reviewed the Code of Conduct and agree to abide by it

Original PR: #467

bats --trace (or via $BATS_TRACE_LEVEL=1 env var, this name might be subject to change): print executed commands
bats --print-output-on-failure (or via $BATS_PRINT_OUTPUT_ON_FAILURE env var): print the value of "$output" when a test fails
bats --show-output-of-succeeding-tests (or via $BATS_SHOW_OUTPUT_OF_SUCCEEDING_TESTS env var)
bats --verbose-run (or via $BATS_VERBOSE_RUN env var): make run print $output by default
bats --gather-test-outputs-in <dir> (or via $BATS_GATHER_TEST_OUTPUTS_IN=D<dir>, but then you have to ensure the directory exists): put output of each test (even succeeding!) into the given folder as files of the format <test number in suite>-<test description>.log

they would fail in the dockerized CI tests

Detecting the differing behavior seems not worth it, right now

to avoid --trace output from helper libraries

They will generate trace output too, so we need to use an extra FD to avoid altering the substution result. Redirecting the trace FD into BATS_OUT allows for interleaving of commands and their output.

NorseGaud · 2021-08-20T20:20:03Z

@martin-schulze-vireso Good opportunity to switch to a proper branch :)

martin-schulze-vireso · 2021-08-23T18:39:16Z

This branch was already on master, could you just hard reset to the branch bats-core/trace_and_more?

NorseGaud · 2021-08-24T12:55:46Z

Found another issue using the code:

function private_config_path() {
    echo "${PWD}/test/duplicate_configs/${1:-$(basename "${BATS_TEST_FILENAME}")}"
}

# @param ${1} relative source in test/config folder
# @param ${2} (optional) container name, defaults to ${BATS_TEST_FILENAME}
# @return path to the folder where the config is duplicated
function duplicate_config_for_container() {
    local OUTPUT_FOLDER
    OUTPUT_FOLDER="$(private_config_path "${2}")"  || return $?
    rm -rf "${OUTPUT_FOLDER:?}/" || return $? # cleanup
    mkdir -p "${OUTPUT_FOLDER}" || return $?
    cp -r "${PWD}/test/config/${1:?}/." "${OUTPUT_FOLDER}" || return $?
    echo "${OUTPUT_FOLDER}"
}

@test "duplicate_config_for_container" {
    local path
    path="$(duplicate_config_for_container duplicate_config_test)"
    
    run cat "${path}/marker"
    assert_line "This marker file is there to identify the correct config being copied"

    run duplicate_config_for_container non-existant-source-folder "${BATS_TEST_NAME}2"
    assert_failure
}

❯ test/bats/bin/bats --trace --verbose-run --filter "duplicate_config_for_container" test/test_helper.bats
 ✗ duplicate_config_for_container
   (in test file test/test_helper.bats, line 120)
     `path="$(duplicate_config_for_container duplicate_config_test)"' failed
   $ [test_helper.bats:119]
   $ local path
   $ path="$(duplicate_config_for_container duplicate_config_test)"
   $ duplicate_config_for_container duplicate_config_test
   $$ [common.bash:158]
   $$ local OUTPUT_FOLDER
   $$ OUTPUT_FOLDER="$(private_config_path "${2}")"
   $$ private_config_path "${2}"
   $$$ [common.bash:151]
   $$$ echo "${PWD}/test/duplicate_configs/${1:-$(basename "${BATS_TEST_FILENAME}")}"
   $$$ basename "${BATS_TEST_FILENAME}"
   $$ rm -rf "${OUTPUT_FOLDER:?}/"
   $$ mkdir -p "${OUTPUT_FOLDER}"
   $$ cp -r "${PWD}/test/config/${1:?}/." "${OUTPUT_FOLDER}"
   $$ echo "${OUTPUT_FOLDER}"
   $ run cat "${path}/marker"
   cat: /Users/norsegaud/docker-mailserver/.git/modules/target/docker-configomat/modules/test/test_helper/bats-assert/marker: No such file or directory
   $ assert_line "This marker file is there to identify the correct config being copied"
   
   -- output does not contain line --
   line   : This marker file is there to identify the correct config being copied
   output : cat: /Users/norsegaud/docker-mailserver/.git/modules/target/docker-configomat/modules/test/test_helper/bats-assert/marker: No such file or directory
   --
   

1 test, 1 failure

It looks as if it's setting the wrong path but I'm not entirely sure.

martin-schulze-vireso · 2021-08-24T14:17:03Z

The path strikes me as rather odd. Why is it in the .git folder? Also, I don't see how this is related to the trace output. I assume the test passes without --trace?

NorseGaud · 2021-08-24T14:37:54Z

The path strikes me as rather odd. Why is it in the .git folder? Also, I don't see how this is related to the trace output. I assume the test passes without --trace?

❯ test/bats/bin/bats --verbose-run --filter "duplicate_config_for_container" test/test_helper.bats 
 ✗ duplicate_config_for_container
   (from function `assert_line' in file test/test_helper/bats-assert/src/assert.bash, line 508,
    in test file test/test_helper.bats, line 123)
     `assert_line "This marker file is there to identify the correct config being copied"' failed
   cat: /Users/norsegaud/docker-mailserver/test/bats/libexec//marker: No such file or directory
   
   -- output does not contain line --
   line   : This marker file is there to identify the correct config being copied
   output : cat: /Users/norsegaud/docker-mailserver/test/bats/libexec//marker: No such file or directory
   --
   

1 test, 1 failure

❯ test/bats/bin/bats --filter "duplicate_config_for_container" test/test_helper.bats
 ✗ duplicate_config_for_container
   (from function `assert_line' in file test/test_helper/bats-assert/src/assert.bash, line 508,
    in test file test/test_helper.bats, line 123)
     `assert_line "This marker file is there to identify the correct config being copied"' failed
   
   -- output does not contain line --
   line   : This marker file is there to identify the correct config being copied
   output : cat: /Users/norsegaud/docker-mailserver/test/bats/libexec//marker: No such file or directory
   --
   

1 test, 1 failure

If I try in the official CI, I see

not ok 213 duplicate_config_for_container
# (in test file test/test_helper.bats, line 120)
#   `path="$(duplicate_config_for_container duplicate_config_test)"' failed
# $ [test_helper.bats:119]
# $ local path
# $ path="$(duplicate_config_for_container duplicate_config_test)"
# $ duplicate_config_for_container duplicate_config_test
# $$ [common.bash:158]
# $$ local OUTPUT_FOLDER
# $$ OUTPUT_FOLDER="$(private_config_path "${2}")"
# $$ private_config_path "${2}"
# $$$ [common.bash:151]
# $$$ echo "${PWD}/test/duplicate_configs/${1:-$(basename "${BATS_TEST_FILENAME}")}"
# $$$ basename "${BATS_TEST_FILENAME}"
# $$ rm -rf "${OUTPUT_FOLDER:?}/"
# $$ mkdir -p "${OUTPUT_FOLDER}"
# $$ cp -r "${PWD}/test/config/${1:?}/." "${OUTPUT_FOLDER}"
# $$ echo "${OUTPUT_FOLDER}"
# $ run cat "${path}/marker"
# cat: /home/runner/work/docker-mailserver/docker-mailserver/test/test_helper/bats-support/marker: No such file or directory
# $ assert_line "This marker file is there to identify the correct config being copied"
# 
# -- output does not contain line --
# line   : This marker file is there to identify the correct config being copied
# output : cat: /home/runner/work/docker-mailserver/docker-mailserver/test/test_helper/bats-support/marker: No such file or directory
# --

Different path, but same issue. Maybe some sort of ENV problem.

The bats submodule is checked out to commit 758698f

If I switch to 1.4.1 it works just fine. I'll keep playing around with it too and see if I can get more precise on what is actually wrong/happening.

NorseGaud · 2021-08-25T12:51:45Z

Turns out it had to do with the use of path as a local variable in the test. Using location instead allows it to pass.

I'm not seeing where in the changes this could have started being an issue though. I'm ok keeping it changed for now.

lib/bats-core/tracing.bash

martin-schulze-vireso · 2021-08-26T19:19:34Z

I think I found a fix for that issue with $path. Can you give it a spin @NorseGaud ?

NorseGaud · 2021-08-26T22:54:45Z

It's perfect! Works great now! ❤️ 🎉

Note that file descriptors 3 and 4 are reserved by Bats. The former is used for adding custom text to the Test Anything Protocol (or TAP) stream [1] and the latter for tracing. [1] https://bats-core.readthedocs.io/en/stable/writing-tests.html#file-descriptor-3-read-this-if-bats-hangs https://bats-core.readthedocs.io/en/stable/writing-tests.html#printing-to-the-terminal [2] Bats commit 635700cd2282b754 bats-core/bats-core#467 bats-core/bats-core#488 containers#1066

Note that file descriptors 3 and 4 are reserved by Bats. The former is used for adding custom text to the Test Anything Protocol (or TAP) stream [1] and the latter for tracing [2]. [1] https://bats-core.readthedocs.io/en/stable/writing-tests.html#file-descriptor-3-read-this-if-bats-hangs https://bats-core.readthedocs.io/en/stable/writing-tests.html#printing-to-the-terminal [2] Bats commit 635700cd2282b754 bats-core/bats-core#467 bats-core/bats-core#488 containers#1066

NorseGaud and others added 23 commits August 20, 2021 21:47

--verbose

9b0cfde

Revise --trace to use debug_trap

95a90a8

Add --print-output--on-failure

df8a3e6

Remove unused TRACE_LEVEL code

30c0af3

Fix test cases

491bd2e

Add --show-output-of-succeeding-tests

26eb1b6

Add --verbose-run

d4ec199

Add --gather-test-outputs-in

b927529

pretty: Colorize summary (refs bats-core#163, bats-core#444)

0dd8f31

Fix shellcheck warnings

250d9b1

Fix relative paths in test expectations

3c2a446

they would fail in the dockerized CI tests

Improve comments

06968ab

Remove test that is implicitly done by next test

330535c

--trace: fix test expectations pr bash version

2d69be1

Detecting the differing behavior seems not worth it, right now

Fix typo

9fce806

Add BATS_DEBUG_EXCLUDE_PATH and bats_add_debug_exclude_path

e0b4178

to avoid --trace output from helper libraries

--trace: Only print file markers on stack push/pop

e8f9281

Fix shellcheck/undefined variable access

9146031

--trace: Fix test expectation

6bf09e7

--trace: Fix error in command substitutions

635700c

They will generate trace output too, so we need to use an extra FD to avoid altering the substution result. Redirecting the trace FD into BATS_OUT allows for interleaving of commands and their output.

pretty: Don't color output of passing tests red

6e6929e

Rename BATS_TEST_COMMAND to BATS_RUN_COMMAND

8e3c458

Fix mistaken PR reference

758698f

NorseGaud requested a review from a team as a code owner August 20, 2021 20:19

NorseGaud force-pushed the verbose-output branch from d5bcf7e to 758698f Compare August 23, 2021 22:31

martin-schulze-vireso requested changes Aug 25, 2021

View reviewed changes

lib/bats-core/tracing.bash Show resolved Hide resolved

NorseGaud force-pushed the verbose-output branch 2 times, most recently from b11b65d to 758698f Compare August 26, 2021 10:50

Don't leak/overwrite common variables from debug trap

387fdfe

martin-schulze-vireso force-pushed the verbose-output branch from 0309012 to 387fdfe Compare August 26, 2021 19:23

martin-schulze-vireso added a commit to NorseGaud/bats-core that referenced this pull request Aug 27, 2021

Add changelog entry for bats-core#488

a5660f0

martin-schulze-vireso added 3 commits August 27, 2021 21:18

Rename succeeding to passing in --show-output-of-succeeding-tests

69449d8

Document new flags

cf19d8e

Add changelog entry for bats-core#488

cce9e74

martin-schulze-vireso force-pushed the verbose-output branch from a5660f0 to cce9e74 Compare August 27, 2021 19:18

martin-schulze-vireso approved these changes Aug 27, 2021

View reviewed changes

martin-schulze-vireso merged commit 8e9f35a into bats-core:master Aug 27, 2021

NorseGaud deleted the verbose-output branch August 28, 2021 14:58

This was referenced Oct 23, 2021

is there a way to see test log of passed tests? #267

Closed

Add a test trace option #136

Closed

Add a $command variable that captures arguments passed to run #163

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Verbose output for faster troubleshooting of test problems #488

Verbose output for faster troubleshooting of test problems #488

Uh oh!

NorseGaud commented Aug 20, 2021 •

edited

Loading

Uh oh!

NorseGaud commented Aug 20, 2021 •

edited

Loading

Uh oh!

martin-schulze-vireso commented Aug 23, 2021

Uh oh!

NorseGaud commented Aug 24, 2021

Uh oh!

martin-schulze-vireso commented Aug 24, 2021

Uh oh!

NorseGaud commented Aug 24, 2021 •

edited

Loading

Uh oh!

NorseGaud commented Aug 25, 2021

Uh oh!

Uh oh!

martin-schulze-vireso commented Aug 26, 2021

Uh oh!

NorseGaud commented Aug 26, 2021 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Verbose output for faster troubleshooting of test problems #488

Verbose output for faster troubleshooting of test problems #488

Uh oh!

Conversation

NorseGaud commented Aug 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NorseGaud commented Aug 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

martin-schulze-vireso commented Aug 23, 2021

Uh oh!

NorseGaud commented Aug 24, 2021

Uh oh!

martin-schulze-vireso commented Aug 24, 2021

Uh oh!

NorseGaud commented Aug 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NorseGaud commented Aug 25, 2021

Uh oh!

Uh oh!

martin-schulze-vireso commented Aug 26, 2021

Uh oh!

NorseGaud commented Aug 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

NorseGaud commented Aug 20, 2021 •

edited

Loading

NorseGaud commented Aug 20, 2021 •

edited

Loading

NorseGaud commented Aug 24, 2021 •

edited

Loading

NorseGaud commented Aug 26, 2021 •

edited

Loading