Improve ginkgo test failure readability. #1999
Conversation
When developers are trying to quickly iterate on tests or determine the cause of runtime test failures, the Cilium logs are important but not the first port of call. With long-running VMs, the Cilium log can be thousands of lines of output between the actual error reported by Ginkgo and where the terminal stops. Rather than print the entire Cilium log, just print a simple message describing how the developer/tester can retrieve the Cilium logs, if necessary. Signed-off-by: Joe Stringer <[email protected]>
|
@eloycoto I welcome some discussion of what you'd like to see in terms of making the Cilium logs accessible in the CI, I think @ianvernon also had some discussion that they should be separate artifact files. |
|
My two cents; I am against dumping a bunch of stuff to console. It makes it hard to parse. Having the output of individual commands written to individual files / having logs published into artifacts that are picked up by Jenkins is much easier. Jenkins should specify what test failed, and not clog console with a bunch of metadata, especially since the Cilium logs can be quite verbose as we run the daemon in debug mode in the CI. |
|
I think this can be solved if you print the journalctl with this command
|
|
The suggestion from @aanm sounds like a pretty reasonable middle ground to me. |
|
@aanm suggestion should be ok for the small test, but in case of large test does it work? I mean for I think that we should consider what Ian said, and export the full logs too. At the moment we have a helper function to know if it is running on Jenkins |
|
@eloycoto my suggestion was for the "grep" to only be printed and the full log to be stored |
|
@eloycoto @joestringer @aanm @ianvernon What's the consensus here? |
|
I'm fine with having a subset of the logs from the test failure for now. However, we need all logs in the case that a test fails in case the subset provided isn't enough. #2026 covers all logs we need from tests. |
SGTM |
|
Superseded by cc12604. |
Split out from original patch in #1975 so that this change could be independently reviewed and debated.
One of the pain points for developers using the runtime testsuites is that the directly relevant information about the failure is split in half, with a giant Cilium log in the middle, eg:
When running the ginkgo tests locally, the Cilium logs get in the way of seeing the actual error, for example
Expected <bool> false to be true. In a lot of cases, the error itself may be revealing enough so that the developer working on the test can understand what the problem is without looking through the Cilium logs. Furthermore, when running locally, it's much easier for a developer to open the Cilium log separately in an editor that allows better search, highlight, etc. capabilities.There is currently a ginkgo PR open (onsi/ginkgo#383) to allow us to restrict the length of these Cilium logs to only those since the beginning of the current testrun (or, a particular point in time). This will independently be useful to allow us to filter out irrelevant logs from previous test runs. However, even with this kind of functionality, if we continue to print the logs directly to the terminal upon failure, then it is still possible for the Cilium logs to be very long, in which case we still have the developer pain point described above.
The proposed patch here replaces the printout of the actual log with a pointer about how to get the logs. It could be extended so that when running in a Jenkins environment, it gives different instructions (perhaps even with a link to the relevant path to fetch the logs, for example).