Flakey / leaky integration tests #394
Description
When we run the tests as described in the contribution guide (ginkgo -r -p), we see that the test for the coverage integration tests are very flakey.
We think that is because of race condition in integration/coverage_test.go when tests run in parallel -- in AfterEach a fixtures directory is being removed while other tests, depending on that, are potentially running.
The error we see quite often is (similar to):
• Failure [3.115 seconds]
Coverage Specs [It] runs coverage analysis in series and in parallel
/Users/pivotal/go/src/github.com/onsi/ginkgo/integration/coverage_test.go:18
Expected error:
<*exec.ExitError | 0xc4200ec880>: {
ProcessState: {
pid: 86028,
status: 256,
rusage: {
Utime: {
Sec: 0,
Usec: 25735,
Pad_cgo_0: [128, 255, 255, 255],
},
Stime: {
Sec: 0,
Usec: 36515,
Pad_cgo_0: [0, 0, 0, 0],
},
Maxrss: 9101312,
Ixrss: 0,
Idrss: 0,
Isrss: 0,
Minflt: 11850,
Majflt: 3,
Nswap: 0,
Inblock: 0,
Oublock: 0,
Msgsnd: 0,
Msgrcv: 0,
Nsignals: 3,
Nvcsw: 374,
Nivcsw: 735,
},
},
Stderr: nil,
}
exit status 1
not to have occurred
/Users/pivotal/go/src/github.com/onsi/ginkgo/integration/coverage_test.go:32
------------------------------
The revision we have seen this in is 7a241e9, we believe that this problem already exists longer, but we didn't triage it.
How to reproduce:
- run
ginkgo -r --randomizeAllSpecs --randomizeSuites --race --trace -p -untilItFails -focus 'Coverage' integration/in the root of the project - See that the tests fail pretty fast (for us consistently on 1st or 2nd run currently)
- run
ginkgo -r --randomizeAllSpecs --randomizeSuites --race --trace -p -untilItFails integration/(without focusing on the coverage tests) - See that the tests fail after a couple of runs, not as consistent as the one above
- run
ginkgo -r --randomizeAllSpecs --randomizeSuites --race --trace -untilItFails -focus 'Coverage' integration/(without-pbut focusing on the coverage tests) - See that the tests succeed consistently