-
Couldn't load subscription status.
- Fork 450
bats: support parallel execution of tests #172
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
74c9b1e to
a2cf056
Compare
Use the *not-yet-merged* parallelism support I've been working on in bats[1] in order to speed up our tests massively. This requires that we use my home project on OBS, but hopefully my patches will be merged eventually. [1]: bats-core/bats-core#172 Signed-off-by: Aleksa Sarai <[email protected]>
Use the *not-yet-merged* parallelism support I've been working on in bats[1] in order to speed up our tests massively. This requires that we use my home project on OBS, but hopefully my patches will be merged eventually. [1]: bats-core/bats-core#172 Signed-off-by: Aleksa Sarai <[email protected]>
0b61dbc to
3d3378c
Compare
On distributions like openSUSE, libexec doesn't really exist so we need to install bats in other places in the source tree. We previously had out-of-tree patches for this, but it looks like they weren't pushed upstream. Signed-off-by: Aleksa Sarai <[email protected]>
It appears as though this test didn't actually test what it was supposed to in 25505bd ("Skip pretty formatting if the first line isn't a TAP plan"). It was supposed to test that bats-format-tap-stream would not try to format non-TAP data -- but instead it just showed that the preamble of test files was executed before printing the TAP header (something that is not going to be the case with the suite refactor). Fixes: 25505bd ("Skip pretty formatting if the first line isn't a TAP plan") Signed-off-by: Aleksa Sarai <[email protected]>
This allows us to remove most of the complicated handling between bats-exec-test and bats-exec-suite (which had a complicated interplay due to the fact they both supported multi-test execution). Instead, multi-test execution is entirely left to bats-exec-suite and bats-exec-test only handles a single test. This entirely removes the need for the magic output handling in bats-exec-test -- and opens the door to suite-wide parallelism in one fell swoop. The only downside is that bats-preprocess is run multiple times on the same source file, but a follow-up patch will improve the caching of pre-processed files such that they will only be processed once. Signed-off-by: Aleksa Sarai <[email protected]>
3d3378c to
f5f2794
Compare
Most well-behaved bats tests (where they aren't writing to shared data between test executions) are able to be executed in parallel, but to date bats has always executed them in serial. This is incredibly wasteful for trivially parallelisable tests, and can result in fairly inflated test times for no good reason. Parallel exeuction is done suite-wide, so it makes no difference how the tests are structured. Unfortunately, because some suites might be implicitly serial (like bats' own test suite) we cannot make the default mode execute in parallel. In addition, there is currently no way of tagging that a *specific* test has to be run in serial (so that the rest can be run in parallel) -- but that can be done in future work. Signed-off-by: Aleksa Sarai <[email protected]>
Testing that parallel execution actually works is a bit difficult, so the most trivial way of doing it is to just have a bunch of sleep tests and make sure that they run faster than would be possible if they were run in serial. Signed-off-by: Aleksa Sarai <[email protected]>
f5f2794 to
7e6dd94
Compare
|
Any updates? @sublimino? |
|
Neat. I will send the randomised ordering PR next. |
|
This commit: 97975a0 Introduces Stacktrace for those living on the bleeding edge running from source: /usr/bin/readlink: illegal option -- f
usage: readlink [-n] [file ...]
|
|
Yes. I was considering re-implementing |
|
Scratch that, I just figured out I can implement it with a really hacky use of |
This allows for tests to be run in parallel, by using GNU parallel (if
available). It's a fairly trivial extension (though it required a fair
amount of work in order for it to be usable).
Most well-behaved bats tests (where they aren't writing to shared data
between test executions) are able to be executed in parallel, but to
date bats has always executed them in serial. This is incredibly
wasteful for trivially parallelisable tests, and can result in fairly
inflated test times for no good reason. Parallel exeuction is done
suite-wide, so it makes no difference how the tests are structured.
Unfortunately, because some suites might be implicitly serial (like
bats' own test suite) we cannot make the default mode execute in
parallel. In addition, there is currently no way of tagging that a
specific test has to be run in serial (so that the rest can be run in
parallel) -- but that can be done in future work.
Fixes #171
Signed-off-by: Aleksa Sarai [email protected]