Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@cyphar
Copy link
Contributor

@cyphar cyphar commented Feb 2, 2019

This allows for tests to be run in parallel, by using GNU parallel (if
available). It's a fairly trivial extension (though it required a fair
amount of work in order for it to be usable).

Most well-behaved bats tests (where they aren't writing to shared data
between test executions) are able to be executed in parallel, but to
date bats has always executed them in serial. This is incredibly
wasteful for trivially parallelisable tests, and can result in fairly
inflated test times for no good reason. Parallel exeuction is done
suite-wide, so it makes no difference how the tests are structured.

Unfortunately, because some suites might be implicitly serial (like
bats' own test suite) we cannot make the default mode execute in
parallel. In addition, there is currently no way of tagging that a
specific test has to be run in serial (so that the rest can be run in
parallel) -- but that can be done in future work.

Fixes #171
Signed-off-by: Aleksa Sarai [email protected]

@cyphar cyphar requested a review from a team as a code owner February 2, 2019 18:35
@cyphar cyphar force-pushed the bats-parallel branch 6 times, most recently from 74c9b1e to a2cf056 Compare February 5, 2019 06:57
@cyphar cyphar changed the title [wip] bats: support parallel execution of tests bats: support parallel execution of tests Feb 5, 2019
cyphar added a commit to cyphar/umoci that referenced this pull request Feb 6, 2019
Use the *not-yet-merged* parallelism support I've been working on in
bats[1] in order to speed up our tests massively. This requires that we
use my home project on OBS, but hopefully my patches will be merged
eventually.

[1]: bats-core/bats-core#172

Signed-off-by: Aleksa Sarai <[email protected]>
cyphar added a commit to cyphar/umoci that referenced this pull request Feb 6, 2019
Use the *not-yet-merged* parallelism support I've been working on in
bats[1] in order to speed up our tests massively. This requires that we
use my home project on OBS, but hopefully my patches will be merged
eventually.

[1]: bats-core/bats-core#172

Signed-off-by: Aleksa Sarai <[email protected]>
@cyphar cyphar force-pushed the bats-parallel branch 2 times, most recently from 0b61dbc to 3d3378c Compare February 10, 2019 13:44
On distributions like openSUSE, libexec doesn't really exist so we need
to install bats in other places in the source tree. We previously had
out-of-tree patches for this, but it looks like they weren't pushed
upstream.

Signed-off-by: Aleksa Sarai <[email protected]>
It appears as though this test didn't actually test what it was supposed
to in 25505bd ("Skip pretty formatting if the first line isn't a
TAP plan"). It was supposed to test that bats-format-tap-stream would
not try to format non-TAP data -- but instead it just showed that the
preamble of test files was executed before printing the TAP header
(something that is not going to be the case with the suite refactor).

Fixes: 25505bd ("Skip pretty formatting if the first line isn't a TAP plan")
Signed-off-by: Aleksa Sarai <[email protected]>
This allows us to remove most of the complicated handling between
bats-exec-test and bats-exec-suite (which had a complicated interplay
due to the fact they both supported multi-test execution).

Instead, multi-test execution is entirely left to bats-exec-suite and
bats-exec-test only handles a single test. This entirely removes the
need for the magic output handling in bats-exec-test -- and opens the
door to suite-wide parallelism in one fell swoop.

The only downside is that bats-preprocess is run multiple times on the
same source file, but a follow-up patch will improve the caching of
pre-processed files such that they will only be processed once.

Signed-off-by: Aleksa Sarai <[email protected]>
Most well-behaved bats tests (where they aren't writing to shared data
between test executions) are able to be executed in parallel, but to
date bats has always executed them in serial. This is incredibly
wasteful for trivially parallelisable tests, and can result in fairly
inflated test times for no good reason. Parallel exeuction is done
suite-wide, so it makes no difference how the tests are structured.

Unfortunately, because some suites might be implicitly serial (like
bats' own test suite) we cannot make the default mode execute in
parallel. In addition, there is currently no way of tagging that a
*specific* test has to be run in serial (so that the rest can be run in
parallel) -- but that can be done in future work.

Signed-off-by: Aleksa Sarai <[email protected]>
Testing that parallel execution actually works is a bit difficult, so
the most trivial way of doing it is to just have a bunch of sleep tests
and make sure that they run faster than would be possible if they were
run in serial.

Signed-off-by: Aleksa Sarai <[email protected]>
@cyphar
Copy link
Contributor Author

cyphar commented Mar 5, 2019

Any updates? @sublimino?

@sublimino
Copy link
Member

This is good to go as far am I am concerned, loads of good discussion on #171 and those who have run existing test suites report that they are unaffected.

Thanks for the contribution @cyphar 🙏

@sublimino sublimino merged commit 8789f91 into bats-core:master Mar 5, 2019
@cyphar cyphar deleted the bats-parallel branch March 5, 2019 10:18
@cyphar
Copy link
Contributor Author

cyphar commented Mar 5, 2019

Neat. I will send the randomised ordering PR next.

@patthiel
Copy link

This commit: 97975a0

Introduces greadlink as a dependency, which isn't included on osx. It results in errors running bats.

Stacktrace for those living on the bleeding edge running from source:

/usr/bin/readlink: illegal option -- f
usage: readlink [-n] [file ...]

brew install coreutils resolved issues running from source for me.

@cyphar
Copy link
Contributor Author

cyphar commented Mar 12, 2019

Yes. I was considering re-implementing readlink -f in bash to not have the dependency but it's really quite annoying to to do correctly. In addition, #196 will add a hard dependency to gshuf which we can't avoid because it's simply too hard to implement in bash (it would require quite unreadable awk scripts to do nicely).

@cyphar
Copy link
Contributor Author

cyphar commented Mar 12, 2019

Scratch that, I just figured out I can implement it with a really hacky use of paste.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants