Another couple quick questions:

• Perhaps add a command to emit the Bash version before running the tests?
• So if one of these fails, Travis'll recognize and still fail the build, right?

I see that all the builds currently pass on Travis, so that's comforting! :-)

I was operating under the assumption that Travis is essentially running under a set -e sort of mentality. I'll test on a localized branch with an intentionally failing test and see what it does.

Regarding emission of Bash version, I could have it add that to the Docker image. Would something like this be acceptable?

Sending build context to Docker daemon  852.5kB
Step 1/6 : ARG bashver=latest
Step 2/6 : FROM bash:${bashver}
latest: Pulling from library/bash
605ce1bd3f31: Pull complete
18ecc0d9c7d6: Pull complete
ff65c95dde64: Pull complete
Digest: sha256:f72240387ad0a579251a00c1d12dd12845e0d3336a3b518b5c267cdddf18f2f2
Status: Downloaded newer image for bash:latest
 ---> e6f9405e3ba1
Step 3/6 : RUN bash --version
 ---> Running in affacecf2837
GNU bash, version 4.4.23(1)-release (x86_64-pc-linux-musl)
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Removing intermediate container affacecf2837
 ---> 51d9198615bb
Step 4/6 : RUN ln -s /opt/bats/bin/bats /usr/sbin/bats
 ---> Running in 3ed497669e20
Removing intermediate container 3ed497669e20
 ---> aa01de42869a
Step 5/6 : COPY . /opt/bats/
 ---> ec5ae3a6265f
Step 6/6 : ENTRYPOINT ["bash", "/usr/sbin/bats"]
 ---> Running in a78113edb0c8
Removing intermediate container a78113edb0c8
 ---> 451e944e85c6
Successfully built 451e944e85c6

My previous commits were relying on the building of the image showing the version via the loop variable, but I realize that's not as accurate as doing something like this.

Sure enough, it does not behave like set -e in the loop. Adjusting that now.

Travis has the concept of a build matrix, which supports generating various cross configurations of OS/runtimes/environments.

We can generate the build matrix to test multiple versions of bash without using a for-loop. This will give us first-class reporting on the travis dashboard (a distinct build job for each bash version) as well as implicit "-e like" exiting.

PR (to this PR) here: cdevinesr#1

Related... correct me if I'm wrong, but for linux builds, the first step in the test script is redundant, no?

script:
- bash -c 'time bin/bats --tap test'
- |
   if [[ "$TRAVIS_OS_NAME" == 'linux' ]]; then
     "docker build"
   fi

We could change the test script to be:

script:
- |
   if [[ "$TRAVIS_OS_NAME" == 'linux' ]]; then
     "docker build"
   else
     bash -c 'time bin/bats --tap test'
   fi

Unless I'm missing something?

The only thing that initial test gets us over the added version coverage is timing, but that timing step could be added to the versioned checks instead, so we can get a better idea about performance on each Bash version. Unless there's a specific reason we'd need the tests executed directly under Linux but not inside Docker (I can't think of anything unique or special that would give us over just putting it into the version checks).

I'll merge your changes (still new to Travis and your implementation is much cleaner than what I was doing).

still new to Travis

no worries, the yml file can be ... obtuse.

For instance, the matrix.include is rather odd. If we left the os having both linux and osx, we'd end up with a 2x6 build matrix (12 jobs). Of course, all we want is the 1x6 linux builds (one for each bash version) and then an extra macOS build. The explicit matrix.include lets us add specific jobs after the build matrix has been calculated. (You can also do matrix.exclude in order to exclude build configurations that match a certain criteria, as well.)

Yeah, that's what I got from reading over your changes and the Travis documentation.

The more I think about it, the more I think perhaps the timing piece may not give us much extra. Travis already shows run time when running jobs, and there's not much I can think of offhand in terms of real benefits that we'd get from doing it manually with time instead. Thoughts?

Hmm, can we set ENTRYPOINT ["bash", "-c", "time", "bats"] in the Dockerfile? It's not essential, but it's really handy to have the timing information from the build.

Alternatively, perhaps we should add time to Bats itself. Thoughts?

It doesn't work, since time bats [args] would need to all be one argument, which removes the ability to pass arguments on demand from the command line (which is the use case we're liable to see from end users).

It can be done with CMD, but again, we'd be forcing the image to only be a self-test image rather than an actual utility people can work with.

Alternatively, we could include two Dockerfiles - one that builds the image people would work with, and one that builds a 'testing' image that just performs the self test with CMD. We could simply reference the self-test Dockerfile during the build with -f.

Ah, asynchronous comments...

So the Travis build time factors in the time it takes for the environment to get set up, which can fluctuate. Running time shows the timing for just the test suite, after the environment's stable (including fetching Docker images).

Hmm, thought what I read in man bash suggested ["bash", "-c", "time", "bin/bats"] would work, but local experimentation demonstrates otherwise. Oh well.

That said, time docker run -it bats/bats:bash-${BASHVER} --tap /opt/bats/test might be fast enough for our purposes. Want to try it?

Looks pretty close to the timing we'd get wrapping the call to Bats itself:

ok 62 a failing test in a suite results in an error exit code
ok 63 running an ad-hoc suite by specifying multiple test files
ok 64 extended syntax in suite
ok 65 recursive support (short option)
ok 66 recursive support (long option)
real	0m2.462s
user	0m0.016s
sys	0m0.004s
The command "if [[ "$TRAVIS_OS_NAME" == 'linux' ]]; then
  docker build --build-arg bashver=${BASHVER} --tag bats/bats:bash-${BASHVER} .
  time docker run -it bats/bats:bash-${BASHVER} --tap /opt/bats/test
fi
" exited with 0.
Done. Your build exited with 0.

I'm good with it, personally.

I'm going approve, but will let @sublimino register his approval as well.

For the record, though, I did crack the ENTRYPOINT nut with the help of https://unix.stackexchange.com/a/144519:

ENTRYPOINT ["bash", "-c", "time bats \"$@\"", "bats"]

And then after building:

$ time docker run -it bats/bats:latest --tap /opt/bats/test
1..66                                                            
ok 1 no arguments prints usage instructions
...
ok 66 recursive support (long option)      
                       
real    0m2.114s                     
user    0m0.630s                   
sys     0m0.060s                           
                                                           
real    0m3.313s                                            
user    0m0.070s                               
sys     0m0.036s

The first time output is from the ENTRYPOINT, and the second is from the time docker run invocation. So the time docker run time does include a bit of overhead, but I don't really like baking the command into the ENTRYPOINT, either.

Hence, time docker run is good enough for now.

BTW, I know this PR kinda ballooned into a more complex discussion pretty quickly, but thanks for your patience and persistence! It's going to be great having this level of coverage across Bash versions with every PR—props to @jasonkarns for the Travis magic, much better than my cruddy trickery.

Also, I certainly feel greatly edified by this whole exchange! Learned a few useful new things. I'd trigger the merge myself right now, but will let @sublimino do the honors if he's satisfied.

Awesome, I turn around for one evening and this happens! 😍

LGTM

Now all that's left is to push passing and built image versions to the docker hub. Does anybody have the keys to add secrets to Travis?

Btw @mbland @jasonkarns my instinct is to squash and merge this, which last time lost gpg signing. Do we have a preference on commit history maintenance? Asking users to rebase can be nightmarish for some etc etc

Meh. For most team repos, I'm strongly against squash merging (since you lose history); but that's primarily so we don't have orphaned branches that can't be tracked down. And that assumes some level of rebasing on the author's part.

Generally, I'm in favor of regular old merge commits.

@sublimino I'm OK not squashing, and I'm less prone to rebase than I used to be. I would say let's just hit the big "Merge pull request" button, but now that I'm really thinking about it, it would be nice to have a signed commit somewhere in the process.

First, could you hit the "Approve changes" link in the PR thread here first, just to clean that up? Then, do you want to push a local signed merge to master? Or I could do it. (And I wonder if we should do that as a course of habit.)

BTW, I'm thinking we could stand to put docker run -it bash:${bashver} --version back in to print the exact version of Bash in the container, but that's easy enough to add to a future PR. I'd rather just get this in ASAP than hold out for another tweak.

@cdevinesr At some point, could you check out https://help.github.com/articles/signing-commits-with-gpg/ so any future PRs will have GPG-signed/verified commits?

I've already planned to do so, and tried once before, but since I do most of my stuff from work, and I primarily use a variation of Cygwin (drive space limitations on my laptop that I have no way or authority to work around, unfortunately), gnupg does not seem to function properly on there.

I'm working on getting enough space available to put together small dev VM with Vagrant that I can get a GPG key set up on. I'll see about accomplishing that today.

@cdevinesr Well, don't let not having GPG set up be a blocker. If we end up doing signed merge commits as a matter of policy, it solves the immediate issue. Having incoming PRs with signed commits is more of a nice-to-have.

Oh, and since you're up and about, feel free to sneak that docker run -it bash:${bashver} --version back in there... 🙂 Only if you've a reasonable chance; I don't want to block this PR any further.

Also thinking we might want to specify BASHVER for the corresponding BASH_VERSIONs from stock Travis images, such as 4.3.11. Again, sneak that in if you can, but it shouldn't be a blocker.

I think as long as we sign the tagged release, that's sufficient; especially since that technically signs the tree up to that point. (And github signs merge commits, which is close.)

Oh crap, I hadn't noticed GitHub signs the merges. Makes sense they'd do that.

@sublimino Can you approve the review and just click the "Merge pull request" button, then?

The official Bash Docker images don't go to that level of granularity, unfortunately. They're broken down strictly by major and minor, and don't let you drill down by point release. The list of available Docker image tags can be found here.

I can add the version check back to the Docker image. I was avoiding it because it obviously adds another layer for something that merely outputs to the command line (so basically we'd have an "empty" layer), but if it's not a problem for you, I don't mind re-adding it. I think having that level of detail could prove useful in the long run, even with the dashboard covering major and minor for us.

No, not as a layer in the Dockerfile, but as a command in the .travis.yml script!

docker build --build-arg bashver=${BASHVER} --tag bats/bats:bash-${BASHVER} .
docker run -it bash:${bashver} --version
time docker run -it bats/bats:bash-${BASHVER} --tap /opt/bats/test

Hmm, https://hub.docker.com/r/library/bash/tags/ does have tags that drill down to the patch level, but it seems only for the latest versions. Bummer.

Yeah, you're seeing what I was referring to. It makes sense to have a tag with point releases, but they don't keep one for each (that'd quickly turn into a lot of images to keep around).

And I see I did that Dockerfile update before your comment. I've reverted that and pushed your suggestion. Still waking up this morning.

Thanks again @cdevinesr and everybody - I'm ready to Push The Button in 2m

I'd be inclined to sign merges in the future as it reduces the amount of code to review and sign at "tagging time", but either effectively achieves the same end.