This is only failing on OSX. Before changing any reference images, we need to make sure that‘s the correct thing to do. First thing to do is check the generated images to see what‘s different. Somehow we should be able to get those images from Travis, but I don‘t know how.

It is also worth looking through the build history (particularly on the master branch) to see if there is a particular commit where it broke. In general the tests are presumed to be correct and the code is presumed to be wrong. Changing the tests generally indicates an API changed an requires a justification.

From gitter:
@tacaswell:

https://travis-ci.com/github/matplotlib/matplotlib/builds/158347125 <- last good build on master
https://travis-ci.com/github/matplotlib/matplotlib/builds/158411743 <- first fail on master
#17025 is that merge and it passed CI

@QuLogic:

I think that was skipped in the 'working' build

SKIPPED [1] lib/matplotlib/tests/test_bbox_tight.py:83: Cannot compare pdf files on this system

@tacaswell:

interesting...

@QuLogic

and a diff of the log says that ghostscript is now installed as a dependency of imagemagick
oh, and 1288 skipped vs 678 skipped

@jklymak:

Same problem I was getting before
I upgraded ghostscript and the above is what the new failed diff looks like

@QuLogic:

ah, so it's new ghostscript that does it? what versions?

@jklymak:

Sorry yes new gs. Stepped away from computer for a while so not sure what the versions are

gs
GPL Ghostscript 9.52 (2020-03-19)

is the new one.

$ gs GPL Ghostscript 9.26 (2018-11-20) was my old one

I can reproduce in a Fedora 32 container with ghostscript 9.52 , but it only changes the result from 5.534 to 6.457. There must be something else different as well. Scratch that, it was just the test image cache.