I'm opening this as an issue separate from #6586 to better document the investigation into the root cause. This affects both the v2.0.x and master branches.

We can make the failures a bit more reproducible by repeating the tests a bit. What I did was the following:

1. Choose one of the mathtext/mathfont tests (e.g. mathfont_*_08)
2. Copy that test image 100 times for each font set

for i in `seq -w 0 99`; do
  for fs in cm dejavusans dejavuserif stix stixsans; do
      cp test_mathtext/mathfont_${fs}_08.png test_fail/mathfont_${fs}_${i}.png
  done
done

3. Modify test_mathtext.py to drop the other tests and run only 08 100 times over.
4. Edit Text._get_layout to print out whether the cache hit worked or failed.

In the end, you will get about 2 or 3 failures out of the 500. On the failures, the cache returns a value. It turns out that every once in a while, the renderer will have the same ID (or be the same one, I don't know). Since the cache key does not include the mathtext fontset, the cached value will be for some other one and thus be in the wrong place.

I'm not sure how to easily fix this. Ideally, the fontset would be part of the key, but it's not actually used by Text objects, but the MathTextParser. According to #7107, baking the rcParam directly into initialization is a hard problem. Maybe we should find a way to disable the cache during test?