After #779, I conducted further testing to find at which freetype versions there were changes in rendering of fonts. I have updated my testing suite in:

https://github.com/astrofrog/freetype_version_testing

to include a test_exhaustive.py script that tests when rendering of a specific character changed as a function of freetype version. The results can be seen graphically in the breaking_points_* plots:

https://github.com/astrofrog/freetype_version_testing/blob/master/breaking_points_grey.png
https://github.com/astrofrog/freetype_version_testing/blob/master/breaking_points_mono.png

The bottom line is that monochromatic rendering changed at 2.3.0, 2.3.10 (only for 'C'), 2.4.0, 2.4.4, and 2.4.5. After that, things are stable. On the other hand, anti-aliased rendering was stable up to 2.4.2, then changed at 2.4.3, 2.4.4, 2.4.5, and 2.4.7.

The change in 2.4.5 that caused the change in monochrome rendering was due to this change in freetype:

2011-01-15  Werner Lemberg  <[email protected]>

    [raster] Make bbox handling the same as with Microsoft's rasterizer.

    Right before B/W rasterizing, the bbox gets simply rounded to
    integers.  This fixes, for example, glyph `three' in font `Helvetica
    Neue LT Com 65 Medium' at 11ppem.

    Thanks to Greg Hitchcock who explained this behaviour.

    * src/raster/ftrend1.c (ft_raster1_render): Implement it.

The specific code that changed in src/raster/ftrend1.c in freetype-2.4.5 was:

cbox.xMin = FT_PIX_FLOOR( cbox.xMin );
cbox.yMin = FT_PIX_FLOOR( cbox.yMin );
cbox.xMax = FT_PIX_CEIL( cbox.xMax );
cbox.yMax = FT_PIX_CEIL( cbox.yMax );

to:

/* undocumented but confirmed: bbox values get rounded */
#if 1
cbox.xMin = FT_PIX_ROUND( cbox.xMin );
cbox.yMin = FT_PIX_ROUND( cbox.yMin );
cbox.xMax = FT_PIX_ROUND( cbox.xMax );
cbox.yMax = FT_PIX_ROUND( cbox.yMax );
#else
cbox.xMin = FT_PIX_FLOOR( cbox.xMin );
cbox.yMin = FT_PIX_FLOOR( cbox.yMin );
cbox.xMax = FT_PIX_CEIL( cbox.xMax );
cbox.yMax = FT_PIX_CEIL( cbox.yMax );
#endif

This doesn't look like it would be easy to workaround. However, we should investigate whether any of the other changes can be worked around.

Anyway, just wanted to summarize the findings, and moving forward we can hopefully make slow progress on finding a solution to make all tests pass with all freetype versions.

@mdboom - before we explore this further (and I'm not planning on doing so for the next couple of weeks), I just wanted to check whether there is any obvious reason why it wouldn't be possible to just bundle freetype (or just the parts that are needed) with matplotlib, and just call the C code directly from ft2font.cpp? Freetype can be used under a BSD-like license that would allow matplotlib to use parts of it as long as you include the license file (see http://www.freetype.org/FTL.TXT). This would be similar to how you included wcslib in pywcs/astropy.wcs, and would make all these testing issues moot. But I imagine the matplotlib developers have discussed this before and ruled it out as an option?

This is just an idle musing of another way of handling our testing to make it easier to communicate test failures. Currently, we're uploading to dropbox, google docs, whatever, so we can share our test failures. I am thinking of a way to leverage github, and their nice image comparison tools. If we had a mode where the test images could overwrite the existing baseline images, then the image failure would show up as a git diff. Then one way of testing would be: from your own fork of mpl, you would create a branch like tests_20120323, and run the tests in an inplace mode where the output images overwrite the baseline if they register as a failure. Then you push this branch to your fork, and anyone can see the test failures as a diff and compare the images using the github image comparison tools, eg

https://github.com/matplotlib/matplotlib/pull/680/files#diff-3

The branches would be throwaways and could be deleted a few days later.

@jdh2358 - this is a great idea!

@astrofrog, it looks like this has been dormant for 2 years so I am closing it. Feel free to reopen it if you think it needs to stay in the "open" category.

Yeah -- we've had a fairly useful "half measure" to this in place for a while (which is to explicitly mark tests that we know fail on particular ranges of freetype versions). It does seem that freetype's output has stabilized in recent years as well -- I have not had to update the images in a while due to upgrading my OS or Travis upgrading their OS for some time, for example.