Note that upstream freetype has moved to CMake on Windows (ftjam is gone at least), so the build scripts will need updated here.

The vulnerability in 2.10.4 is in PNG support, which is not enabled in our FreeType build. All other published CVEs have been reviewed and are not applicable to FreeType as used in Matplotlib.

There are pros and cons to moving forward, but known vulnerabilities are not (currently) one of them.

Is the "local" build really the default, or just a testing default?

It is the default; manylinux does not guarantee freetype is available on Linux and it isn't on macOS or Windows either.

The main reason we are pinned to 2.6.1 is that we do 0 tolerance image comparison tests, if we change the version of freetype, the text in the images changes a bit and breaks our tests (in the past we had a default non-zero tolerance and that let through several horrific bugs where we were rendering the wrong glyph but it still was "within tolerance" 🤦🏻 ). We need to revive the GSOC work from last summer to improve the management of our baseline images before we change the freetype version we are pinned to.

Down-stream packagers can (and do at least in the case of Fedora and Conda) distribute newer versions of freetype and link us against that version. See https://matplotlib.org/stable/devel/dependencies.html#c-libraries for instructions on how to link against your system freetype.

I would also reference https://github.com/QuLogic/mpl-images for people wanting to run tests against a more recent FreeType.

In #21271 the repo size was mentioned as an issue, have you considered using git-lfs to counter this?

have you considered using git-lfs to counter this

iirc when we looked at the space / bandwidth you could get it was too small to be useful. The problem is not that we have a few big files the problem is we have have a large number of small files. If we were going to go a git-level solution I think sub-modules would be a better path.

However, a git-level solution does not get around the problem of needing to switch which version of the images you are using to match the freetype you are building with. Even if we were to track and checkin re-built images with the new freetype we would likely still be broken for a majority of the down-stream packagers as we standardized on a different version than they ship (which is arguably better than now where we are broken for everyone but the current case could be said to be more fair 😉 ).

It seems the easiest is to just have an image repo and if folks are concerned about its size, they can just do a shallow clone. The only problem with that is getting CI to work smoothly between the two repos in that you'd want the main PR to use the new or modified image in the accompanying image repo pr

Or maybe even try a submodule setup. https://github.blog/2016-02-01-working-with-submodules/

It seems the easiest is to just have an image repo and if folks are concerned about its size, they can just do a shallow clone. The only problem with that is getting CI to work smoothly between the two repos in that you'd want the main PR to use the new or modified image in the accompanying image repo pr

A URL to a remote repo can be specified in https://github.com/matplotlib/pytest-mpl for the baseline image directory. SunPy uses this feature and updates the image repo when main is pushed to. Would it make sense for matplotlib to migrate to using the matplotlib/pytest-mpl plugin?

A URL to a remote repo can be specified in https://github.com/matplotlib/pytest-mpl for the baseline image directory. SunPy uses this feature and updates the image repo when main is pushed to. Would it make sense for matplotlib to migrate to using the matplotlib/pytest-mpl plugin?

That sounds good. I assume a new PR can compare against its own new/modified image?

Yes, however for a remote baseline image repository it's best to use the Hybrid Mode so generated images are compared against sha hashes of the baseline image files, which would be stored in the matplotlib/matplotlib repo. This way you don't need to update the remote baseline image repository before the PR is merged in order for the tests to pass.

@ConorMacBride Following up on this: what happens if someone writing a PR changes an existing image or adds a new image? Do they open two PRs, one in each repo? Seems that CI wouldn't know where to find the new PR in the image repo?

@jklymak The image repo is only updated when the PR is merged into the main (matplotlib/matplotlib) branch, so only one PR in total is needed. This update can be configured to happen automatically as SunPy did here. This CI runs in the main repository and pushes the updates to the remote image repository.

When using the Hybrid Mode, only the updated/new image hashes need to be added to the main PR for the tests to pass. This is because actual images are only compared against the baseline images if the generated hashes do not match. (The hashes are sensitive to the particular OS and freetype version, but a hash library for each version can be included in the main repo, and the correct library is selected for the test environment.) I've been working on some tools (matplotlib/pytest-mpl#128) to hopefully make it easier for contributors to update the hashes and see how their changes affect the baseline images.

to hopefully make it easier for contributors to update the hashes and see how their changes affect the baseline images.

Right, this seems key, because if I change a bunch of images, it is really helpful for there to be a way to browse the changes in the PR. However, I guess if I generate an updated image, I would still make a PR on the image repo and folks could browse the diff there, even if the main-repo tests are run on the hash?

The manual PR to the image repo shouldn't be necessary if you can have the CI provider make the test artifacts (i.e. generated and diff images) available to download. SunPy uses CircleCI for the image comparison tests as it can serve HTML artifacts, so the difference images are all presented in a HTML report, this for example. Although I have a PR to present a report like this. This way the baseline images always correspond to the state of the main branch of the main repo.