Our type-tests aren't perfect, we don't type-check our stubs, and don't use stubtest. But even if we would do all that, there's always a chance that seemingly innocent changes to the stubs could unintentionally lead to unforeseen typing issues in downstream packages.

The mypy_primer tool was built to help with issues like these, and helps reduce the number of these static-typing-related unknown unknowns.

From https://github.com/hauntsaninja/mypy_primer README.MD:

mypy_primer is a tool for regression testing Python static type checkers.

mypy_primer makes it easy to run different versions of a type checker over several million lines of open source Python projects with typing for the purpose of evaluating changes.

Here's what mypy_primer does:

• Clones a copy of mypy (potentially from a fork you specified)
• Checks out a "new" and "old" revision of mypy
• Clones a hardcoded list of projects (potentially filtered by you)
• Installs necessary stubs and dependencies per project
• Runs the appropriate mypy command per project
• Shows you the potentially differing results!
• Lets you bisect to find the mypy commit that causes a given change

I have little knowledge about NumPy's CI, but I think it could help a lot if we could incorporate mypy_primer in it, and preferably run it only if changes are made to at least one .pyi stub file.