It would be a good idea to have a common test suite for different Python type checkers like Mypy, PyCharm, PyType. By introducing it, we could ensure that our checkers have a common understanding of the type system.

The existing checkers detect various kinds of errors depending on the depth of code analysis and type inference. The idea is to use the common test suite just for testing basic scenarios every checker should understand according to PEP 484.

As a starting point, we could use a subset of Mypy tests and Mypy test format.

For example, a typical Mypy test looks like that:

[case testTupleAssignmentWithTupleTypes]
from typing import Tuple
t1 = None # type: Tuple[A]
t2 = None # type: Tuple[B]
t3 = None # type: Tuple[A, A]
t4 = None # type: Tuple[A, B]
t5 = None # type: Tuple[B, A]

t1 = t2 # E: Incompatible types in assignment (expression has type "Tuple[B]", variable has type "Tuple[A]")
t1 = t3 # E: Incompatible types in assignment (expression has type "Tuple[A, A]", variable has type "Tuple[A]")
t3 = t1 # E: Incompatible types in assignment (expression has type "Tuple[A]", variable has type "Tuple[A, A]")
t3 = t4 # E: Incompatible types in assignment (expression has type "Tuple[A, B]", variable has type "Tuple[A, A]")
t3 = t5 # E: Incompatible types in assignment (expression has type "Tuple[B, A]", variable has type "Tuple[A, A]")

# Ok
t1 = t1
t2 = t2
t3 = t3
t4 = t4
t5 = t5

class A: pass
class B: pass
[builtins fixtures/tuple.pyi]

There are several directives in square brackets like [case ...] for defining a test case, [builtins ...] for including minimalistic stubs for built-ins, and some others. The expected error messages are written using the # E: ... comments.

I would like to introduce these tests in the typing repo and maybe setup Travis CI for running them.