We wouldn't need to support [builtins ...], which specifies a fairly minimal test-only stub to use for a particular mypy test case. These tests could probably use the real stubs for builtins and other modules, since these would be more like acceptance tests rather than unit tests.

@JukkaL Do you suggest to use just regular Typeshed in the tests?

I'd suggest using whatever stubs each tool is shipped with. For mypy this would be typeshed, but other tools might use custom stubs for builtins, for example.

Also I would assume the error messages wouldn't have to be identical for the different tools.

Maybe we'll need a separate library for parsing such test files? Otherwise there would be a lot of hacky duplicate code -- mypy has its version of this parser (and it's evolving, e.g. python/mypy#3347).

Or even better, we could have no parsing of tests at all by using YAML for them. YAML is easy to type and extendable so we could introduce additional attributes easily without breaking existing test runners.

An example using YAML:

- name: testTupleAssignmentWithTupleTypes
  files:
    - content: |
        from typing import Tuple
        t1 = None # type: Tuple[A]
        t2 = None # type: Tuple[B]
        t3 = None # type: Tuple[A, A]
        t4 = None # type: Tuple[A, B]
        t5 = None # type: Tuple[B, A]

        t1 = t2 # E: Incompatible types in assignment (expression has type "Tuple[B]", variable has type "Tuple[A]")
        t1 = t3 # E: Incompatible types in assignment (expression has type "Tuple[A, A]", variable has type "Tuple[A]")
        t3 = t1 # E: Incompatible types in assignment (expression has type "Tuple[A]", variable has type "Tuple[A, A]")
        t3 = t4 # E: Incompatible types in assignment (expression has type "Tuple[A, B]", variable has type "Tuple[A, A]")
        t3 = t5 # E: Incompatible types in assignment (expression has type "Tuple[B, A]", variable has type "Tuple[A, A]")

        # Ok
        t1 = t1
        t2 = t2
        t3 = t3
        t4 = t4
        t5 = t5

        class A: pass
        class B: pass

Having a common parsing library for Mypy-specific format in Python wouldn't help PyCharm since our codebase runs on JVM (and Jython is too slow, unfortunately).

Parsing .test files is actually pretty simple -- most of the related code is about creating files related to test cases, cleaning them up and handling various special cases that wouldn't be needed for these tests.

One nice thing about the existing mypy test file format is that the source code isn't indented so it's easy to copy&paste between a source file and a test file. Not sure if that would be practical with YAML.

Still, it would require a parser and a some sort of a standard for the file format that cares about its extensibility. Having a block de-dented after copy&paste should be fairly easy in most text editors.

+1 for YAML from me, -0.5 for the current Mypy format.

@gvanrossum What do you think about the format for the tests? Should it be Mypy test format, YAML, or something else?

Hm. I worry that yaml indentation and python indentation will make things hard to follow or edit. But we may need to experiment to (dis)prove our concerns.

I'm going to implement a prototype with a simple YAML-based format and test runners for Mypy and PyCharm here at the PyCon sprints. Let's see if the resulting tests will be easy to read and edit.

A common test suite is now realized in the form of the typing spec conformance tests.