Move pythoneval to use mypy.api.run #4025
Conversation
This replaces the old subprocess based method. In addition, the test cases are no longer run, in order to reduce test time. This has led to a 10-20% speedup in pytest. This might help with python#3895, as it removes the subprocesses. Fixes python#1671
|
This PR looks like a good idea regardless of performance and flakes, since it makes the test more cohesive. StringIO sounds very good! In addition to potential speedup of test execution, it will also speed up test discovery but an order of magnitude (I intended to do that by deferring the act of writing the files). I'm interested to know what are the potential downsides; I can't see any. Regarding |
|
Do the tests re-analyze typeshed each time? |
That is why I decided to not include that in this PR. I do like the idea of splitting them up. StringIO would require a bit of work on the mypy internals, but I think it is worth it. As for typeshed, yes, I believe they do re-analyze each run, but an optimization there would likely be complicated. |
|
But incremental mode might help? (I think caching mypy's state after analysis might be very useful, but that's slightly OT) |
|
Incremental may help, if you want to continue this discussion the best place is probably #1668. |
|
This change has the additional benefit of making us completely independent of a runtime Python 2 interpreter, fixing #4039. |
While I appreciate the speedup, I worry that over time these test cases might stop working when executed. There are still quite a few places where mypy passes code that doesn't actually work at runtime. Every time a test case in this category is added or changed we would risk that we'd be testing something imaginary. This is particularly worrisome for tests that are trying to ascertain that mypy matches subtle runtime behavior. |
|
Okay, that makes sense. I believe several of the tests can be removed anyway (they test extremely basic things), so that will cause some speed increases. |
|
So do you want to reuse this PR for the new approach, or open a new one?
|
|
I will open a new one, I think it will be cleaner. |
This replaces the old subprocess based method. In addition, the test cases
are no longer run, in order to reduce test time. This has led to a 10-20%
speedup in pytest. This might help with #3895, as it removes the
subprocesses.
Fixes #1671
Two follow up ideas:
testcmdlineto usemypy.api.runas well