Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 255fb61

Browse files
gnpricegnprice
authored
Set up to use pytest for our data-driven tests, and switch testcheck over to it (#1944)
This is a step toward #1673 (switching entirely to pytest from myunit and runtests.py), using some of the ideas developed in @kirbyfan64's PR #1723. Both `py.test` with no arguments and `py.test mypy/test/testcheck.py` work just as you'd hope, while `./runtests.py` continues to run all the tests. The output is very similar to the myunit output. It doesn't spew voluminous stack traces or other verbose data, and it continues using `assert_string_arrays_equal` to produce nicely-formatted comparisons when e.g. type-checker error messages differ. On error it even includes the filename and line number for the test case itself, which I've long wanted, and I think pytest's separator lines and coloration make the output slightly easier to read when there are multiple failures. The `-i` option from myunit is straightforwardly ported over as `--update-data`, giving it a longer name because it feels like the kind of heavyweight and uncommon operation that deserves such a name. It'd be equally straightforward to port over `-u`, but in the presence of source control I think `--update-data` does the job on its own. One small annoyance is that if there's a failure early in a test run, pytest doesn't print the detailed report on that failure until the whole run is over. This has often annoyed me in using pytest on other projects; useful workarounds include passing `-x` to make it stop at the first failure, `-k` to filter the set of tests to be run, or (especially with our tests where errors often go through `assert_string_arrays_equal`) `-s` to let stdout and stderr pass through immediately. For interactive use I think it'd nearly always be preferable to do what myunit does by immediately printing the detailed information, so I may come back to this later to try to get pytest to do that. We don't yet take advantage of `xdist` to parallelize within a `py.test` run (though `xdist` works for me out of the box in initial cursory testing.) For now we just stick with the `runtests.py` parallelization, so we set up a separate `py.test` command for each test module.
1 parent 67d89b9 commit 255fb61

12 files changed

Lines changed: 160 additions & 33 deletions

File tree

‎README.md‎

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -184,19 +184,26 @@ To run all tests, run the script `runtests.py` in the mypy repository:
184184
Note that some tests will be disabled for older python versions.
185185

186186
This will run all tests, including integration and regression tests,
187-
and will type check mypy and verify that all stubs are valid. You can also
188-
run unit tests only, which run pretty quickly:
189-
190-
$ ./runtests.py unit-test
187+
and will type check mypy and verify that all stubs are valid.
191188

192189
You can run a subset of test suites by passing positive or negative
193190
filters:
194191

195192
$ ./runtests.py lex parse -x lint -x stub
196193

197-
If you want to run individual unit tests, you can run `myunit` directly, or
194+
For example, to run unit tests only, which run pretty quickly:
195+
196+
$ ./runtests.py unit-test pytest
197+
198+
The unit test suites are driven by a mixture of test frameworks:
199+
mypy's own `myunit` framework, and `pytest`, which we're in the
200+
process of migrating to. For finer control over which unit tests are
201+
run and how, you can run `py.test` or `scripts/myunit` directly, or
198202
pass inferior arguments via `-a`:
199203

204+
$ py.test mypy/test/testcheck.py -v -k MethodCall
205+
$ ./runtests.py -v 'pytest mypy/test/testcheck' -a -v -a -k -a MethodCall
206+
200207
$ PYTHONPATH=$PWD scripts/myunit -m mypy.test.testlex -v '*backslash*'
201208
$ ./runtests.py mypy.test.testlex -a -v -a '*backslash*'
202209

‎conftest.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
pytest_plugins = [
2+
'mypy.test.data',
3+
]

‎mypy/myunit/__init__.py‎

Lines changed: 1 addition & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,6 @@
1414
is_quiet = False
1515
patterns = [] # type: List[str]
1616
times = [] # type: List[Tuple[float, str]]
17-
APPEND_TESTCASES = ''
18-
UPDATE_TESTCASES = False
1917

2018

2119
class AssertionFailure(Exception):
@@ -199,7 +197,6 @@ def __init__(self, suites: List[Suite]) -> None:
199197

200198
def main(args: List[str] = None) -> None:
201199
global patterns, is_verbose, is_quiet
202-
global APPEND_TESTCASES, UPDATE_TESTCASES
203200
if not args:
204201
args = sys.argv[1:]
205202
is_verbose = False
@@ -213,12 +210,6 @@ def main(args: List[str] = None) -> None:
213210
is_verbose = True
214211
elif a == '-q':
215212
is_quiet = True
216-
elif a == '-u':
217-
APPEND_TESTCASES = '.new'
218-
UPDATE_TESTCASES = True
219-
elif a == '-i':
220-
APPEND_TESTCASES = ''
221-
UPDATE_TESTCASES = True
222213
elif a == '-m':
223214
i += 1
224215
if i == len(args):
@@ -227,7 +218,7 @@ def main(args: List[str] = None) -> None:
227218
elif not a.startswith('-'):
228219
patterns.append(a)
229220
else:
230-
sys.exit('Usage: python -m mypy.myunit [-v] [-q] [-u | -i]'
221+
sys.exit('Usage: python -m mypy.myunit [-v] [-q]'
231222
+ ' -m mypy.test.module [-m mypy.test.module ...] [filter ...]')
232223
i += 1
233224
if len(patterns) == 0:

‎mypy/test/collect.py‎

Whitespace-only changes.

‎mypy/test/data.py‎

Lines changed: 80 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,21 +6,26 @@
66
from os import remove, rmdir
77
import shutil
88

9+
import pytest # type: ignore # no pytest in typeshed
910
from typing import Callable, List, Tuple, Set, Optional
1011

1112
from mypy.myunit import TestCase, SkipTestCaseException
1213

1314

1415
def parse_test_cases(
1516
path: str,
16-
perform: Callable[['DataDrivenTestCase'], None],
17+
perform: Optional[Callable[['DataDrivenTestCase'], None]],
1718
base_path: str = '.',
1819
optional_out: bool = False,
1920
include_path: str = None,
2021
native_sep: bool = False) -> List['DataDrivenTestCase']:
2122
"""Parse a file with test case descriptions.
2223
2324
Return an array of test cases.
25+
26+
NB this function and DataDrivenTestCase are shared between the
27+
myunit and pytest codepaths -- if something looks redundant,
28+
that's likely the reason.
2429
"""
2530

2631
if not include_path:
@@ -336,3 +341,77 @@ def fix_win_path(line: str) -> str:
336341
filename, lineno, message = m.groups()
337342
return '{}:{}{}'.format(filename.replace('/', '\\'),
338343
lineno or '', message)
344+
345+
346+
##
347+
#
348+
# pytest setup
349+
#
350+
##
351+
352+
353+
def pytest_addoption(parser):
354+
group = parser.getgroup('mypy')
355+
group.addoption('--update-data', action='store_true', default=False,
356+
help='Update test data to reflect actual output'
357+
' (supported only for certain tests)')
358+
359+
360+
def pytest_pycollect_makeitem(collector, name, obj):
361+
if not isinstance(obj, type) or not issubclass(obj, DataSuite):
362+
return None
363+
return MypyDataSuite(name, parent=collector)
364+
365+
366+
class MypyDataSuite(pytest.Class):
367+
def collect(self):
368+
for case in self.obj.cases():
369+
yield MypyDataCase(case.name, self, case)
370+
371+
372+
class MypyDataCase(pytest.Item):
373+
def __init__(self, name: str, parent: MypyDataSuite, obj: DataDrivenTestCase) -> None:
374+
self.skip = False
375+
if name.endswith('-skip'):
376+
self.skip = True
377+
name = name[:-len('-skip')]
378+
379+
super().__init__(name, parent)
380+
self.obj = obj
381+
382+
def runtest(self):
383+
if self.skip:
384+
pytest.skip()
385+
update_data = self.config.getoption('--update-data', False)
386+
self.parent.obj(update_data=update_data).run_case(self.obj)
387+
388+
def setup(self):
389+
self.obj.set_up()
390+
391+
def teardown(self):
392+
self.obj.tear_down()
393+
394+
def reportinfo(self):
395+
return self.obj.file, self.obj.line, self.obj.name
396+
397+
def repr_failure(self, excinfo):
398+
if excinfo.errisinstance(SystemExit):
399+
# We assume that before doing exit() (which raises SystemExit) we've printed
400+
# enough context about what happened so that a stack trace is not useful.
401+
# In particular, uncaught exceptions during semantic analysis or type checking
402+
# call exit() and they already print out a stack trace.
403+
excrepr = excinfo.exconly()
404+
else:
405+
self.parent._prunetraceback(excinfo)
406+
excrepr = excinfo.getrepr(style='short')
407+
408+
return "data: {}:{}\n{}".format(self.obj.file, self.obj.line, excrepr)
409+
410+
411+
class DataSuite:
412+
@classmethod
413+
def cases(cls) -> List[DataDrivenTestCase]:
414+
return []
415+
416+
def run_case(self, testcase: DataDrivenTestCase) -> None:
417+
raise NotImplementedError

‎mypy/test/helpers.py‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -85,9 +85,8 @@ def assert_string_arrays_equal(expected: List[str], actual: List[str],
8585
raise AssertionFailure(msg)
8686

8787

88-
def update_testcase_output(testcase: DataDrivenTestCase, output: List[str], append: str) -> None:
88+
def update_testcase_output(testcase: DataDrivenTestCase, output: List[str]) -> None:
8989
testcase_path = os.path.join(testcase.old_cwd, testcase.file)
90-
newfile = testcase_path + append
9190
data_lines = open(testcase_path).read().splitlines()
9291
test = '\n'.join(data_lines[testcase.line:testcase.lastline])
9392

@@ -111,7 +110,7 @@ def update_testcase_output(testcase: DataDrivenTestCase, output: List[str], appe
111110

112111
data_lines[testcase.line:testcase.lastline] = [test]
113112
data = '\n'.join(data_lines)
114-
with open(newfile, 'w') as f:
113+
with open(testcase_path, 'w') as f:
115114
print(data, file=f)
116115

117116

‎mypy/test/testcheck.py‎

Lines changed: 16 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,10 @@
99
from typing import Tuple, List, Dict, Set
1010

1111
from mypy import build, defaults
12-
import mypy.myunit # for mutable globals (ick!)
1312
from mypy.build import BuildSource, find_module_clear_caches
14-
from mypy.myunit import Suite, AssertionFailure
13+
from mypy.myunit import AssertionFailure
1514
from mypy.test.config import test_temp_dir, test_data_prefix
16-
from mypy.test.data import parse_test_cases, DataDrivenTestCase
15+
from mypy.test.data import parse_test_cases, DataDrivenTestCase, DataSuite
1716
from mypy.test.helpers import (
1817
assert_string_arrays_equal, normalize_error_messages,
1918
testcase_pyversion, update_testcase_output,
@@ -67,42 +66,45 @@
6766
]
6867

6968

70-
class TypeCheckSuite(Suite):
69+
class TypeCheckSuite(DataSuite):
70+
def __init__(self, *, update_data=False):
71+
self.update_data = update_data
7172

72-
def cases(self) -> List[DataDrivenTestCase]:
73+
@classmethod
74+
def cases(cls) -> List[DataDrivenTestCase]:
7375
c = [] # type: List[DataDrivenTestCase]
7476
for f in files:
7577
c += parse_test_cases(os.path.join(test_data_prefix, f),
76-
self.run_test, test_temp_dir, True)
78+
None, test_temp_dir, True)
7779
return c
7880

79-
def run_test(self, testcase: DataDrivenTestCase) -> None:
81+
def run_case(self, testcase: DataDrivenTestCase) -> None:
8082
incremental = 'incremental' in testcase.name.lower() or 'incremental' in testcase.file
8183
optional = 'optional' in testcase.file
8284
if incremental:
8385
# Incremental tests are run once with a cold cache, once with a warm cache.
8486
# Expect success on first run, errors from testcase.output (if any) on second run.
8587
# We briefly sleep to make sure file timestamps are distinct.
8688
self.clear_cache()
87-
self.run_test_once(testcase, 1)
89+
self.run_case_once(testcase, 1)
8890
time.sleep(0.1)
89-
self.run_test_once(testcase, 2)
91+
self.run_case_once(testcase, 2)
9092
elif optional:
9193
try:
9294
experiments.STRICT_OPTIONAL = True
93-
self.run_test_once(testcase)
95+
self.run_case_once(testcase)
9496
finally:
9597
experiments.STRICT_OPTIONAL = False
9698
else:
97-
self.run_test_once(testcase)
99+
self.run_case_once(testcase)
98100

99101
def clear_cache(self) -> None:
100102
dn = defaults.MYPY_CACHE
101103

102104
if os.path.exists(dn):
103105
shutil.rmtree(dn)
104106

105-
def run_test_once(self, testcase: DataDrivenTestCase, incremental=0) -> None:
107+
def run_case_once(self, testcase: DataDrivenTestCase, incremental=0) -> None:
106108
find_module_clear_caches()
107109
program_text = '\n'.join(testcase.input)
108110
module_name, program_name, program_text = self.parse_module(program_text)
@@ -140,8 +142,8 @@ def run_test_once(self, testcase: DataDrivenTestCase, incremental=0) -> None:
140142
a = e.messages
141143
a = normalize_error_messages(a)
142144

143-
if output != a and mypy.myunit.UPDATE_TESTCASES:
144-
update_testcase_output(testcase, a, mypy.myunit.APPEND_TESTCASES)
145+
if output != a and self.update_data:
146+
update_testcase_output(testcase, a)
145147

146148
assert_string_arrays_equal(
147149
output, a,

‎mypy/test/update.py‎

Whitespace-only changes.

‎mypy/waiter.py‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -281,6 +281,18 @@ def parse_test_stats_from_output(output: str, fail_type: Optional[str]) -> Tuple
281281
Return tuple (number of tests, number of test failures). Default
282282
to the entire task representing a single test as a fallback.
283283
"""
284+
285+
# pytest
286+
m = re.search('^=+ (.*) in [0-9.]+ seconds =+\n\Z', output, re.MULTILINE)
287+
if m:
288+
counts = {}
289+
for part in m.group(1).split(', '): # e.g., '3 failed, 32 passed, 345 deselected'
290+
count, key = part.split()
291+
counts[key] = int(count)
292+
return (sum(c for k, c in counts.items() if k != 'deselected'),
293+
counts.get('failed', 0))
294+
295+
# myunit
284296
m = re.search('^([0-9]+)/([0-9]+) test cases failed(, ([0-9]+) skipped)?.$', output,
285297
re.MULTILINE)
286298
if m:
@@ -289,6 +301,7 @@ def parse_test_stats_from_output(output: str, fail_type: Optional[str]) -> Tuple
289301
re.MULTILINE)
290302
if m:
291303
return int(m.group(1)), 0
304+
292305
# Couldn't find test counts, so fall back to single test per tasks.
293306
if fail_type is not None:
294307
return 1, 1

‎pytest.ini‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
[pytest]
2+
# testpaths is new in 2.8
3+
minversion = 2.8
4+
5+
testpaths = mypy/test
6+
7+
python_files = test*.py
8+
9+
# empty patterns for default python collector, to stick to our plugin's collector
10+
python_classes =
11+
python_functions =

0 commit comments

Comments
 (0)