Closed
Description
For the last couple months, it seems buildbots have been failing -m test test_unittest -R3:3, then re-trying the 3 failed test cases and passing on the re-try. Look at https://buildbot.python.org/all/#/builders/123 for example. The tests consistently pass the first warmup run, then fail on the second warmup run.
Example stdout from running -m test test_unittest -R3:3 -v:
======================================================================
FAIL: test_loadTestsFromName__module_not_loaded (test.test_unittest.test_loader.Test_TestLoader.test_loadTestsFromName__module_not_loaded)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Users\sween\Source\Repos\cpython2\cpython\Lib\test\test_unittest\test_loader.py", line 594, in test_loadTestsFromName__module_not_loaded
self.assertEqual(list(suite), [])
AssertionError: Lists differ: [<unittest.loader._FailedTest testMethod=dummy>] != []
First list contains 1 additional elements.
First extra element 0:
<unittest.loader._FailedTest testMethod=dummy>
- [<unittest.loader._FailedTest testMethod=dummy>]
+ []
======================================================================
FAIL: test_loadTestsFromNames__module_not_loaded (test.test_unittest.test_loader.Test_TestLoader.test_loadTestsFromNames__module_not_loaded)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Users\sween\Source\Repos\cpython2\cpython\Lib\test\test_unittest\test_loader.py", line 1019, in test_loadTestsFromNames__module_not_loaded
self.assertEqual(list(suite), [unittest.TestSuite()])
AssertionError: Lists differ: [<uni[15 chars]tSuite tests=[<unittest.loader._FailedTest testMethod=dummy>]>] != [<uni[15 chars]tSuite tests=[]>]
First differing element 0:
<unit[13 chars]stSuite tests=[<unittest.loader._FailedTest testMethod=dummy>]>
<unit[13 chars]stSuite tests=[]>
- [<unittest.suite.TestSuite tests=[<unittest.loader._FailedTest testMethod=dummy>]>]
+ [<unittest.suite.TestSuite tests=[]>]
======================================================================
FAIL: test_loadTestsFromNames__unknown_attr_name (test.test_unittest.test_loader.Test_TestLoader.test_loadTestsFromNames__unknown_attr_name)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Users\sween\Source\Repos\cpython2\cpython\Lib\test\test_unittest\test_loader.py", line 715, in test_loadTestsFromNames__unknown_attr_name
error, test = self.check_deferred_error(loader, list(suite)[0])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\sween\Source\Repos\cpython2\cpython\Lib\test\test_unittest\test_loader.py", line 620, in check_deferred_error
self.assertEqual(1, len(loader.errors))
AssertionError: 1 != 2
----------------------------------------------------------------------
Ran 1001 tests in 8.861s
FAILED (failures=3, skipped=55)
test_unittest failed (3 failures)
== Tests result: FAILURE ==
1 test failed:
test_unittest
Total duration: 18.3 sec
Tests result: FAILURE
I bisected to here:
commit c735d54
Author: Victor Stinner [email protected]
Date: Tue Jun 21 10:27:59 2022 +0200gh-93839: Move Lib/unttest/test/ to Lib/test/test_unittest/ (#94043) * Move Lib/unittest/test/ to Lib/test/test_unittest/ * Remove Lib/test/test_unittest.py * Replace unittest.test with test.test_unittest * Remove unittest.load_tests() * Rewrite unittest __init__.py and __main__.py * Update build system, CODEOWNERS, and wasm_assets.py
cc @vstinner