Not all tests are installed. #115421

Yhg1s · 2024-02-13T17:31:41Z

Bug report

Bug description:

Not all of the tests in Lib/test are being installed, which means some tests are (silently) not run from an installed python:

% /tmp/testinstall/bin/python3 -m test test_multiprocessing_fork
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/tmp/testinstall/lib/python3.13/test/__main__.py", line 2, in <module>
    main(_add_python_opts=True)
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/testinstall/lib/python3.13/test/libregrtest/main.py", line 680, in main
    Regrtest(ns, _add_python_opts=_add_python_opts).main(tests=tests)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/tmp/testinstall/lib/python3.13/test/libregrtest/main.py", line 662, in main
    selected, tests = self.find_tests(tests)
                      ~~~~~~~~~~~~~~~^^^^^^^
  File "/tmp/testinstall/lib/python3.13/test/libregrtest/main.py", line 198, in find_tests
    selected = split_test_packages(selected)
               ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/tmp/testinstall/lib/python3.13/test/libregrtest/findtests.py", line 70, in split_test_packages
    splitted.extend(findtests(testdir=subdir, exclude=exclude,
                    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                              split_test_dirs=split_test_dirs,
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                              base_mod=name))
                              ^^^^^^^^^^^^^^
  File "/tmp/testinstall/lib/python3.13/test/libregrtest/findtests.py", line 43, in findtests
    for name in os.listdir(testdir):
                ~~~~~~~~~~^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/testinstall/lib/python3.13/test/test_multiprocessing_fork'

From a quick look it seems we're missing test_concurrent_futures, test_interpreters, test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn. This is a problem in 3.12 and earlier as well.

Besides tests it's also missing the test.support.interpreters package:

% ls Lib/test/support/interpreters/
channels.py  __init__.py  queues.py
% ls /tmp/testinstall/lib/python3.13/test/support/interpreters
ls: cannot access '/tmp/testinstall/lib/python3.13/test/support/interpreters': No such file or directory

... which has not caused issues because its main user, test/test_subinterpreters, isn't being installed, and the other uses of it in the testsuite import ignore the import error if it's not available.

This is a problem back to at least 3.10 (which is missing test_capi as well as various testdata directories).

We should either make the Makefile install all of the test subdirectories automatically, or have a CI check to make sure all subdirectories are listed in TESTSUBDIRS in the Makefile.

CPython versions tested on:

3.10, 3.11, 3.12, CPython main branch

Operating systems tested on:

No response

Linked PRs

The text was updated successfully, but these errors were encountered:

vstinner · 2024-02-13T17:42:30Z

cc @sobolevn: oh, this bug is bad :-(

sobolevn · 2024-02-13T18:02:49Z

I somehow dodged this bullet this time: none of the directories are mine.
But, I had this bug several times. I think that we need a better check for that.

I will send a PR with the proposed check that we can run in CI.

Update the list of installed test subdirectories with all newly added subdirectories of Lib/test, so that the tests in those directories are properly installed.

Yhg1s · 2024-02-14T00:02:29Z

I've fixed main by updating the list so they're included in the upcoming 3.13.0a4 release. For main we should try and figure out how to avoid the explicit list of test subdirectories to install, so I'll leave the bug open for that. For 3.12-3.10, though, I'm not sure it's worth fixing (and it changes what's getting installed, which is uncomfortable in a patch release), and I don't really want to try to do any invasive changes to the Makefile to avoid listing the directories in them.

…ython#115422) Update the list of installed test subdirectories with all newly added subdirectories of Lib/test, so that the tests in those directories are properly installed.

)

…ythonGH-115813)

encukou · 2024-03-07T11:53:10Z

The test is in -- thank you, @sobolevn!

The change to Makefile isn't that invasive, so you might want to reconsider if this is good for 3.12. PR here: #116462

Yhg1s · 2024-03-07T13:21:38Z

Now it's down to @pablogsal to decide if it's worth backporting to 3.11 :) I believe the missing directories in 3.11, and thus tests that aren't being run from installed Python (like during the release process) are test_concurrent_futures and test_multiprocessing_*.

…H-115813) (#116462) * gh-115421: Test that our Makefile has all needed test folders (GH-115813) * Update the list of installed test subdirectories --------- Co-authored-by: Nikita Sobolev <[email protected]>

pablogsal · 2024-03-07T22:22:10Z

Now it's down to @pablogsal to decide if it's worth backporting to 3.11

I am happy backporting it to 3.11 👍

…hem (pythonGH-115813) (pythonGH-116462) * pythongh-115421: Test that our Makefile has all needed test folders (pythonGH-115813) * Update the list of installed test subdirectories --------- Co-authored-by: Nikita Sobolev <[email protected]>

…H-115813) This backports: - GH-115813 - GH-115422 Unlike on the main branch, new directories are added to the end, so they're a bit easier to patch out if a redistributor needs to do so. On main & 3.12, there's a special case for `idlelib/idle_test`; on 3.11 TESTSUBDIRS has several more entries that are not in `test/`. This backport ignores all of them (including idlelib). (The alternative would be list them, as additions to TEST_HOME_DIR. But that's probably too invasive; people might split stdlib up in surprising ways.) Co-authored-by: Nikita Sobolev <[email protected]>

encukou · 2024-03-11T11:58:50Z

Thanks to everyone involved :)

…ythonGH-115813)

Yhg1s added the type-bug An unexpected behavior, bug, or error label Feb 13, 2024

bedevere-app bot mentioned this issue Feb 13, 2024

gh-115421: Update the list of installed test subdirectories. #115422

Merged

befeleme mentioned this issue Feb 15, 2024

ScriptTestCase.test_script_run fails with Python 3.13.0a3 pypa/distlib#214

Closed

sobolevn added a commit to sobolevn/cpython that referenced this issue Feb 15, 2024

pythongh-115421: List all test subdirs in Makefile by makesetup

5faf6e2

bedevere-app bot mentioned this issue Feb 15, 2024

gh-115421: List all test subdirs in Makefile by makesetup #115511

Closed

sobolevn added a commit to sobolevn/cpython that referenced this issue Feb 22, 2024

pythongh-115421: Test that our Makefile has all needed test folders

6a1ab15

sobolevn added a commit to sobolevn/cpython that referenced this issue Feb 22, 2024

pythongh-115421: Test that our Makefile has all needed test folders

d9f6baf

bedevere-app bot mentioned this issue Feb 22, 2024

gh-115421: Test that our Makefile has all needed test folders #115813

Merged

sobolevn added a commit to sobolevn/cpython that referenced this issue Feb 22, 2024

pythongh-115421: Test that our Makefile has all needed test folders

63ca14b

sobolevn added a commit to sobolevn/cpython that referenced this issue Feb 22, 2024

pythongh-115421: Test that our Makefile has all needed test folders

b04014f

encukou pushed a commit that referenced this issue Mar 7, 2024

gh-115421: Test that our Makefile has all needed test folders (GH-115813

72dbea2

)

encukou pushed a commit to encukou/cpython that referenced this issue Mar 7, 2024

pythongh-115421: Test that our Makefile has all needed test folders (p…

f37cd4b

…ythonGH-115813)

encukou pushed a commit to encukou/cpython that referenced this issue Mar 7, 2024

pythongh-115421: Test that our Makefile has all needed test folders (p…

3b51b14

…ythonGH-115813)

bedevere-app bot mentioned this issue Mar 7, 2024

[3.12] gh-115421: List all test subdirs in Makefile, and test them (GH-115813) #116462

Merged

bedevere-app bot mentioned this issue Mar 8, 2024

[3.11] gh-115421: List all test/ subdirs in Makefile, and test them (GH-115813) #116498

Merged

encukou closed this as completed Mar 11, 2024

adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024

pythongh-115421: Test that our Makefile has all needed test folders (p…

6ac9f0b

…ythonGH-115813)

diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024

pythongh-115421: Test that our Makefile has all needed test folders (p…

0221b8d

…ythonGH-115813)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Not all tests are installed. #115421

Not all tests are installed. #115421

Yhg1s commented Feb 13, 2024 •

edited by bedevere-app bot

Loading

vstinner commented Feb 13, 2024

Uh oh!

sobolevn commented Feb 13, 2024

Uh oh!

Yhg1s commented Feb 14, 2024

Uh oh!

encukou commented Mar 7, 2024

Uh oh!

Yhg1s commented Mar 7, 2024

Uh oh!

pablogsal commented Mar 7, 2024 •

edited

Loading

Uh oh!

encukou commented Mar 11, 2024

Uh oh!

Uh oh!

Not all tests are installed. #115421

Not all tests are installed. #115421

Comments

Yhg1s commented Feb 13, 2024 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

vstinner commented Feb 13, 2024

Uh oh!

sobolevn commented Feb 13, 2024

Uh oh!

Yhg1s commented Feb 14, 2024

Uh oh!

encukou commented Mar 7, 2024

Uh oh!

Yhg1s commented Mar 7, 2024

Uh oh!

pablogsal commented Mar 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

encukou commented Mar 11, 2024

Uh oh!

Yhg1s commented Feb 13, 2024 •

edited by bedevere-app bot

Loading

pablogsal commented Mar 7, 2024 •

edited

Loading