WIP: gh-114467: Support posix_spawn in subprocess.Popen with cwd != None #114468

kulikjak · 2024-01-23T08:29:19Z

This is a work in progress as there is still one issue to solve:

Running subprocess.run(["/usr/bin/ls"], cwd="/nonexistent") should fail with:
FileNotFoundError: [Errno 2] No such file or directory: '/nonexistent'

but with this PR, it fails with:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/ls'

which is incorrect, because it's not '/usr/bin/ls' that is missing, it's the directory. I have yet to determine how to fix this.

(this also causes test.test_subprocess.POSIXProcessTestCase.test_exception_cwd to fail).

Issue: Add support for posix_spawn call in subprocess.Popen with cwd != None #114467

📚 Documentation preview 📚: https://cpython-previews--114468.org.readthedocs.build/

kulikjak · 2024-01-23T08:43:32Z

Oh, and based on my quick search, I just realized that some systems have posix_spawn_file_actions_addchdir_np (OpenBSD, glib, MacOS), some have posix_spawn_file_actions_addchdir (NetBSD), and some have both (Oracle Solaris). That is another thing that needs some work.

erlend-aasland · 2024-01-23T09:00:22Z

Marking this as a draft (while it is WIP).

gpshead · 2024-01-23T09:06:04Z

The test_exception_cwd behavior was added for a recent bugfix. In the case of posix_spawn, I don't believe we can differentiate error causes when a chdir action is used. All we get is the errno but no indication as to which operation internal to posix_spawn it came from. So for user friendliness we probably want a compromise error message when posix_spawn was used and a cwd was specified on errors that might involve a path that includes both paths in the error message and indicates that we don't have information as to which the error applies to. (while we could attempt to introspect the system and guess, that wouldn't be technically correct due to timing issues and more complicated, so it's better not to do that)

serhiy-storchaka

Please add an entry in What's New.

Lib/subprocess.py

kulikjak · 2024-01-23T13:36:05Z

The test_exception_cwd behavior was added for a recent bugfix. In the case of posix_spawn, I don't believe we can differentiate error causes when a chdir action is used. All we get is the errno but no indication as to which operation internal to posix_spawn it came from. So for user friendliness we probably want a compromise error message when posix_spawn was used and a cwd was specified on errors that might involve a path that includes both paths in the error message and indicates that we don't have information as to which the error applies to.

Thanks! My experimentation led me to the same conclusion that determining the problematic path might not be feasible. Having both paths in the exception message seems like a good solution; I will look into it.

…chdir_np

Co-authored-by: Serhiy Storchaka <[email protected]>

…wd when using posix_spawn.

kulikjak · 2025-03-24T15:57:43Z

Sorry it took me so long, but I finally found some time to move this further. The exception is now better, although the fact that it's not easily possible to find out whether it's the command or cwd that doesn't exist is IMHO not exactly elegant...

kulikjak · 2025-03-24T16:00:42Z

Here is how the exception looks now:

Traceback (most recent call last):
  File "/scratch/with.py", line 6, in <module>
    res = subprocess.run(["/usr/bin/ls"], cwd="/nonexistent")
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 548, in run
    with Popen(*popenargs, **kwargs) as process:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 1028, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.11/subprocess.py", line 1846, in _execute_child
    self._posix_spawn(args, executable, env, restore_signals,
  File "/usr/lib/python3.11/subprocess.py", line 1790, in _posix_spawn
    self.pid = os.posix_spawn(executable, args, env, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: Either '/usr/bin/ls' or '/nonexistent' doesn't exist.

serhiy-storchaka · 2025-03-27T10:15:19Z

Modules/posixmodule.c

+                Py_XDECREF(*cwd);
+                *cwd = path;


POSIX_SPAWN_CHDIR can be passed multiple times with different arguments. In thet case the temporary bytes object saved in *cwd can be deallocated, and the posix_spawn_file_actions_addchdir_np() argument can became a bad pointer.

I think that we need to use temp_buffer here.

Please add a test for posix_spawn() with multiple POSIX_SPAWN_CHDIR actions.

Yeah, I am not sure how to handle multiple POSIX_SPAWN_CHDIR actions.

We can either:

call posix_spawn_file_actions_addchdir_np for each POSIX_SPAWN_CHDIR in which case posix_spawn will fail when any of those doesn't exist (or when other error occurs as you pointed out below) even if that one wouldn't be the final one. In such case, the error message might be pretty big.

we can omit all but the last one, in which case it will not behave as some might expect. But we would have only two paths to display in the error message.

As for the bad pointer, it's possible, but cwd always holds the latest one so whether it's called with one or muliple ones shouldn't matter here? But I might be completely wrong. Also, if we want to save all the paths (in case 1. is the prefered one), that might change.

posix_spawn_file_actions_addchdir_np also affects how files are found for later posix_spawn_file_actions_add_open, so you need to do them all.

serhiy-storchaka · 2025-03-27T10:30:45Z

Modules/posixmodule.c

@@ -7518,6 +7543,17 @@ py_posix_spawn(int use_posix_spawnp, PyObject *module, path_t *path, PyObject *a

    if (err_code) {
        errno = err_code;
+#ifdef HAVE_POSIX_SPAWN_FILE_ACTIONS_ADDCHDIR_NP
+        if (errno == ENOENT && cwd != NULL) {


This is not the only common error possible here. For example, you can get EPERM or EACCES if any of the executable or the cwd paths are not readable for the user, or ENAMETOOLONG if they are too long, or ELOOP if they contain symlink loops, or ENOTDIR if any of intermediate path component is not a directory. It is not possible to handle all errors.

Maybe use PyErr_SetFromErrnoWithFilenameObjects() (note s at end) with too path arguments? It will perhaps not work if there are multiple POSIX_SPAWN_CHDIR actions, but this is an extremely uncommon case.

The problem with PyErr_SetFromErrnoWithFilenameObjects is that the exception looks like this:

File "/usr/lib/python3.13/subprocess.py", line 1799, in _posix_spawn self.pid = os.posix_spawn(executable, args, env, **kwargs) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/ls' -> b'/nonexistent'

and I don't like the arrow, which is why I used the PyErr_Format. But otherwise it works nicely and the change is simpler:

-PyErr_SetFromErrnoWithFilenameObject(PyExc_OSError, path->object); +PyErr_SetFromErrnoWithFilenameObjects(PyExc_OSError, path->object, cwd);

Multiple POSIX_SPAWN_CHDIR actions make it more complicated...

kulikjak requested review from gpshead, erlend-aasland and corona10 as code owners January 23, 2024 08:29

bedevere-app bot added the awaiting review label Jan 23, 2024

bedevere-app bot mentioned this pull request Jan 23, 2024

Add support for posix_spawn call in subprocess.Popen with cwd != None #114467

Open

erlend-aasland marked this pull request as draft January 23, 2024 09:00

bedevere-app bot removed the awaiting review label Jan 23, 2024

serhiy-storchaka reviewed Jan 23, 2024

View reviewed changes

Lib/subprocess.py Outdated Show resolved Hide resolved

kulikjak and others added 6 commits March 24, 2025 16:41

Add configure check for posix_spawn_file_actions_addchdir_np

9c1aefc

Add support for os.POSIX_SPAWN_CHDIR and posix_spawn_file_actions_add…

655e555

…chdir_np

Add part of documentation

915e021

Minor fix

6a1b61a

better check for cwd being empty

0be683f

Co-authored-by: Serhiy Storchaka <[email protected]>

Implement better error reporting for non-existent executable and/or c…

12553e1

…wd when using posix_spawn.

kulikjak force-pushed the posix_spawn-cwd-support branch from 611a339 to 12553e1 Compare March 24, 2025 15:52

Merge branch 'python:main' into posix_spawn-cwd-support

d4c31c1

fix reference counting

7f77bf9

serhiy-storchaka reviewed Mar 27, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

WIP: gh-114467: Support posix_spawn in subprocess.Popen with cwd != None #114468

WIP: gh-114467: Support posix_spawn in subprocess.Popen with cwd != None #114468

Uh oh!

kulikjak commented Jan 23, 2024 •

edited by github-actions bot

Loading

Uh oh!

kulikjak commented Jan 23, 2024

Uh oh!

erlend-aasland commented Jan 23, 2024

Uh oh!

gpshead commented Jan 23, 2024

Uh oh!

serhiy-storchaka left a comment

Uh oh!

Uh oh!

kulikjak commented Jan 23, 2024

Uh oh!

kulikjak commented Mar 24, 2025

Uh oh!

kulikjak commented Mar 24, 2025

Uh oh!

serhiy-storchaka Mar 27, 2025

Uh oh!

kulikjak Apr 2, 2025

Uh oh!

dmcooke May 28, 2025

Uh oh!

serhiy-storchaka Mar 27, 2025

Uh oh!

kulikjak Apr 2, 2025

Uh oh!

Uh oh!

Uh oh!

WIP: gh-114467: Support posix_spawn in subprocess.Popen with cwd != None #114468

Are you sure you want to change the base?

WIP: gh-114467: Support posix_spawn in subprocess.Popen with cwd != None #114468

Uh oh!

Conversation

kulikjak commented Jan 23, 2024 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kulikjak commented Jan 23, 2024

Uh oh!

erlend-aasland commented Jan 23, 2024

Uh oh!

gpshead commented Jan 23, 2024

Uh oh!

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kulikjak commented Jan 23, 2024

Uh oh!

kulikjak commented Mar 24, 2025

Uh oh!

kulikjak commented Mar 24, 2025

Uh oh!

serhiy-storchaka Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

kulikjak Apr 2, 2025

Choose a reason for hiding this comment

Uh oh!

dmcooke May 28, 2025

Choose a reason for hiding this comment

Uh oh!

serhiy-storchaka Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

kulikjak Apr 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kulikjak commented Jan 23, 2024 •

edited by github-actions bot

Loading