Thanks to visit codestin.com
Credit goes to github.com

Skip to content

bpo-31961: Fix support of path-like executables in subprocess. #5914

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
May 28, 2019
Merged

bpo-31961: Fix support of path-like executables in subprocess. #5914

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
9e87239
bpo-31961: Fix support of path-like executables in subprocess.
serhiy-storchaka Feb 26, 2018
8cce344
More changes:
serhiy-storchaka Feb 27, 2018
0b4e876
Revert "Revert "bpo-31961: subprocess now accepts path-like args (GH-…
serhiy-storchaka Feb 28, 2018
e72cf67
Merge branch 'revert-revert-31961' into subprocess-pathlike
serhiy-storchaka Feb 28, 2018
ccd3464
Restore NEWS.
serhiy-storchaka Feb 28, 2018
291b0c6
Merge branch 'master' into subprocess-pathlike
serhiy-storchaka Mar 2, 2018
cee318b
Minimize docs changes.
serhiy-storchaka Mar 27, 2018
43eef6c
Merge branch 'master' into subprocess-pathlike
serhiy-storchaka Mar 27, 2018
5c0dd88
Retarget to 3.8 and add a news entry.
serhiy-storchaka Mar 27, 2018
ab8ebfe
Merge branch 'master' into subprocess-pathlike
serhiy-storchaka Jul 8, 2018
80f0af3
Merge branch 'master' into subprocess-pathlike
serhiy-storchaka Sep 18, 2018
9a0e746
Merge branch 'master' into subprocess-pathlike
serhiy-storchaka Sep 19, 2018
148b13d
Merge branch 'master' into subprocess-pathlike
serhiy-storchaka Oct 30, 2018
6a8d394
Add support for bytes and path-like args.
serhiy-storchaka Oct 30, 2018
10fba36
Merge branch 'master' into subprocess-pathlike
serhiy-storchaka Nov 18, 2018
7d51e79
Improve error messages for invalid args.
serhiy-storchaka Nov 18, 2018
1073a0f
Fix the NEWS wording to be correct English.
gpshead May 18, 2019
fe37262
Merge branch 'master' into subprocess-pathlike
serhiy-storchaka May 25, 2019
c6b0ade
Merge branch 'subprocess-pathlike' of github.com:serhiy-storchaka/cpy…
serhiy-storchaka May 25, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 26 additions & 3 deletions Doc/library/subprocess.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -347,7 +347,8 @@ functions.
the class uses the Windows ``CreateProcess()`` function. The arguments to
:class:`Popen` are as follows.

*args* should be a sequence of program arguments or else a single string.
*args* should be a sequence of program arguments or else a single string
or :term:`path-like object`.
By default, the program to execute is the first item in *args* if *args* is
a sequence. If *args* is a string, the interpretation is
platform-dependent and described below. See the *shell* and *executable*
Expand Down Expand Up @@ -381,6 +382,15 @@ functions.
manner described in :ref:`converting-argument-sequence`. This is because
the underlying ``CreateProcess()`` operates on strings.

.. versionchanged:: 3.6
*args* parameter accepts a :term:`path-like object` if *shell* is
``False`` and a sequence containing path-like objects on POSIX.

.. versionchanged:: 3.8
*args* parameter accepts a :term:`path-like object` if *shell* is
``False`` and a sequence containing bytes and path-like objects
on Windows.

The *shell* argument (which defaults to ``False``) specifies whether to use
the shell as the program to execute. If *shell* is ``True``, it is
recommended to pass *args* as a string rather than as a sequence.
Expand Down Expand Up @@ -436,6 +446,13 @@ functions.
:program:`ps`. If ``shell=True``, on POSIX the *executable* argument
specifies a replacement shell for the default :file:`/bin/sh`.

.. versionchanged:: 3.6
*executable* parameter accepts a :term:`path-like object` on POSIX.

.. versionchanged:: 3.8
*executable* parameter accepts a bytes and :term:`path-like object`
on Windows.

*stdin*, *stdout* and *stderr* specify the executed program's standard input,
standard output and standard error file handles, respectively. Valid values
are :data:`PIPE`, :data:`DEVNULL`, an existing file descriptor (a positive
Expand Down Expand Up @@ -492,13 +509,19 @@ functions.
The *pass_fds* parameter was added.

If *cwd* is not ``None``, the function changes the working directory to
*cwd* before executing the child. *cwd* can be a :class:`str` and
*cwd* before executing the child. *cwd* can be a string, bytes or
:term:`path-like <path-like object>` object. In particular, the function
looks for *executable* (or for the first item in *args*) relative to *cwd*
if the executable path is a relative path.

.. versionchanged:: 3.6
*cwd* parameter accepts a :term:`path-like object`.
*cwd* parameter accepts a :term:`path-like object` on POSIX.

.. versionchanged:: 3.7
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI - these 3.6 and 3.7 versionchanged updates should be made to the docs today to clarify the existing behavior on their own (and backported to at least the 3.7 docs)

*cwd* parameter accepts a :term:`path-like object` on Windows.

.. versionchanged:: 3.8
*cwd* parameter accepts a bytes object on Windows.

If *restore_signals* is true (the default) all signals that Python has set to
SIG_IGN are restored to SIG_DFL in the child process before the exec.
Expand Down
25 changes: 22 additions & 3 deletions Lib/subprocess.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -521,7 +521,7 @@ def list2cmdline(seq):
# "Parsing C++ Command-Line Arguments"
result = []
needquote = False
for arg in seq:
for arg in map(os.fsdecode, seq):
bs_buf = []

# Add a space to separate this argument from the others
Expand Down Expand Up @@ -1203,9 +1203,23 @@ def _execute_child(self, args, executable, preexec_fn, close_fds,

assert not pass_fds, "pass_fds not supported on Windows."

if not isinstance(args, str):
if isinstance(args, str):
pass
elif isinstance(args, bytes):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The elif chain could be simplified to have less repeated code:

elif isinstance(args, (bytes, os.PathLike)):
    if shell:
        raise TypeError(f'{type(args)} args not allowed when shell=True on Windows')`
    args = list2cmdline([args])

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Error messages should be different. Path-like args is not allowed on all platforms, bytes args is not allowed only on Windows.

if shell:
raise TypeError('bytes args is not allowed on Windows')
args = list2cmdline([args])
elif isinstance(args, os.PathLike):
if shell:
raise TypeError('path-like args is not allowed when '
'shell is true')
args = list2cmdline([args])
else:
args = list2cmdline(args)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be addressed in list2cmdline to be consistent with how POSIX converts every element of args via PyUnicode_FSConverter. For example:

if isinstance(args, bytes):
    args = os.fsdecode(args)
elif isinstance(args, os.PathLike):
    if shell:
        raise ValueError('PathLike args is not allowed when '
                         'shell is true.')
    else:
        args = [args]

if not isinstance(args, str):
    args = list2cmdline(args)

if executable is not None:
    executable = os.fsdecode(executable)

Then in list2cmdline, decode the items via os.fsdecode :

for arg in seq:
    if not isinstance(arg, str):
        arg = os.fsdecode(arg)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does look like it would be nicer, allowing more mixing and matching of types.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eryksun 's comment is related to the old version of the PR. I addressed it, except that that I decided to reject bytes args on Windows with shell=True as it is ambiguous and is not actually needed. It is enough to have a difference in interpreting str args.


if executable is not None:
executable = os.fsdecode(executable)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still think bytes and pathlike should be supported for every item of args. The POSIX implementation doesn't limit this to args[0] (though not documented), and I think that's proper. A script shouldn't have to special-case calling os.fsdecode for a pathlike argument. That's inefficient for POSIX, for which os.fsencode would be better. While os.fspath would be efficient, it's insufficient because it might return bytes on Windows.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The POSIX implementation accepts also bytes as args and as any item of args. If allow path-like arguments, we should consider also adding support of bytes arguments and program name.

# Process startup details
if startupinfo is None:
startupinfo = STARTUPINFO()
Expand Down Expand Up @@ -1262,7 +1276,7 @@ def _execute_child(self, args, executable, preexec_fn, close_fds,
int(not close_fds),
creationflags,
env,
os.fspath(cwd) if cwd is not None else None,
os.fsdecode(cwd) if cwd is not None else None,
startupinfo)
finally:
# Child is launched. Close the parent's copy of those pipe
Expand Down Expand Up @@ -1510,6 +1524,11 @@ def _execute_child(self, args, executable, preexec_fn, close_fds,

if isinstance(args, (str, bytes)):
args = [args]
elif isinstance(args, os.PathLike):
if shell:
raise TypeError('path-like args is not allowed when '
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for consistency of error messages with my suggestion above I think TypeError(f'{type(args)} args is not allowed when ' ...). It is good for that to render as <class os.PathLike> as that is less confusing to the error message reader than "path-like" which they might be more likely to mis-interpret in English as suggesting anything that looks like a path within a string.

'shell is true')
args = [args]
else:
args = list(args)

Expand Down
55 changes: 55 additions & 0 deletions Lib/test/test_subprocess.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -304,6 +304,18 @@ def test_executable(self):
"doesnotexist")
self._assert_python([doesnotexist, "-c"], executable=sys.executable)

def test_bytes_executable(self):
doesnotexist = os.path.join(os.path.dirname(sys.executable),
"doesnotexist")
self._assert_python([doesnotexist, "-c"],
executable=os.fsencode(sys.executable))

def test_pathlike_executable(self):
doesnotexist = os.path.join(os.path.dirname(sys.executable),
"doesnotexist")
self._assert_python([doesnotexist, "-c"],
executable=FakePath(sys.executable))

def test_executable_takes_precedence(self):
# Check that the executable argument takes precedence over args[0].
#
Expand All @@ -320,6 +332,16 @@ def test_executable_replaces_shell(self):
# when shell=True.
self._assert_python([], executable=sys.executable, shell=True)

@unittest.skipIf(mswindows, "executable argument replaces shell")
def test_bytes_executable_replaces_shell(self):
self._assert_python([], executable=os.fsencode(sys.executable),
shell=True)

@unittest.skipIf(mswindows, "executable argument replaces shell")
def test_pathlike_executable_replaces_shell(self):
self._assert_python([], executable=FakePath(sys.executable),
shell=True)

# For use in the test_cwd* tests below.
def _normalize_cwd(self, cwd):
# Normalize an expected cwd (for Tru64 support).
Expand Down Expand Up @@ -358,6 +380,11 @@ def test_cwd(self):
temp_dir = self._normalize_cwd(temp_dir)
self._assert_cwd(temp_dir, sys.executable, cwd=temp_dir)

def test_cwd_with_bytes(self):
temp_dir = tempfile.gettempdir()
temp_dir = self._normalize_cwd(temp_dir)
self._assert_cwd(temp_dir, sys.executable, cwd=os.fsencode(temp_dir))

def test_cwd_with_pathlike(self):
temp_dir = tempfile.gettempdir()
temp_dir = self._normalize_cwd(temp_dir)
Expand Down Expand Up @@ -1473,6 +1500,34 @@ def test_run_kwargs(self):
env=newenv)
self.assertEqual(cp.returncode, 33)

def test_run_with_pathlike_path(self):
# bpo-31961: test run(pathlike_object)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like to make the first line descriptive comment a docstring. it shows up nicely in verbose testing mode.

# the name of a command that can be run without
# any argumenets that exit fast
prog = 'tree.com' if mswindows else 'ls'
path = shutil.which(prog)
if path is None:
self.skipTest(f'{prog} required for this test')
path = FakePath(path)
res = subprocess.run(path, stdout=subprocess.DEVNULL)
self.assertEqual(res.returncode, 0)
with self.assertRaises(TypeError):
subprocess.run(path, stdout=subprocess.DEVNULL, shell=True)

def test_run_with_bytes_path_and_arguments(self):
# bpo-31961: test run([bytes_object, b'additional arguments'])
path = os.fsencode(sys.executable)
args = [path, '-c', b'import sys; sys.exit(57)']
res = subprocess.run(args)
self.assertEqual(res.returncode, 57)

def test_run_with_pathlike_path_and_arguments(self):
# bpo-31961: test run([pathlike_object, 'additional arguments'])
path = FakePath(sys.executable)
args = [path, '-c', 'import sys; sys.exit(57)']
res = subprocess.run(args)
self.assertEqual(res.returncode, 57)

def test_capture_output(self):
cp = self.run_python(("import sys;"
"sys.stdout.write('BDFL'); "
Expand Down
6 changes: 6 additions & 0 deletions Misc/NEWS.d/next/Library/2018-03-27-13-28-16.bpo-31961.GjLoYu.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
Added support for bytes and path-like objects in :func:`subprocess.Popen`
on Windows. The *args* parameter now accepts a :term:`path-like object` if
*shell* is ``False`` and a sequence containing bytes and path-like objects.
The *executable* parameter now accepts a bytes and :term:`path-like object`.
The *cwd* parameter now accepts a bytes object.
Based on patch by Anders Lorentsen.