bpo-32146: multiprocessing freeze_support needed outside win32 #5195
Conversation
| return None | ||
|
|
||
| # command ends with main(fd) - extract fd | ||
| r = int(argv[-1].rsplit('(')[1].split(')')[0]) |
There was a problem hiding this comment.
Here we are parsing what gets produced in semaphore_tracker.SemaphoreTracker.ensure_running. It goes to semaphore_tracker.main - a single integer.
Lib/multiprocessing/spawn.py
Outdated
| # listener_fd and alive_r are integers | ||
| # preload is a list | ||
| # kwds map strings to lists | ||
| main_args = argv[-1].split('main(')[1].rsplit(')', 1)[0].split(', ', 3) |
There was a problem hiding this comment.
Here we are parsing a more difficult expression, which is produced by forkserver.ForkServer.ensure_running. It goes to forkserver.main.
The listener_fd and alive_r arguments are integers. The preload argument is a list, hence I have introduced ast.literal_eval.
The keyword arguments come to us as **{'sys_path': ['list_of_paths'], 'main_path': ['list_of_paths']}. I used literal_eval here as the least awkward option.
There was a problem hiding this comment.
An alternative to the splitting stuff, since ast is already in the mix:
parsed_cmd = ast.parse(argv[-1])
args = [ast.literal_eval(parsed_cmd.body[1].value.args[i]) for i in range(3)]
kwds = ast.literal_eval(parsed_cmd.body[1].value.keywords[0].value)Needed in the "multiple preload modules" case.
|
Oops, I didn't mean to request that review. Apologies! |
NP, it happened automatically and you couldn't do anything about that even if you wanted to :-) |
|
@applio Would you be willing to review this? |
|
FYI, PR #5850 adds a note about this to the docs. It would probably good to that one in for 3.7. I expect this PR will go through some revisions, but the docs one should be easy. |
|
@pitrou yet another multiprocessing change ;-) It's also related to https://bugs.python.org/issue32146 |
This PR un-breaks "frozen" executables (e.g., those produced by PyInstaller or cx_Freeze that (a) run on Unix, and (b) use
multiprocessing, and (c) use thespawnorforkserverstart methods.I linked to this dissection of the problem in the issue; it is a good overview of what goes wrong currently. In a nutshell, the
forkserverandsemaphore_trackermodules try to launch new processes with command line arguments that the "frozen" executable doesn't support:When using argparse, the error is clear:
On Windows this problem is avoided because
freeze_supportcaptures the arguments, parses them, executes the appropriate code, and then exits here. It recognizes this by way of a dummy command line argument,--multiprocessing-fork.This PR:
win32check that preventsfreeze_supportfrom having an effect on Unix--multiprocessing-forkserverand--multiprocessing-semaphore-trackercommand line arguments, which are only used with "frozen" executablesfreeze_supportsuch that it captures and parses the command line arguments for the fork server and semaphore tracker, and launches them.This fixes the problem for me on both PyInstaller and cx_Freeze on Unix.
I couldn't find that there were any tests for the existing
freeze_supportcode. I added some basic ones for the command line manipulation that I've added here, but it may be useful to file an issue to add others.I will add a few more comments in-line. I'm a new-ish contributor, so if discussion of the approach to fixing this issue should take place elsewhere (in the bug tracker or on a mailing list), please let me know (gently). Many thanks!
https://bugs.python.org/issue32146