Thanks to visit codestin.com
Credit goes to github.com

Skip to content

gh-117378: Fix multiprocessing forkserver preload sys.path inheritance. #126538

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gpshead merged 7 commits into from
Nov 9, 2024
+97 −0
Merged

gh-117378: Fix multiprocessing forkserver preload sys.path inheritance. #126538

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
c40b26d
gh-117378: Fix multiprocessing forkserver preload sys.path inheritance.
gpshead Nov 7, 2024
702e0b7
NEWS entry
gpshead Nov 7, 2024
7ad114f
move sys.path setting up per review.
gpshead Nov 8, 2024
cf07467
Redo the test as an sys.path integration test.
gpshead Nov 9, 2024
4fe3389
avoid := for readability
gpshead Nov 9, 2024
3f486f8
reword NEWS
gpshead Nov 9, 2024
2444519
news sphinx-lint fix
gpshead Nov 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
reword NEWS
  • Loading branch information
@gpshead
gpshead committed Nov 9, 2024
commit 3f486f8c2a949040c8944b9308d4a114d684181e
26 changes: 16 additions & 10 deletions Misc/NEWS.d/next/Library/2024-11-07-01-40-11.gh-issue-117378.o9O5uM.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,17 @@
Fixed the :mod:`multiprocessing` ``"forkserver"`` start method forkserver
process was to correctly inherit the parent's :data:`sys.path` during the
importing of :func:`multiprocessing.set_forkserver_preload` modules in the
same manner as :data:`sys.path` is configured when executing work items in
the worker processes.

This bug could cause some forkserver module preloading to silently fail to
be preloaded, leading to a performance degration in child processes due to
additional repeated work. It could also have led to a side effect of ``""``
still being in :data:`sys.path` during forkserver preload imports instead of
the absolute path of the directory that workers see.
process to correctly inherit the parent's :data:`sys.path` during the importing
of :func:`multiprocessing.set_forkserver_preload` modules in the same manner as
:data:`sys.path` is configured in workers before executing work items.

This bug caused some forkserver module preloading to silently fail to preload.
This manifested as a performance degration in child processes when the
`sys.path` was required due to additional repeated work in every worker.

It could also have a side effect of ``""`` remaining in :data:`sys.path` during
forkserver preload imports instead of the absolute path from :func:`os.getcwd`
at multiprocessing import time used in the worker ``sys.path``.

Potentially leading to incorrect imports from the wrong location during
preload. We are unaware of that actually happening. The issue was discovered
by someone observing unexpected preload performance gains.