Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 5183856

Browse files
benjaminpbenjaminp
committed
Merged revisions 75246 via svnmerge from
svn+ssh://[email protected]/python/branches/py3k ................ r75246 | benjamin.peterson | 2009-10-04 15:32:25 -0500 (Sun, 04 Oct 2009) | 29 lines Merged revisions 74841 via svnmerge from svn+ssh://[email protected]/python/trunk ........ r74841 | thomas.wouters | 2009-09-16 14:55:54 -0500 (Wed, 16 Sep 2009) | 23 lines Fix issue #1590864, multiple threads and fork() can cause deadlocks, by acquiring the import lock around fork() calls. This prevents other threads from having that lock while the fork happens, and is the recommended way of dealing with such issues. There are two other locks we care about, the GIL and the Thread Local Storage lock. The GIL is obviously held when calling Python functions like os.fork(), and the TLS lock is explicitly reallocated instead, while also deleting now-orphaned TLS data. This only fixes calls to os.fork(), not extension modules or embedding programs calling C's fork() directly. Solving that requires a new set of API functions, and possibly a rewrite of the Python/thread_*.c mess. Add a warning explaining the problem to the documentation in the mean time. This also changes behaviour a little on AIX. Before, AIX (but only AIX) was getting the import lock reallocated, seemingly to avoid this very same problem. This is not the right approach, because the import lock is a re-entrant one, and reallocating would do the wrong thing when forking while holding the import lock. Will backport to 2.6, minus the tiny AIX behaviour change. ........ ................
1 parent 9507887 commit 5183856

5 files changed

Lines changed: 108 additions & 22 deletions

File tree

‎Doc/c-api/init.rst‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -520,6 +520,22 @@ supports the creation of additional interpreters (using
520520
:cfunc:`Py_NewInterpreter`), but mixing multiple interpreters and the
521521
:cfunc:`PyGILState_\*` API is unsupported.
522522

523+
Another important thing to note about threads is their behaviour in the face
524+
of the C :cfunc:`fork` call. On most systems with :cfunc:`fork`, after a
525+
process forks only the thread that issued the fork will exist. That also
526+
means any locks held by other threads will never be released. Python solves
527+
this for :func:`os.fork` by acquiring the locks it uses internally before
528+
the fork, and releasing them afterwards. In addition, it resets any
529+
:ref:`lock-objects` in the child. When extending or embedding Python, there
530+
is no way to inform Python of additional (non-Python) locks that need to be
531+
acquired before or reset after a fork. OS facilities such as
532+
:cfunc:`posix_atfork` would need to be used to accomplish the same thing.
533+
Additionally, when extending or embedding Python, calling :cfunc:`fork`
534+
directly rather than through :func:`os.fork` (and returning to or calling
535+
into Python) may result in a deadlock by one of Python's internal locks
536+
being held by a thread that is defunct after the fork.
537+
:cfunc:`PyOS_AfterFork` tries to reset the necessary locks, but is not
538+
always able to.
523539

524540
.. ctype:: PyInterpreterState
525541

‎Include/import.h‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,14 @@ PyAPI_FUNC(PyObject *) PyImport_ReloadModule(PyObject *m);
2727
PyAPI_FUNC(void) PyImport_Cleanup(void);
2828
PyAPI_FUNC(int) PyImport_ImportFrozenModule(char *);
2929

30+
#ifdef WITH_THREAD
31+
PyAPI_FUNC(void) _PyImport_AcquireLock(void);
32+
PyAPI_FUNC(int) _PyImport_ReleaseLock(void);
33+
#else
34+
#define _PyImport_AcquireLock()
35+
#define _PyImport_ReleaseLock() 1
36+
#endif
37+
3038
PyAPI_FUNC(struct filedescr *) _PyImport_FindModule(
3139
const char *, PyObject *, char *, size_t, FILE **, PyObject **);
3240
PyAPI_FUNC(int) _PyImport_IsScript(struct filedescr *);

‎Lib/test/test_fork1.py‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,14 @@
11
"""This test checks for correct fork() behavior.
22
"""
33

4+
import errno
5+
import imp
46
import os
7+
import signal
8+
import sys
59
import time
10+
import threading
11+
612
from test.fork_wait import ForkWait
713
from test.support import run_unittest, reap_children, get_attribute
814

@@ -23,6 +29,41 @@ def wait_impl(self, cpid):
2329
self.assertEqual(spid, cpid)
2430
self.assertEqual(status, 0, "cause = %d, exit = %d" % (status&0xff, status>>8))
2531

32+
def test_import_lock_fork(self):
33+
import_started = threading.Event()
34+
fake_module_name = "fake test module"
35+
partial_module = "partial"
36+
complete_module = "complete"
37+
def importer():
38+
imp.acquire_lock()
39+
sys.modules[fake_module_name] = partial_module
40+
import_started.set()
41+
time.sleep(0.01) # Give the other thread time to try and acquire.
42+
sys.modules[fake_module_name] = complete_module
43+
imp.release_lock()
44+
t = threading.Thread(target=importer)
45+
t.start()
46+
import_started.wait()
47+
pid = os.fork()
48+
try:
49+
if not pid:
50+
m = __import__(fake_module_name)
51+
if m == complete_module:
52+
os._exit(0)
53+
else:
54+
os._exit(1)
55+
else:
56+
t.join()
57+
# Exitcode 1 means the child got a partial module (bad.) No
58+
# exitcode (but a hang, which manifests as 'got pid 0')
59+
# means the child deadlocked (also bad.)
60+
self.wait_impl(pid)
61+
finally:
62+
try:
63+
os.kill(pid, signal.SIGKILL)
64+
except OSError:
65+
pass
66+
2667
def test_main():
2768
run_unittest(ForkTest)
2869
reap_children()

‎Modules/posixmodule.c‎

Lines changed: 31 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3831,11 +3831,21 @@ Return 0 to child process and PID of child to parent process.");
38313831
static PyObject *
38323832
posix_fork1(PyObject *self, PyObject *noargs)
38333833
{
3834-
pid_t pid = fork1();
3834+
pid_t pid;
3835+
int result;
3836+
_PyImport_AcquireLock();
3837+
pid = fork1();
3838+
result = _PyImport_ReleaseLock();
38353839
if (pid == -1)
38363840
return posix_error();
38373841
if (pid == 0)
38383842
PyOS_AfterFork();
3843+
if (result < 0) {
3844+
/* Don't clobber the OSError if the fork failed. */
3845+
PyErr_SetString(PyExc_RuntimeError,
3846+
"not holding the import lock");
3847+
return NULL;
3848+
}
38393849
return PyLong_FromPid(pid);
38403850
}
38413851
#endif
@@ -3850,11 +3860,21 @@ Return 0 to child process and PID of child to parent process.");
38503860
static PyObject *
38513861
posix_fork(PyObject *self, PyObject *noargs)
38523862
{
3853-
pid_t pid = fork();
3863+
pid_t pid;
3864+
int result;
3865+
_PyImport_AcquireLock();
3866+
pid = fork();
3867+
result = _PyImport_ReleaseLock();
38543868
if (pid == -1)
38553869
return posix_error();
38563870
if (pid == 0)
38573871
PyOS_AfterFork();
3872+
if (result < 0) {
3873+
/* Don't clobber the OSError if the fork failed. */
3874+
PyErr_SetString(PyExc_RuntimeError,
3875+
"not holding the import lock");
3876+
return NULL;
3877+
}
38583878
return PyLong_FromPid(pid);
38593879
}
38603880
#endif
@@ -3957,14 +3977,22 @@ To both, return fd of newly opened pseudo-terminal.\n");
39573977
static PyObject *
39583978
posix_forkpty(PyObject *self, PyObject *noargs)
39593979
{
3960-
int master_fd = -1;
3980+
int master_fd = -1, result;
39613981
pid_t pid;
39623982

3983+
_PyImport_AcquireLock();
39633984
pid = forkpty(&master_fd, NULL, NULL, NULL);
3985+
result = _PyImport_ReleaseLock();
39643986
if (pid == -1)
39653987
return posix_error();
39663988
if (pid == 0)
39673989
PyOS_AfterFork();
3990+
if (result < 0) {
3991+
/* Don't clobber the OSError if the fork failed. */
3992+
PyErr_SetString(PyExc_RuntimeError,
3993+
"not holding the import lock");
3994+
return NULL;
3995+
}
39683996
return Py_BuildValue("(Ni)", PyLong_FromPid(pid), master_fd);
39693997
}
39703998
#endif

‎Python/import.c‎

Lines changed: 12 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -261,8 +261,8 @@ static PyThread_type_lock import_lock = 0;
261261
static long import_lock_thread = -1;
262262
static int import_lock_level = 0;
263263

264-
static void
265-
lock_import(void)
264+
void
265+
_PyImport_AcquireLock(void)
266266
{
267267
long me = PyThread_get_thread_ident();
268268
if (me == -1)
@@ -286,8 +286,8 @@ lock_import(void)
286286
import_lock_level = 1;
287287
}
288288

289-
static int
290-
unlock_import(void)
289+
int
290+
_PyImport_ReleaseLock(void)
291291
{
292292
long me = PyThread_get_thread_ident();
293293
if (me == -1 || import_lock == NULL)
@@ -302,23 +302,16 @@ unlock_import(void)
302302
return 1;
303303
}
304304

305-
/* This function is called from PyOS_AfterFork to ensure that newly
306-
created child processes do not share locks with the parent. */
305+
/* This function used to be called from PyOS_AfterFork to ensure that newly
306+
created child processes do not share locks with the parent, but for some
307+
reason only on AIX systems. Instead of re-initializing the lock, we now
308+
acquire the import lock around fork() calls. */
307309

308310
void
309311
_PyImport_ReInitLock(void)
310312
{
311-
#ifdef _AIX
312-
if (import_lock != NULL)
313-
import_lock = PyThread_allocate_lock();
314-
#endif
315313
}
316314

317-
#else
318-
319-
#define lock_import()
320-
#define unlock_import() 0
321-
322315
#endif
323316

324317
static PyObject *
@@ -335,7 +328,7 @@ static PyObject *
335328
imp_acquire_lock(PyObject *self, PyObject *noargs)
336329
{
337330
#ifdef WITH_THREAD
338-
lock_import();
331+
_PyImport_AcquireLock();
339332
#endif
340333
Py_INCREF(Py_None);
341334
return Py_None;
@@ -345,7 +338,7 @@ static PyObject *
345338
imp_release_lock(PyObject *self, PyObject *noargs)
346339
{
347340
#ifdef WITH_THREAD
348-
if (unlock_import() < 0) {
341+
if (_PyImport_ReleaseLock() < 0) {
349342
PyErr_SetString(PyExc_RuntimeError,
350343
"not holding the import lock");
351344
return NULL;
@@ -2200,9 +2193,9 @@ PyImport_ImportModuleLevel(char *name, PyObject *globals, PyObject *locals,
22002193
PyObject *fromlist, int level)
22012194
{
22022195
PyObject *result;
2203-
lock_import();
2196+
_PyImport_AcquireLock();
22042197
result = import_module_level(name, globals, locals, fromlist, level);
2205-
if (unlock_import() < 0) {
2198+
if (_PyImport_ReleaseLock() < 0) {
22062199
Py_XDECREF(result);
22072200
PyErr_SetString(PyExc_RuntimeError,
22082201
"not holding the import lock");

0 commit comments

Comments
 (0)