Multiple independent Python interpreters in a C/C++ program?

**Christian Heimes** · Jun 27 '08, 04:17 PM

Re: Multiple independent Python interpreters in a C/C++ program?

Diez B. Roggisch schrieb:

AFAIK there was a thread a few month ago that stated that this is
actually possible - but mostly 3rd-party-c-extension aren't capable of
supporting that. Martin von Loewis had a word in that, maybe googling
with that in mind reveals the discussion.
>
And of course its a *bad* idea to pass objects between threads...

PEP 3121 – Extension Module Initialization and Finalization | peps.python.org

http://www.python.org/dev/peps/pep-3121/

Extension module initialization currently has a few deficiencies. There is no cleanup for modules, the entry point name might give naming conflicts, the entry functions don’t follow the usual calling convention, and multiple interpreters are not support...

Christian

**Rhamphoryncus** · Jun 27 '08, 04:17 PM

Re: Multiple independent Python interpreters in a C/C++ program?

On Apr 11, 10:24 am, [email protected] wrote:

This question was posed to me today. Given a C/C++ program we can clearly
embed a Python interpreter in it. Is it possible to fire up multiple
interpreters in multiple threads? For example:
>
C++ main
thread 1
Py_Initialize()
thread 2
Py_Initialize()
>
Do I wind up with two completely independent interpreters, one per thread?
I'm thinking this doesn't work (there are bits which aren't thread-safe and
are only protected by the GIL), but wanted to double-check to be sure.

Since they interpreters would never truly be separate, I think it's
best to bite the bullet and use multiple threads within one
interpreter. The only really difference is pure python modules aren't
duplicated, but why would you need that? Surely not for any kind of
security.

**sturlamolden** · Jun 27 '08, 04:17 PM

Re: Multiple independent Python interpreters in a C/C++ program?

On Apr 11, 6:24 pm, [email protected] wrote:

Do I wind up with two completely independent interpreters, one per thread?
I'm thinking this doesn't work (there are bits which aren't thread-safe and
are only protected by the GIL), but wanted to double-check to be sure.

You can create a new interpreter with a call to Py_NewInterpret er.

However, the GIL is a global object for the process. If you have more
than two interpreters in the process, they share the same GIL.

In tcl, each thread has its own interpreter instance and no GIL is
shared. This circumvents most of the problems with a global GIL.

In theory, a GIL private to each interpreter would make Python more
scalable. The current GIL behaves like the BKL in earlier Linux
kernels. However, some third-party software, notably Apache's
mod_python, is claimed to depend on this behaviour.

**sturlamolden** · Jun 27 '08, 04:17 PM

Re: Multiple independent Python interpreters in a C/C++ program?

On Apr 11, 6:24 pm, [email protected] wrote:

Do I wind up with two completely independent interpreters, one per thread?
I'm thinking this doesn't work (there are bits which aren't thread-safe and
are only protected by the GIL), but wanted to double-check to be sure.

You can create a new subinterpreter with a call to Py_NewInterpret er.
You get a nwe interpreter, but not an independent one. The GIL is a
global object for the process. If you have more than one interpreter
in the process, they share the same GIL.

In tcl, each thread has its own interpreter instance and no GIL is
shared. This circumvents most of the problems with a global GIL.

In theory, a GIL private to each (sub)interprete r would make Python
more scalable. The current GIL behaves like the BKL in earlier Linux
kernels. However, some third-party software, notably Apache's
mod_python, is claimed to depend on this behaviour.

**sturlamolden** · Jun 27 '08, 04:17 PM

C API design flaw (was: Re: Multiple independent Pythoninterpret ers in a C/C++ program?)

On Apr 12, 7:05 pm, sturlamolden <sturlamol...@y ahoo.nowrote:

In theory, a GIL private to each (sub)interprete r would make Python
more scalable. The current GIL behaves like the BKL in earlier Linux
kernels. However, some third-party software, notably Apache's
mod_python, is claimed to depend on this behaviour.

I just looked into the reason why ctypes, mod_python, etc. depend on a
shared GIL. Well ... PyGILState_Ensu re() does not take an argument, so
it does not know which interpreter's GIL to acquire. Duh!

The sad fact is, the functions in Python's C API does not take the
interpreter as an argument, so they default to the one that is
currently active (i.e. the one that PyThreadState_G et()->interp points
to). This is unlike Java's JNI, in which all functions take the
"environmen t" (a Java VM instance) as the first argument.

IMHO, I consider this a major design flaw in Python's C API. In a more
well thought API, PyGILState_Ensu re would take the interpreter
returned by Py_NewInterpret er as an argument, and thus know the
interpreter with which to synchronize.

I complained about this before, but the answer I got was that the
simplified GIL API would not work if interpreters has a separate GIL.
Obviously it would not work as long as the PyGILState_Ensu re does not
take any arguments. The lack of a leading VM argument in
PyGILState_Ensu re, and almost every function in Python's C API, is the
heart of the problem.

**Graham Dumpleton** · Jun 27 '08, 04:17 PM

Re: Multiple independent Python interpreters in a C/C++ program?

On Apr 13, 3:05 am, sturlamolden <sturlamol...@y ahoo.nowrote:

On Apr 11, 6:24 pm, [email protected] wrote:
>

Do I wind up with two completely independent interpreters, one per thread?
I'm thinking this doesn't work (there are bits which aren't thread-safe and
are only protected by the GIL), but wanted to double-check to be sure.

>
You can create a new subinterpreter with a call to Py_NewInterpret er.
You get a nwe interpreter, but not an independent one. The GIL is a
global object for the process. If you have more than one interpreter
in the process, they share the same GIL.
>
In tcl, each thread has its own interpreter instance and no GIL is
shared. This circumvents most of the problems with a global GIL.
>
In theory, a GIL private to each (sub)interprete r would make Python
more scalable. The current GIL behaves like the BKL in earlier Linux
kernels. However, some third-party software, notably Apache'smod_pyt hon, is claimed to depend on this behaviour.

I wouldn't use mod_python as a good guide on how to do this as it
doesn't properly use PyGILState_Ensu re() for main interpreter like it
should. If you want an example of how to do this, have a look at code
for mod_wsgi instead. If you want it to work for Python 3.0 as well as
Python 2.X, make sure you look at mod_wsgi source code from subversion
repository trunk as tweak had to be made to source to support Python
3.0. This is because in Python 3.0 it is no longer sufficient to hold
only the GIL when using string/unicode functions, you also need a
proper thread state to be active now.

Do note that although multiple sub interpreters can be made to work,
destroying sub interpreters within the context of a running process,
ie., before the process ends, can be a cause for various problems with
third party C extension modules and thus would advise that once a sub
interpreter has been created, you keep it and use it for the life of
the process.

Graham

**Rhamphoryncus** · Jun 27 '08, 04:17 PM

Re: C API design flaw (was: Re: Multiple independent Pythoninterpret ers in a C/C++ program?)

On Apr 12, 2:02 pm, sturlamolden <sturlamol...@y ahoo.nowrote:

On Apr 12, 7:05 pm, sturlamolden <sturlamol...@y ahoo.nowrote:
>

In theory, a GIL private to each (sub)interprete r would make Python
more scalable. The current GIL behaves like the BKL in earlier Linux
kernels. However, some third-party software, notably Apache's
mod_python, is claimed to depend on this behaviour.

>
I just looked into the reason why ctypes, mod_python, etc. depend on a
shared GIL. Well ... PyGILState_Ensu re() does not take an argument, so
it does not know which interpreter's GIL to acquire. Duh!
>
The sad fact is, the functions in Python's C API does not take the
interpreter as an argument, so they default to the one that is
currently active (i.e. the one that PyThreadState_G et()->interp points
to). This is unlike Java's JNI, in which all functions take the
"environmen t" (a Java VM instance) as the first argument.
>
IMHO, I consider this a major design flaw in Python's C API. In a more
well thought API, PyGILState_Ensu re would take the interpreter
returned by Py_NewInterpret er as an argument, and thus know the
interpreter with which to synchronize.
>
I complained about this before, but the answer I got was that the
simplified GIL API would not work if interpreters has a separate GIL.
Obviously it would not work as long as the PyGILState_Ensu re does not
take any arguments. The lack of a leading VM argument in
PyGILState_Ensu re, and almost every function in Python's C API, is the
heart of the problem.

Alternatively, the multiple interpreter API could be ripped out, and
you could just use separate threads. Do you have a use case that
requires it?

Multiple independent Python interpreters in a C/C++ program?

Multiple independent Python interpreters in a C/C++ program?

Comment

Comment

Comment

Comment

Comment

Comment

Comment