PEP 554 states:

To mitigate that impact and accelerate compatibility, we will do the following:

• be clear that extension modules are not required to support use in multiple interpreters
• raise ImportError when an incompatible module is imported in a subinterpreter
• provide resources (e.g. docs) to help maintainers reach compatibility
• reach out to the maintainers of Cython and of the most used extension modules (on PyPI) to get feedback and possibly provide assistance

The PEP also links to Isolating Extensions which has a lot of theory, but does not clearly state how to migrate a large existing c-extension library like NumPy to support subinterpreters. I think we would need to:

• move to HeapTypes
• move all static state into module state
• carefully analyze code for possible shared state.

I am a bit unclear whether subinterpreters share a single GIL, if not we would also have to carefully examine the code for possible race conditions.

This is a lot of work, and may have performance implications. What is your use case for subinterpreters? Do you think you could help with the effort or find funding for this effort?

Relevant mailing list threads and issues:

• Support for Multiple Interpreters (Subinterpreters) in numpy (Aug 2022)
• Dealing with static local variables in Numpy (Aug 2023)
• Issues labelled with Embedded

This is a lot of work, and may have performance implications. What is your use case for subinterpreters? Do you think you could help with the effort or find funding for this effort?

I suspect the vast majority of changes to be relatively easy, but there is still the same problem that we need someone to explicitly dedicate time on this, and I doubt it will be one of the current core devs.
We even added a warning a long time back saying exactly that, but it seems CPython changes to make subinterpreter support better in the long-run now enforces an error rather than a warning.

PEP 554 states: …

FWIW the recent CPython changes should be from PEP 684 “Per-Interpreter GIL”; PEP 554, for the Python API and subinterpreter management features, is still in draft status.

There's a very strong use case for subinterpreters since PEP 684 for parallel processing that I expect would be useful to a lot of numpy client code: using subinterpreters in separate threads will enable shared memory (at least in a read-only case) with significantly less hassle than multiprocessing.

I’m also very excited about the potential opportunities of using subinterpreters with numpy, and agree with what @mdekstrand said. In particular, the latest draft of PEP 734 discusses sharing data via the buffer protocol (and already implemented in the private interpreters module since 3.13a1). Since ndarrays can export their buffer or be created from one without copies, this could be a very nice pattern:

• pickle your array with protocol 5 to get some serialized metadata plus the memoryview,
• pass that view to a bunch of interpreters (which is basically instant) as well as the small metadata,
• and unpickle: now all of them are sharing the data in each of their arrays
• And if you don’t want to worry about data races, seems like np can handle that by setting the readonly flag.

You get concurrency with performant, opt-in data sharing, without the hassles of managing subprocesses and using multiprocessing.shared_memory where you have to create a shared buffer of fixed size ahead of time and only create arrays using that. With interpreters you can take any random array you got and easily share it.

InterpreterPoolExecutor's are to be introduced in 3.14 124548.

At present, numpy imports fail due to "ImportError: module numpy._core._multiarray_umath does not support loading in subinterpreters":

See following example: TPE and PPE work, IPE does not.

from concurrent.futures import ThreadPoolExecutor
from concurrent.futures import ProcessPoolExecutor
from interpreters_backport.concurrent.futures.interpreter import InterpreterPoolExecutor

def try_tpe():
    with ThreadPoolExecutor() as executor:
        executor.submit(exec, "import numpy as np;print('TPE:', np.random.rand(2))")

def try_ppe():
    with ProcessPoolExecutor() as executor:
        executor.submit(exec, "import numpy as np;print('PPE:', np.random.rand(2))")

def try_ipe():
    with InterpreterPoolExecutor() as executor:
        f = executor.submit(exec, "import numpy as np;print('IPE:', np.random.rand(2))")
        f.result()

if __name__ == "__main__":
    try_tpe()
    try_ppe()
    try_ipe()

Footnote: I noted this recent comment, which perhaps closed the door on hope here; #27192 (comment)

It's a sad thing that even the possibilities are closed. Now Python is heading towards true multithreaded parallelism with InterpreterPoolExecutor and Numpy will not be able to deal with new demands..

It's a sad thing that even the possibilities are closed. Now Python is heading towards true multithreaded parallelism with InterpreterPoolExecutor and Numpy will not be able to deal with new demands..

They aren't closed. A lot of work has happened (and is still happening) to supported free-threaded CPython over the past 6 months. Pretty much all that work is also directly relevant to support for subinterpreters. The extra work needed for subinterpreters - primarily moving to heap types I believe - will need someone to dig in and do that work though. None of the active maintainers are working on this, however contributions are very much welcome.

The list of tasks is still this, although the documentation has gotten better:

• move all static types to HeapTypes (i.e. search for static PyTypeObject and replace that with PyType_FromModuleAndSpec() or some other heap allocation of the type. This can be done one type at a time, and must be benchmarked for performance implications.
• move all static state into module state. Much of the groundwork for this has been done in the free-threading code fixes. Again, better to go in small increments and to benchmark for performance implications.
• carefully analyze code for other possible shared state.

@mattip I imagine cython support for subinterpreters would also be required? cython/cython#6445

Yes, that is part of the "carefully analyze code for other possible shared state". Cython does support the constructs needed for generating code compatible with subinterpreters, the question is more "are we using unsafe coding practices in our cython code?"

Another area that will need adaptation is the f2py generated wrappers.