gh-132775: Fix Interpreter.call() main Visibility #135595

ericsnowcurrently · 2025-06-16T23:37:49Z

As noted in the new tests, there are a few situations we must carefully accommodate
for functions that get pickled during interp.call(). We do so by running the script
from the main interpreter's __main__ module in a hidden module in the other
interpreter. That hidden module is used as the function __globals__.

Issue: Improvements to test.support.interpreters.Interpreter #132775

bedevere-bot · 2025-06-16T23:51:09Z

🤖 New build scheduled with the buildbot fleet by @ericsnowcurrently for commit 8be5bc2 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F135595%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

neonene · 2025-06-17T12:34:16Z

Python/crossinterp.c

-finally:
-    Py_DECREF(msgobj);
-    return res;
+    assert(PyImport_GetModule(&_Py_ID(__main__)) == main->cached.module);


The refleaks in test_crossinterp.py come from PyImport_GetModule() in the assertion.

neonene · 2025-06-17T12:34:22Z

Python/crossinterp.c

+    assert(main->cached.module != NULL);
+    assert(main->cached.loaded != NULL);
+    PyObject *exc = _PyErr_GetRaisedException(tstate);
+    assert(PyImport_GetModule(&_Py_ID(__main__)) == main->cached.loaded);


neonene · 2025-06-17T12:34:55Z

Python/crossinterp.c

-        }
+finally:
+    if (exc != NULL) {
+        ctx->main.cached.failed = _PyErr_GetRaisedException(tstate);


ctx->main.cached.failed can be set by *_isolated_main() above via sync_module_capture_exc(), in which case test_func_in___main___invalid fails with refleaks.

neonene · 2025-06-17T14:23:52Z

Python/crossinterp.c

+    if (_Py_CheckMainModule(mod) < 0) {
+        // This is probably unrecoverable, so don't bother caching the error.
+        assert(_PyErr_Occurred(tstate));
+        return -1;


I agree that Py_XDECREF(mod) does not need to follow all _Py_CheckMainModule() in this file on error, unlike in ceval.c.

This is a little off topic and maybe unlikely, but I'd prefer _Py_CheckMainModule() to decref the error message, as the function can be used without _Py_GetMainModule():

cpython/Python/pystate.c

Lines 1142 to 1145 in 8dd8b5c

PyObject *msg = PyUnicode_FromString("invalid __main__ module");

if (msg != NULL) {

(void)PyErr_SetImportError(msg, &_Py_ID(__main__), NULL);

}

ericsnowcurrently · 2025-06-17T17:28:44Z

Once again, @neonene, thanks for taking the time to track down these subtle details!

bedevere-bot · 2025-06-17T17:50:03Z

🤖 New build scheduled with the buildbot fleet by @ericsnowcurrently for commit 364d81c 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F135595%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

ericsnowcurrently · 2025-06-17T18:49:02Z

!buildbot AMD64 CentOS9 NoGIL Refleaks

bedevere-bot · 2025-06-17T18:49:05Z

🤖 New build scheduled with the buildbot fleet by @ericsnowcurrently for commit 364d81c 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F135595%2Fmerge

The command will test the builders whose names match following regular expression: AMD64 CentOS9 NoGIL Refleaks

The builders matched are:

AMD64 CentOS9 NoGIL Refleaks PR

miss-islington-app · 2025-06-17T19:17:02Z

Thanks @ericsnowcurrently for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

…-135595) As noted in the new tests, there are a few situations we must carefully accommodate for functions that get pickled during interp.call(). We do so by running the script from the main interpreter's __main__ module in a hidden module in the other interpreter. That hidden module is used as the function __globals__. (cherry picked from commit 269e19e) Co-authored-by: Eric Snow <[email protected]>

bedevere-app · 2025-06-17T19:17:12Z

GH-135638 is a backport of this pull request to the 3.14 branch.

As noted in the new tests, there are a few situations we must carefully accommodate for functions that get pickled during interp.call(). We do so by running the script from the main interpreter's __main__ module in a hidden module in the other interpreter. That hidden module is used as the function __globals__. (cherry picked from commit 269e19e, AKA gh-135595) Co-authored-by: Eric Snow <[email protected]>

…-135595) As noted in the new tests, there are a few situations we must carefully accommodate for functions that get pickled during interp.call(). We do so by running the script from the main interpreter's __main__ module in a hidden module in the other interpreter. That hidden module is used as the function __globals__.

ericsnowcurrently added 3 commits June 16, 2025 17:36

Add tests.

9c06ffc

Swap out __main__ instead of injecting into it.

3cab144

Keep a per-interprter cache.

8be5bc2

ericsnowcurrently added skip news needs backport to 3.14 bugs and security fixes labels Jun 16, 2025

bedevere-app bot mentioned this pull request Jun 16, 2025

Improvements to test.support.interpreters.Interpreter #132775

Closed

ericsnowcurrently marked this pull request as ready for review June 16, 2025 23:46

bedevere-app bot added the awaiting core review label Jun 16, 2025

ericsnowcurrently added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Jun 16, 2025

bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Jun 16, 2025

neonene reviewed Jun 17, 2025

View reviewed changes

ericsnowcurrently added 4 commits June 17, 2025 11:10

Fix a refleak.

a3a8d05

Fix a refleak in an error case.

676507c

Add missing decrefs.

27ffba6

Add a missing decref in _Py_CheckMainModule().

364d81c

ericsnowcurrently requested a review from ZeroIntensity as a code owner June 17, 2025 17:27

ericsnowcurrently added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Jun 17, 2025

bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Jun 17, 2025

ericsnowcurrently merged commit 269e19e into python:main Jun 17, 2025
52 checks passed

bedevere-app bot removed the awaiting core review label Jun 17, 2025

ericsnowcurrently deleted the fix-interp-call-main-visibility branch June 17, 2025 19:17

bedevere-app bot removed the needs backport to 3.14 bugs and security fixes label Jun 17, 2025

	PyObject *msg = PyUnicode_FromString("invalid __main__ module");
	if (msg != NULL) {
	(void)PyErr_SetImportError(msg, &_Py_ID(__main__), NULL);
	}

Uh oh!

gh-132775: Fix Interpreter.call() __main__ Visibility #135595

gh-132775: Fix Interpreter.call() __main__ Visibility #135595

Uh oh!

Conversation

ericsnowcurrently commented Jun 16, 2025 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-bot commented Jun 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ericsnowcurrently commented Jun 17, 2025

Uh oh!

bedevere-bot commented Jun 17, 2025

Uh oh!

ericsnowcurrently commented Jun 17, 2025

Uh oh!

bedevere-bot commented Jun 17, 2025

Uh oh!

Uh oh!

miss-islington-app bot commented Jun 17, 2025

Uh oh!

bedevere-app bot commented Jun 17, 2025

Uh oh!

Uh oh!

gh-132775: Fix Interpreter.call() main Visibility #135595

gh-132775: Fix Interpreter.call() main Visibility #135595

ericsnowcurrently commented Jun 16, 2025 •

edited by bedevere-app bot

Loading