Description
Bug report
Bug description:
Note: I'm using 3.11.6 to showcase the behavior because that's easiest, but the build problems exist in 3.12+
When cross compiling Python, typically the foreign build is targeting a different architecture, but this is not always the case.
It's possible that an x86-64 host may be building a Python for a "foreign" x86-64 machine. This typically means that there may be some difference in libc version or CPU instruction support.
When performing a cross compile for the same architecture (by this I mean the combination of Python version + SOABI), Python 3.12+ will attempt to load foreign libraries as part of some of the target dependencies for the build_all make target and will potentially fail.
When cross compiling, builds specify --with-build-python during configure which specifies a host-safe version of python to use to perform python based tasks on behalf of the foreign python build. When configured, PYTHON_FOR_BUILD will be set to a rather complex command that generally evaluates to something like:
_PYTHON_PROJECT_BASE=/mnt/development/buildroot/output/build/python3-3.11.6 _PYTHON_HOST_PLATFORM=linux-x86_64 PYTHONPATH=/mnt/development/buildroot/output/build/python3-3.11.6/build/lib.linux-x86_64-3.11/:./Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata__linux_x86_64-linux-gnu /mnt/development/buildroot/output/host/bin/python3
This is currently a problem for checksharedmods:
# dependency on BUILDPYTHON ensures that the target is run last
.PHONY: checksharedmods
checksharedmods: sharedmods $(PYTHON_FOR_BUILD_DEPS) $(BUILDPYTHON)
@$(RUNSHARED) $(PYTHON_FOR_BUILD) $(srcdir)/Tools/build/check_extension_modules.py
When run, it tries to run check_extension_modules via PYTHON_FOR_BUILD, however this script has nested in its import dependencies a dependency on externally built modules which poses a problem.
vfazio@vfazio4 ~/development/buildroot/output/build/python3-3.11.6 $ _PYTHON_PROJECT_BASE=/mnt/development/buildroot/output/build/python3-3.11.6 _PYTHON_HOST_PLATFORM=linux-x86_64 PYTHONPATH=/mnt/development/buildroot/output/build/python3-3.11.6/build/lib.linux-x86_64-3.11/:./Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata__linux_x86_64-linux-uclibc /mnt/development/buildroot/output/host/bin/python3 -c "import sys; print(sys.version); import math"
3.11.6 (main, Feb 12 2024, 09:05:53) [GCC 11.4.0]
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: libc.so.0: cannot open shared object file: No such file or directory
Note, this was introduced as part of 3.12:
This also can show up with glibc like so
When the PYTHON_FOR_BUILD command is updated to not include the local build directory in PYTHONPATH, everything works fine.
This makes sense because it will use the host's libraries instead of the target's libraries... Inserting that path into PYTHONPATH alters the search order for extensions, and because the python versions match (this is a requirement for cross builds) and SOABI matches (which is due to the triplets being the same), it will try to use the libraries from the target path:
>>> sys.path
['', '/mnt/development/buildroot/output/build/python3-3.11.6/build/lib.linux-x86_64-3.11', '/mnt/development/buildroot/output/build/python3-3.11.6/Lib', '/mnt/development/buildroot/output/host/lib/python311.zip', '/mnt/development/buildroot/output/host/lib/python3.11', '/mnt/development/buildroot/output/host/lib/python3.11/lib-dynload', '/home/vfazio/.local/lib/python3.11/site-packages', '/mnt/development/buildroot/output/host/lib/python3.11/site-packages']
>>> sys.meta_path
[<_distutils_hack.DistutilsMetaFinder object at 0x7f031311cf50>, <class '_frozen_importlib.BuiltinImporter'>, <class '_frozen_importlib.FrozenImporter'>, <class '_frozen_importlib_external.PathFinder'>]
>>> sys.meta_path[3]
<class '_frozen_importlib_external.PathFinder'>
>>> sys.meta_path[3].find_spec("math")
ModuleSpec(name='math', loader=<_frozen_importlib_external.ExtensionFileLoader object at 0x7f0313092d50>, origin='/mnt/development/buildroot/output/build/python3-3.11.6/build/lib.linux-x86_64-3.11/math.cpython-311-x86_64-linux-gnu.so')
>>> sys.meta_path[3].find_module("math").load_module()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<frozen importlib._bootstrap_external>", line 605, in _check_name_wrapper
File "<frozen importlib._bootstrap_external>", line 1120, in load_module
File "<frozen importlib._bootstrap_external>", line 945, in load_module
File "<frozen importlib._bootstrap>", line 290, in _load_module_shim
File "<frozen importlib._bootstrap>", line 721, in _load
File "<frozen importlib._bootstrap>", line 676, in _load_unlocked
File "<frozen importlib._bootstrap>", line 573, in module_from_spec
File "<frozen importlib._bootstrap_external>", line 1233, in create_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
ImportError: libc.so.0: cannot open shared object file: No such file or directory
So, the "easy solution" is to remove:
diff --git a/configure.ac b/configure.ac
index 384718db1f..6b01083336 100644
--- a/configure.ac
+++ b/configure.ac
@@ -164,7 +164,7 @@ AC_ARG_WITH([build-python],
dnl Build Python interpreter is used for regeneration and freezing.
ac_cv_prog_PYTHON_FOR_REGEN=$with_build_python
PYTHON_FOR_FREEZE="$with_build_python"
- PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) '$with_build_python
+ PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) '$with_build_python
AC_MSG_RESULT([$with_build_python])
], [
AS_VAR_IF([cross_compiling], [yes],However, I assume that was added for a reason and is maybe necessary for other steps (though in testing, it didn't appear to be necessary for the compile to complete).
I'm not sure it ever makes sense to load external modules from the foreign target python when cross compiling, This only works currently because for foreign architecture builds, the triplet will mismatch and cause it to fall back to the host's versions of the libraries. It'd only be safe to do this if the modules were completely source based.
If PYTHONPATH has to be set, then there's maybe an argument that python version and SOABI do not provide enough differentiation and I'm not sure of what the best solution is there.
Alternatively, the check_extension_modules script could be rewritten to reduce the use of external modules to perform its parsing and temporarily work around this issue, but a similar issue could be reintroduced in a subsequent script without policies on what can and can't be used in scripts called during build.
CPython versions tested on:
3.12
3.11
Operating systems tested on:
Linux