Description
Matlab is shipped with MKL shared lib with 64-bit integers (at least on Linux, x86_64), PyPI's NumPy is shipped with OpenBLAS dynamic lib with 32-bit integers. Both of them feature same symbols, e.g., dgemmv, so they can't be loaded at the same time. Calling Matlab's BLAS from NumPy and vice versa inevitably leads to crash (segfault or fatal error caught by MKL typechecking).
One should be able to build NumPy from source to avoid this clash. But it would be very useful if NumPy was distributed (on PyPI) such that this is not an issue. There are at least two options:
-
use static linking to link to OpenBLAS and GFortran. AFAIK, this should be possible with
SEARCH_STATIC_FIRST=1, seesite.cfg.example. This should be a general improvement for robustness of linking with NumPy. These shared libs are used only from internally and do not need to be visible from outside. -
use the new BLAS suffix feature of ENH: allow using symbol-suffixed 64-bit BLAS/LAPACK for numpy.dot and linalg #15012 to distribute NumPy with OpenBLAS symbols renamed. This has been already suggested. I am afraid that this might not be enough, as it would prevent incompatible BLAS clash, but not so much for GFortran.
Are there build recipes for the wheels?
Reproducing code example:
# Segfauls (NumPy calls Matlab's MKL with 64-bit ints)
matlab -batch 'A = eye(2); py.numpy.linalg.inv(A)'
# LD_PRELOAD NumPy's BLAS and GFortran
NP_LIBS_DIR=$(python3 -c "import numpy, os; print(os.path.abspath(os.path.join(os.path.dirname(numpy.__file__), '.libs')))")
export LD_PRELOAD=$(find $NP_LIBS_DIR -type f -print0 | tr '\0' ':')
echo LD_PRELOAD=$LD_PRELOAD
# Works fine
matlab -batch 'A = eye(2); py.numpy.linalg.inv(A)'
# Segfauls (Matlab calls NumPy's BLAS with 32-bit ints)
matlab -batch 'polyfit(1:10, sin(1:10), 2)'Error message:
MATLAB is selecting SOFTWARE OPENGL rendering.
--------------------------------------------------------------------------------
Segmentation violation detected at Wed Dec 04 15:13:25 2019 +0100
--------------------------------------------------------------------------------
[...]
LD_PRELOAD=/home/jan/.local/lib/python3.6/site-packages/numpy/.libs/libgfortran-ed201abd.so.3.0.0:/home/jan/.local/lib/python3.6/site-packages/numpy/.libs/libopenblasp-r0-34a18dc3.3.7.so:
MATLAB is selecting SOFTWARE OPENGL rendering.
ans =
Python ndarray:
1 0
0 1
Use details function to view the properties of the Python object.
Use double function to convert to a MATLAB array.
MATLAB is selecting SOFTWARE OPENGL rendering.
--------------------------------------------------------------------------------
Segmentation violation detected at Wed Dec 04 15:13:45 2019 +0100
--------------------------------------------------------------------------------
Numpy/Python version information:
Python 3.6.9 (default, Nov 7 2019, 10:44:02)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, numpy; print(numpy.__version__, sys.version)
1.17.4 3.6.9 (default, Nov 7 2019, 10:44:02)
[GCC 8.3.0]
Matlab version information:
R2019b Update 1 (9.7.0.1216025) 64-bit (glnxa64)
September 26, 2019