BUG: _npy_cabs is no longer exported in 2.1.0 #27270

bashtage · 2024-08-22T22:20:23Z

Describe the issue:

The symbol for npy_cabs is no longer exported. This causes certain modules in statamodels to crash on import due to missing symbol in the .so (confirmed on OSX).

Reproduce the code example:

Install SciPy 1.14.1 and NumPy 2.1.0 and then import statamodels.tsa.api.

Error message:

Traceback (most recent call last):
  File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/test.py", line 1, in <module>
    import statsmodels.api as sm
  File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/api.py", line 136, in <module>
    from .regression.recursive_ls import RecursiveLS
  File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/regression/recursive_ls.py", line 14, in <module>
    from statsmodels.tsa.statespace.mlemodel import (
  File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/tsa/statespace/mlemodel.py", line 33, in <module>
    from .simulation_smoother import SimulationSmoother
  File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/tsa/statespace/simulation_smoother.py", line 11, in <module>
    from .kalman_smoother import KalmanSmoother
  File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/tsa/statespace/kalman_smoother.py", line 11, in <module>
    from statsmodels.tsa.statespace.representation import OptionWrapper
  File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/tsa/statespace/representation.py", line 10, in <module>
    from .tools import (
  File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/tsa/statespace/tools.py", line 14, in <module>
    from . import (_initialization, _representation, _kalman_filter,
  File "statsmodels/tsa/statespace/_initialization.pyx", line 1, in init statsmodels.tsa.statespace._initialization
ImportError: dlopen(/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/tsa/statespace/_representation.cpython-312-darwin.so, 0x0002): symbol not found in flat namespace '_npy_cabs'

Python and NumPy Versions:

NumPy 2.1.0

Runtime Environment:

No response

Context for the issue:

I am a maintainer of statamodels. Reported in statsmodels/statsmodels#9333

See also scipy/scipy#21434

SciPy also used to export this symbol. I suspect it was picking up what NumPy exported and so when built against 2.0.x it would export npy_cabs. When built against NumPy 2.1.x, SciPy 1.14.1 does not.

andyfaff · 2024-08-22T23:29:52Z

npy_cabs is used in scipy.special, including it from numpy/npy_math.h. At build time scipy can find and use that symbol. Does statsmodels not include the npy_math header when building?

andyfaff · 2024-08-22T23:33:55Z

~~If I do nm --export-symbols _multiarray_umath.cpython-312-darwin.so (from .../miniconda3/envs/dev3/lib/python3.12/site-packages/numpy/core) then _npy_cabs is present.~~

EDIT: I was looking at the wrong numpy version.

nickodell · 2024-08-23T03:28:58Z

If I do nm --export-symbols _multiarray_umath.cpython-312-darwin.so (from .../miniconda3/envs/dev3/lib/python3.12/site-packages/numpy/core) then _npy_cabs is present.

Odd. For me on pip on NumPy 2.1.0, that command does not show those symbols:

$ nm --export-symbols venv/lib/python3.12/site-packages/numpy/_core/_multiarray_umath.cpython-312-darwin.so         1 ↵
_PyInit__multiarray_umath
__ZN2np7highway10qsort_simd11QSort_ASIMDIdEEvPT_l
__ZN2np7highway10qsort_simd11QSort_ASIMDIfEEvPT_l
__ZN2np7highway10qsort_simd11QSort_ASIMDIiEEvPT_l
__ZN2np7highway10qsort_simd11QSort_ASIMDIjEEvPT_l
__ZN2np7highway10qsort_simd11QSort_ASIMDIxEEvPT_l
__ZN2np7highway10qsort_simd11QSort_ASIMDIyEEvPT_l
__ZN2np7highway10qsort_simd13QSort_ASIMDHPINS_4HalfEEEvPT_l
__ZN2np7highway10qsort_simd13QSort_ASIMDHPIsEEvPT_l
__ZN2np7highway10qsort_simd13QSort_ASIMDHPItEEvPT_l
__ZN3hwy18N_NEON_WITHOUT_AES6detail10Sort16RowsILm4ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingINS_9float16_tEEEEEEES6_EEvT0_PT1_mSC_
__ZN3hwy18N_NEON_WITHOUT_AES6detail10Sort16RowsILm4ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIsEEEEEEsEEvT0_PT1_mSB_
__ZN3hwy18N_NEON_WITHOUT_AES6detail10Sort16RowsILm4ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingItEEEEEEtEEvT0_PT1_mSB_
__ZN3hwy18N_NEON_WITHOUT_AES6detail10Sort16RowsILm8ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingINS_9float16_tEEEEEEES6_EEvT0_PT1_mSC_
__ZN3hwy18N_NEON_WITHOUT_AES6detail10Sort16RowsILm8ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIsEEEEEEsEEvT0_PT1_mSB_
__ZN3hwy18N_NEON_WITHOUT_AES6detail10Sort16RowsILm8ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingItEEEEEEtEEvT0_PT1_mSB_
__ZN3hwy18N_NEON_WITHOUT_AES6detail8Sort2To2INS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingINS_9float16_tEEEEEEES6_EEvT_PT0_mSC_
__ZN3hwy18N_NEON_WITHOUT_AES6detail8Sort2To2INS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIsEEEEEEsEEvT_PT0_mSB_
__ZN3hwy18N_NEON_WITHOUT_AES6detail8Sort2To2INS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingItEEEEEEtEEvT_PT0_mSB_
__ZN3hwy18N_NEON_WITHOUT_AES6detail8Sort3To4INS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingINS_9float16_tEEEEEEES6_EEvT_PT0_mSC_
__ZN3hwy18N_NEON_WITHOUT_AES6detail8Sort3To4INS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIsEEEEEEsEEvT_PT0_mSB_
__ZN3hwy18N_NEON_WITHOUT_AES6detail8Sort3To4INS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingItEEEEEEtEEvT_PT0_mSB_
__ZN3hwy18N_NEON_WITHOUT_AES6detail9Sort8RowsILm1ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingINS_9float16_tEEEEEEES6_EEvT0_PT1_mSC_
__ZN3hwy18N_NEON_WITHOUT_AES6detail9Sort8RowsILm1ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIsEEEEEEsEEvT0_PT1_mSB_
__ZN3hwy18N_NEON_WITHOUT_AES6detail9Sort8RowsILm1ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingItEEEEEEtEEvT0_PT1_mSB_
__ZN3hwy18N_NEON_WITHOUT_AES6detail9Sort8RowsILm2ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingINS_9float16_tEEEEEEES6_EEvT0_PT1_mSC_
__ZN3hwy18N_NEON_WITHOUT_AES6detail9Sort8RowsILm2ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIsEEEEEEsEEvT0_PT1_mSB_
__ZN3hwy18N_NEON_WITHOUT_AES6detail9Sort8RowsILm2ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingItEEEEEEtEEvT0_PT1_mSB_
__ZN3hwy18N_NEON_WITHOUT_AES6detail9Sort8RowsILm4ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingINS_9float16_tEEEEEEES6_EEvT0_PT1_mSC_
__ZN3hwy18N_NEON_WITHOUT_AES6detail9Sort8RowsILm4ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIsEEEEEEsEEvT0_PT1_mSB_
__ZN3hwy18N_NEON_WITHOUT_AES6detail9Sort8RowsILm4ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingItEEEEEEtEEvT0_PT1_mSB_
__ZN3hwy6N_NEON6detail10Sort16RowsILm4ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIfEEEEEEfEEvT0_PT1_mSB_
__ZN3hwy6N_NEON6detail10Sort16RowsILm4ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIiEEEEEEiEEvT0_PT1_mSB_
__ZN3hwy6N_NEON6detail10Sort16RowsILm4ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIjEEEEEEjEEvT0_PT1_mSB_
__ZN3hwy6N_NEON6detail8Sort2To2INS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIdEEEEEEdEEvT_PT0_mSB_
__ZN3hwy6N_NEON6detail8Sort2To2INS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIfEEEEEEfEEvT_PT0_mSB_
__ZN3hwy6N_NEON6detail8Sort2To2INS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIiEEEEEEiEEvT_PT0_mSB_
__ZN3hwy6N_NEON6detail8Sort2To2INS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIjEEEEEEjEEvT_PT0_mSB_
__ZN3hwy6N_NEON6detail8Sort2To2INS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIxEEEEEExEEvT_PT0_mSB_
__ZN3hwy6N_NEON6detail8Sort2To2INS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIyEEEEEEyEEvT_PT0_mSB_
__ZN3hwy6N_NEON6detail8Sort3To4INS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIdEEEEEEdEEvT_PT0_mSB_
__ZN3hwy6N_NEON6detail8Sort3To4INS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIfEEEEEEfEEvT_PT0_mSB_
__ZN3hwy6N_NEON6detail8Sort3To4INS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIiEEEEEEiEEvT_PT0_mSB_
__ZN3hwy6N_NEON6detail8Sort3To4INS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIjEEEEEEjEEvT_PT0_mSB_
__ZN3hwy6N_NEON6detail8Sort3To4INS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIxEEEEEExEEvT_PT0_mSB_
__ZN3hwy6N_NEON6detail8Sort3To4INS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIyEEEEEEyEEvT_PT0_mSB_
__ZN3hwy6N_NEON6detail9Sort8RowsILm1ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIdEEEEEEdEEvT0_PT1_mSB_
__ZN3hwy6N_NEON6detail9Sort8RowsILm1ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIfEEEEEEfEEvT0_PT1_mSB_
__ZN3hwy6N_NEON6detail9Sort8RowsILm1ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIiEEEEEEiEEvT0_PT1_mSB_
__ZN3hwy6N_NEON6detail9Sort8RowsILm1ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIjEEEEEEjEEvT0_PT1_mSB_
__ZN3hwy6N_NEON6detail9Sort8RowsILm1ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIxEEEEEExEEvT0_PT1_mSB_
__ZN3hwy6N_NEON6detail9Sort8RowsILm1ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIyEEEEEEyEEvT0_PT1_mSB_
__ZN3hwy6N_NEON6detail9Sort8RowsILm2ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIdEEEEEEdEEvT0_PT1_mSB_
__ZN3hwy6N_NEON6detail9Sort8RowsILm2ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIfEEEEEEfEEvT0_PT1_mSB_
__ZN3hwy6N_NEON6detail9Sort8RowsILm2ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIiEEEEEEiEEvT0_PT1_mSB_
__ZN3hwy6N_NEON6detail9Sort8RowsILm2ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIjEEEEEEjEEvT0_PT1_mSB_
__ZN3hwy6N_NEON6detail9Sort8RowsILm2ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIxEEEEEExEEvT0_PT1_mSB_
__ZN3hwy6N_NEON6detail9Sort8RowsILm2ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIyEEEEEEyEEvT0_PT1_mSB_
__ZN3hwy6N_NEON6detail9Sort8RowsILm4ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIfEEEEEEfEEvT0_PT1_mSB_
__ZN3hwy6N_NEON6detail9Sort8RowsILm4ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIiEEEEEEiEEvT0_PT1_mSB_
__ZN3hwy6N_NEON6detail9Sort8RowsILm4ENS1_12SharedTraitsINS1_10TraitsLaneINS1_14OrderAscendingIjEEEEEEjEEvT0_PT1_mSB_
__ZN3hwy6detail17Fill16BytesStaticEPv
__ZZN3hwy18N_NEON_WITHOUT_AES6detail11IdxFromBitsINS_9float16_tELm8EEENS0_6Vec128IT_XT0_EEENS_7SizeTagILm2EEEyE5table
__ZZN3hwy18N_NEON_WITHOUT_AES6detail11IdxFromBitsIsLm8EEENS0_6Vec128IT_XT0_EEENS_7SizeTagILm2EEEyE5table
__ZZN3hwy18N_NEON_WITHOUT_AES6detail11IdxFromBitsItLm8EEENS0_6Vec128IT_XT0_EEENS_7SizeTagILm2EEEyE5table
__ZZN3hwy18N_NEON_WITHOUT_AES6detail14IdxFromNotBitsINS_9float16_tELm8EEENS0_6Vec128IT_XT0_EEENS_7SizeTagILm2EEEyE5table
__ZZN3hwy18N_NEON_WITHOUT_AES6detail14IdxFromNotBitsIsLm8EEENS0_6Vec128IT_XT0_EEENS_7SizeTagILm2EEEyE5table
__ZZN3hwy18N_NEON_WITHOUT_AES6detail14IdxFromNotBitsItLm8EEENS0_6Vec128IT_XT0_EEENS_7SizeTagILm2EEEyE5table
__ZZN3hwy6detail23GetGeneratorStateStaticEvE5state

It does show those symbols in 2.0.1, though.

andyfaff · 2024-08-23T03:41:00Z

I was looking at the wrong numpy version, sorry.

rgommers · 2024-08-23T06:32:41Z

This is almost certainly a consequence of gh-26286, which landed in 2.1.0. And SciPy 1.14.1 building against numpy 2.1.0 explains why scipy.special stopped re-exporting the npy_* symbols from libnpymath.a.

I am still a bit confused about what is going wrong here. My first guess was that statsmodels code was forgetting to call import_array in the relevant Cython code, but that seems to be present:
https://github.com/statsmodels/statsmodels/blob/eef6ad3d5f18862877c691a2fa25a57df70c1aa5/statsmodels/tsa/statespace/_representation.pyx.in#L25-L28

What I think is happening is that statsmodels is relying on libnpymath from more extension modules than it is properly accounting for with this:
https://github.com/statsmodels/statsmodels/blob/eef6ad3d5f18862877c691a2fa25a57df70c1aa5/setup.py#L225

It should be reproducible locally in a statsmodels dev environment with numpy 2.1.0 and scipy 1.14.1 installed.

rgommers · 2024-08-23T06:46:52Z

It looks like statsmodels/tsa/statespace/_representation.pyx.in is using BLAS functionality (through SciPy), and that seems to require linking libnpymath (?) says this condition:

https://github.com/statsmodels/statsmodels/blob/eef6ad3d5f18862877c691a2fa25a57df70c1aa5/setup.py#L317

However, the tsa.statespace extensions are treated differently, with include and library dirs explicitly omitted:
https://github.com/statsmodels/statsmodels/blob/eef6ad3d5f18862877c691a2fa25a57df70c1aa5/setup.py#L331-L344

That looks like a bug to me. I'm away for most of today and on holiday till mid next week, but hopefully that's enough of a pointer to identify the fix.

rgommers · 2024-08-24T18:57:26Z

Quick update: I reproduced the issue and found (by reading the Cython-generated C sources) that the npy_cabs and co are coming from:

from statsmodels.src.math cimport *

statsmodels/src/math.pxd provides npy_cabs as cabs and they're used like that in _representation.pyx.in.

The link lines are correct though it looks like, they include -lnpymath:

Building against numpy 2.1.0:

building 'statsmodels.tsa.statespace._representation' extension
clang -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /Users/rgommers/mambaforge/envs/sm-debug/include -arch arm64 -fPIC -O2 -isystem /Users/rgommers/mambaforge/envs/sm-debug/include -arch arm64 -DCYTHON_TRACE_NOGIL=0 -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -Istatsmodels/src -I/Users/rgommers/mambaforge/envs/sm-debug/lib/python3.12/site-packages/numpy/core/include -I/Users/rgommers/mambaforge/envs/sm-debug/lib/python3.12/site-packages/numpy/_core/include -I/Users/rgommers/mambaforge/envs/sm-debug/lib/python3.12/site-packages/numpy/_core/include -I/Users/rgommers/mambaforge/envs/sm-debug/include/python3.12 -c statsmodels/tsa/statespace/_representation.c -o build/temp.macosx-11.0-arm64-cpython-312/statsmodels/tsa/statespace/_representation.o
clang -bundle -undefined dynamic_lookup -Wl,-rpath,/Users/rgommers/mambaforge/envs/sm-debug/lib -L/Users/rgommers/mambaforge/envs/sm-debug/lib -Wl,-rpath,/Users/rgommers/mambaforge/envs/sm-debug/lib -L/Users/rgommers/mambaforge/envs/sm-debug/lib build/temp.macosx-11.0-arm64-cpython-312/statsmodels/tsa/statespace/_representation.o -L/Users/rgommers/mambaforge/envs/sm-debug/lib/python3.12/site-packages/numpy/_core/include/../lib -lnpymath -o build/lib.macosx-11.0-arm64-cpython-312/statsmodels/tsa/statespace/_representation.cpython-312-darwin.so
ld: warning: duplicate -rpath '/Users/rgommers/mambaforge/envs/sm-debug/lib' ignored
ld: warning: object file (/Users/rgommers/mambaforge/envs/sm-debug/lib/python3.12/site-packages/numpy/_core/lib/libnpymath.a[3](meson-generated_npy_math_complex.c.o)) was built for newer 'macOS' version (14.0) than being linked (11.0)

Building against numpy 2.0.0:

building 'statsmodels.tsa.statespace._representation' extension
clang -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /Users/rgommers/mambaforge/envs/sm-debug/include -arch arm64 -fPIC -O2 -isystem /Users/rgommers/mambaforge/envs/sm-debug/include -arch arm64 -DCYTHON_TRACE_NOGIL=0 -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/Users/rgommers/mambaforge/envs/sm-debug/lib/python3.12/site-packages/numpy/core/include -Istatsmodels/src -I/Users/rgommers/mambaforge/envs/sm-debug/lib/python3.12/site-packages/numpy/_core/include -I/Users/rgommers/mambaforge/envs/sm-debug/lib/python3.12/site-packages/numpy/_core/include -I/Users/rgommers/mambaforge/envs/sm-debug/include/python3.12 -c statsmodels/tsa/statespace/_representation.c -o build/temp.macosx-11.0-arm64-cpython-312/statsmodels/tsa/statespace/_representation.o
clang -bundle -undefined dynamic_lookup -Wl,-rpath,/Users/rgommers/mambaforge/envs/sm-debug/lib -L/Users/rgommers/mambaforge/envs/sm-debug/lib -Wl,-rpath,/Users/rgommers/mambaforge/envs/sm-debug/lib -L/Users/rgommers/mambaforge/envs/sm-debug/lib build/temp.macosx-11.0-arm64-cpython-312/statsmodels/tsa/statespace/_representation.o -L/Users/rgommers/mambaforge/envs/sm-debug/lib/python3.12/site-packages/numpy/_core/include/../lib -lnpymath -o build/lib.macosx-11.0-arm64-cpython-312/statsmodels/tsa/statespace/_representation.cpython-312-darwin.so
ld: warning: duplicate -rpath '/Users/rgommers/mambaforge/envs/sm-debug/lib' ignored
ld: warning: object file (/Users/rgommers/mambaforge/envs/sm-debug/lib/python3.12/site-packages/numpy/_core/lib/libnpymath.a[3](meson-generated_npy_math_complex.c.o)) was built for newer 'macOS' version (14.0) than being linked (11.0)

Not yet sure why -lnpymath isn't enough. Or what the deal is with the macOS version warnings.

rgommers · 2024-08-24T22:59:05Z

Looking at the symbols in the _representation extension module:

From the statsmodels 0.14.2 wheel from PyPI:

% nm ~/mambaforge/envs/sm-debug/lib/python3.12/site-packages/statsmodels/tsa/statespace/_representation.cpython-312-darwin.so | rg npy_cabs
                 U _npy_cabs

Local build of 0.14.2 on macOS arm64 14.x, setuptools 72.2.0 and numpy 2.0.0 (and for completeness I verified that the result is the same when building against 2.0.0rc1):

% nm statsmodels/tsa/statespace/_representation.cpython-312-darwin.so | rg npy_cabs
0000000000092508 T _npy_cabs
00000000000924c4 T _npy_cabsf
000000000009254c T _npy_cabsl

Same local build but with numpy 2.1.0:

nm statsmodels/tsa/statespace/_representation.cpython-312-darwin.so | rg npy_cabs
0000000000092508 t _npy_cabs
00000000000924c4 t _npy_cabsf
000000000009254c t _npy_cabsl

Compare to SciPy's cython_special:

% nm ~/mambaforge/envs/sm-debug/lib/python3.12/site-packages/scipy/special/cython_special.cpython-312-darwin.so | rg npy_cabs
00000000001d5cf0 t _npy_cabs

It doesn't seem easy to explain why the symbols in the statsmodels wheel on PyPI went missing (U in nm output) - CI logs have long since disappeared. Everything seems to work as advertised for me, the change from T (global) to t (local) symbols when linking libnpymath are working as expected.

@bashtage could you build new statsmodels wheels with the release procedure on the 0.14.x branch perhaps, so we have logs to look at and can see if it's reproducible?

bashtage · 2024-08-25T08:18:24Z

What happens if you build with NumPy 2.0 and SciPy 1.13 and then run against NumPy 2.1 and the latest SciPy. I don't have a Mac but have been unable to reproduce using CI runners.

rgommers · 2024-08-25T09:19:46Z

What happens if you build with NumPy 2.0 and SciPy 1.13

I get the global symbols after the build:

% pip install scipy==1.13.0 numpy==2.0.0
% python setup.py build_ext -i
% nm statsmodels/tsa/statespace/_representation.cpython-312-darwin.so | rg npy_cabs
0000000000092508 T _npy_cabs
00000000000924c4 T _npy_cabsf
000000000009254c T _npy_cabsl

and then run against NumPy 2.1 and the latest SciPy.

% pip install scipy==1.13.0 numpy==2.0.0 build
% python -m build -wnx  # no build isolation to ensure we build against the correct versions
% pip install scipy==1.14.1 numpy==2.1.0
% pip install dist/*.whl
% python -c "import statsmodels.api"  # works
% python -c "import statsmodels.tsa.statespace"  # works

All looks good. The only thing I can think of that may have happened is that the statsmodels wheel builds, which used --pre to install from the scientific-python nightly bucket, went wrong and at build time HAS_NUMPY in setup.py was false, making -lnpymath go missing. I don't think it is possible to get undefined libnpymath symbols in your wheels if -lnpymath was present.

I'd suggest either rebuilding the macOS 0.14.2 wheels with a higher build number (see https://snarky.ca/what-to-do-when-you-botch-a-release-on-pypi/), or tagging 0.14.3 and uploading that to PyPI after verifying the wheels are fine.

EwoutH · 2024-09-09T20:07:56Z

We have a similar issue in our CI, see this run for example. Numpy 2.1.1, SciPy 1.14.1 and statsmodels 0.14.2. Also happening only on macOS.

bashtage · 2024-09-09T20:20:22Z

This isn't a NumPy bug. The issue is that when SciPy exported the symbol, statamodels would link to the SciPy one when built. If SciPy doesn't export it, which it stopped in 1.14.1, then statamodels will link against NumPy when built against SciPy 1.14.1. The problem arises when building against 1.13.x and the running on >=1.14.1, but then only on OSX.

bashtage · 2024-09-11T21:38:54Z

I just triggered a rebuild

https://github.com/MacPython/statsmodels-wheels/actions/runs/10819773348/job/30018391314

and the test fails (for some reason CIBW doesn't fail, but if you click in you will see

 ImportError: dlopen(/private/var/folders/py/lcjn3y352g1106vf1rqk521r0000gn/T/cibw-run-vbsb2fhv/cp312-macosx_x86_64/venv-test/lib/python3.12/site-packages/statsmodels/tsa/statespace/_representation.cpython-312-darwin.so, 0x0002): symbol not found in flat namespace '_npy_cabs'

bashtage · 2024-09-12T17:55:17Z

I cannot get OSX to work on any Python now. The key compilation lines are

2024-09-12T17:37:12.6515370Z   gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -g -arch x86_64 -DCYTHON_TRACE_NOGIL=0 -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -Istatsmodels/src -I/private/var/folders/py/lcjn3y352g1106vf1rqk521r0000gn/T/pip-build-env-58x9dpn4/overlay/lib/python3.9/site-packages/numpy/_core/include -I/private/var/folders/py/lcjn3y352g1106vf1rqk521r0000gn/T/cibw-run-2bh0vas7/cp39-macosx_x86_64/build/venv/include -I/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c statsmodels/tsa/statespace/_representation.c -o build/temp.macosx-10.9-x86_64-cpython-39/statsmodels/tsa/statespace/_representation.o
2024-09-12T17:37:19.0748820Z   gcc -bundle -undefined dynamic_lookup -g -arch x86_64 build/temp.macosx-10.9-x86_64-cpython-39/statsmodels/tsa/statespace/_representation.o -L/private/var/folders/py/lcjn3y352g1106vf1rqk521r0000gn/T/pip-build-env-58x9dpn4/overlay/lib/python3.9/site-packages/numpy/_core/include/../lib -lnpymath -o build/lib.macosx-10.9-x86_64-cpython-39/statsmodels/tsa/statespace/_representation.cpython-39-darwin.so
2024-09-12T17:37:19.1095880Z   ld: warning: ignoring file '/private/var/folders/py/lcjn3y352g1106vf1rqk521r0000gn/T/pip-build-env-58x9dpn4/overlay/lib/python3.9/site-packages/numpy/_core/lib/libnpymath.a[5](src_npymath_npy_math.c.o)': found architecture 'arm64', required architecture 'x86_64'
2024-09-12T17:37:19.1101130Z   ld: warning: ignoring file '/private/var/folders/py/lcjn3y352g1106vf1rqk521r0000gn/T/pip-build-env-58x9dpn4/overlay/lib/python3.9/site-packages/numpy/_core/lib/libnpymath.a[4](src_npymath_halffloat.cpp.o)': found architecture 'arm64', required architecture 'x86_64'
2024-09-12T17:37:19.1103670Z   ld: warning: ignoring file '/private/var/folders/py/lcjn3y352g1106vf1rqk521r0000gn/T/pip-build-env-58x9dpn4/overlay/lib/python3.9/site-packages/numpy/_core/lib/libnpymath.a[3](meson-generated_npy_math_complex.c.o)': found architecture 'arm64', required architecture 'x86_64'
2024-09-12T17:37:19.1105830Z   ld: warning: ignoring file '/private/var/folders/py/lcjn3y352g1106vf1rqk521r0000gn/T/pip-build-env-58x9dpn4/overlay/lib/python3.9/site-packages/numpy/_core/lib/libnpymath.a[2](meson-generated_ieee754.c.o)': found architecture 'arm64', required architecture 'x86_64'

Those last 4 seem suspicious to me. Any idea why I would be seeing them?

bashtage · 2024-09-12T18:20:47Z

It looks like CIBW is broken on OSX. It appears to be downloading arm64 wheels for the build, but then downloading x86_64 for the test, all for an x86_64 build.

xref pypa/cibuildwheel#2003

nickodell · 2024-09-12T18:56:43Z

Those last 4 seem suspicious to me. Any idea why I would be seeing them?

I've seen a similar error when migrating my data from a x86 mac to an ARM mac. I had to reinstall an ARM version of GCC, delete all temporary build files, and delete ccache's cache (it doesn't take the computer architecture into account)

bashtage · 2024-09-12T20:00:41Z

I have traced the issue to how CIBW behaves when the runner is arm64. For now it does not work correctly when building x86_64 wheels that need to link to the NumPy math library. It does work correctly you only need headers.

The workaround is to limit the rubber to macos-13 rather than macos-latest which lets wheels for both arches be correctly built.

henryiii · 2024-09-12T20:26:23Z

You are manually trying to link to numpy/_core/lib/libnpymath.a, which is being pulled from the ARM wheel rather than the target platform (because pip downloads native wheels, not cross-compiled ones). ~~Is that public? It seems you have to manually construct the path by going up and then back down from the include path you are given. If you could rebuild it, it would be fine.~~ (Edit: found docs, and yes, it does not support cross-compilation) I also don't understand why the ARM build link to libnpymath.a on Intel would work?

andyfaff · 2024-09-13T01:52:59Z

FWIW I built statsmodels/main just now, and ran the test suite. I have numpy2.1.1 and scipy1.14.1, python3.12.5. The whole test suite ran without crashing, although there were two failures. I am on macOS 14.6.1 with an M3 processor.

I can also import statsmodels.tsa.api without crashing.

Is this time to close this issue from numpy's side?

bashtage · 2024-09-13T06:49:39Z

This is 100% not a NumPy bug.

This issue has to do with how CIBW works when hosted on either x86_64 or arm64. On x86_64, it will emulate arm and so correctly cross-compile when you need to link against the numpy math library. When hosted on arm, it will not emulate and so just cross compiles, and as a result, it attempts to link x86_64 binaries against an arm64 numpy math library. Hence when it gets to running, it can't find the symbol.

I think this issue was hidden by SciPy's export of npy_abs. SciPy correctly compiled and linked against the correct arch library, and then made this symbol available. When SciPy stopped exporting it, there was no npy_cabs to find, and hence the crash.

The solution is to use macos-13 for building x86_64 wheels, and macos-14 (or macos-latest) for arm64.

henryiii · 2024-09-13T20:57:10Z

This is 100% not a NumPy bug.

This is a known NumPy limitation, as you can see from the docs.

On x86_64, it will emulate arm and so correctly cross-compile

a) You can emulate or cross compile. If you are emulating, you are not cross-compiling. You are making a native compile on an emulated system.

b) Apple does not support emulating Apple Silicon from Intel. Only the other way around. So cibuildwheel certainly does not emulate ARM from Intel.

If this works on Intel, it's probably working by accident. If you link to NumPy's static library, you must do a native compile (emulated or physical).

You currently must use macos-13 to compile Intel and macos-14 to compile ARM, anything else is undefined behavior. cibuildwheel may in the future allow emulating Intel from ARM, which would allow macos-14 to compile the Intel wheel too. macos-13 will never be able to do a emulated ARM compile.

henryiii · 2024-09-13T20:59:05Z

And #20880 would allow a solution to this, I think, but looks like it's progressing a bit slowly.

bashtage added the 00 - Bug label Aug 22, 2024

bashtage closed this as completed Sep 13, 2024

Uh oh!

BUG: _npy_cabs is no longer exported in 2.1.0 #27270

BUG: _npy_cabs is no longer exported in 2.1.0 #27270

Comments

bashtage commented Aug 22, 2024 • edited by rgommers Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe the issue:

Reproduce the code example:

Error message:

Python and NumPy Versions:

Runtime Environment:

Context for the issue:

andyfaff commented Aug 22, 2024

Uh oh!

andyfaff commented Aug 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nickodell commented Aug 23, 2024 • edited by rgommers Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andyfaff commented Aug 23, 2024

Uh oh!

rgommers commented Aug 23, 2024

Uh oh!

rgommers commented Aug 23, 2024

Uh oh!

rgommers commented Aug 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rgommers commented Aug 24, 2024

Uh oh!

bashtage commented Aug 25, 2024

Uh oh!

rgommers commented Aug 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

EwoutH commented Sep 9, 2024

Uh oh!

bashtage commented Sep 9, 2024

Uh oh!

bashtage commented Sep 11, 2024

Uh oh!

bashtage commented Sep 12, 2024

Uh oh!

bashtage commented Sep 12, 2024

Uh oh!

nickodell commented Sep 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bashtage commented Sep 12, 2024

Uh oh!

henryiii commented Sep 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andyfaff commented Sep 13, 2024

Uh oh!

bashtage commented Sep 13, 2024

Uh oh!

henryiii commented Sep 13, 2024

Uh oh!

henryiii commented Sep 13, 2024

Uh oh!

bashtage commented Aug 22, 2024 •

edited by rgommers

Loading

andyfaff commented Aug 22, 2024 •

edited

Loading

nickodell commented Aug 23, 2024 •

edited by rgommers

Loading

rgommers commented Aug 24, 2024 •

edited

Loading

rgommers commented Aug 25, 2024 •

edited

Loading

nickodell commented Sep 12, 2024 •

edited

Loading

henryiii commented Sep 12, 2024 •

edited

Loading