BUG: Inplace builds (more?) do not install their `init.pxd` well enough #25135

mgorny · 2023-11-13T12:22:50Z

Describe the issue:

The 1.26.2 introduced a new test failure:

FAILED numpy/core/tests/test_cython.py::test_conv_intp - TypeError: 'float' object cannot be interpreted as an integer

Full log below. I've used maintenance/1.26.x branch as of 4962e22 (which is equivalent to 1.26.2, except for the version number).

Reproduce the code example:

git clone https://github.com/numpy/numpy --recurse-submodules -b maintenance/1.26.x --depth 10
cd numpy
python3.11 -m venv .venv
. .venv/bin/activate
pip install -e . -r test_requirements.txt
cp ./build/cp311/numpy/core/*.h numpy/core/include/numpy/
python -m pytest numpy/core/tests/test_cython.py::test_conv_intp

Error message:

========================================================= test session starts =========================================================
platform linux -- Python 3.11.6, pytest-7.4.0, pluggy-1.3.0
rootdir: /tmp/numpy
configfile: pytest.ini
plugins: hypothesis-6.81.1, xdist-3.4.0, cov-4.1.0
collected 1 item                                                                                                                      

numpy/core/tests/test_cython.py F                                                                                               [100%]

============================================================== FAILURES ===============================================================
___________________________________________________________ test_conv_intp ____________________________________________________________

install_temp = None

    def test_conv_intp(install_temp):
        import checks
    
        class myint:
            def __int__(self):
                return 3
    
        # These conversion passes via `__int__`, not `__index__`:
>       assert checks.conv_intp(3.) == 3

checks     = <module 'checks' from '/tmp/pytest-of-mgorny/pytest-5/test_conv_intp0/build/checks.cpython-311-x86_64-linux-gnu.so'>
install_temp = None
myint      = <class 'numpy.core.tests.test_cython.test_conv_intp.<locals>.myint'>

numpy/core/tests/test_cython.py:134: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   TypeError: 'float' object cannot be interpreted as an integer


checks.pyx:34: TypeError
-------------------------------------------------------- Captured stdout setup --------------------------------------------------------
1.2.3
The Meson build system
Version: 1.2.3
Source dir: /tmp/numpy/numpy/core/tests/examples/cython
Build dir: /tmp/pytest-of-mgorny/pytest-5/test_conv_intp0/build
Build type: native build
Project name: checks
Project version: undefined
C compiler for the host machine: ccache cc (gcc 13.2.1 "cc (Gentoo 13.2.1_p20231014 p8) 13.2.1 20231014")
C linker for the host machine: cc ld.bfd 2.41
Cython compiler for the host machine: cython (cython 3.0.5)
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program python3 found: YES (/tmp/numpy/.venv/bin/python3.11)
Found pkg-config: /usr/bin/pkg-config (2.0.3)
Run-time dependency python found: YES 3.11
Build targets in project: 1

Found ninja-1.11.1 at /usr/bin/ninja
[1/3] cython -M --fast-fail -3 /tmp/numpy/numpy/core/tests/examples/cython/checks.pyx -o checks.cpython-311-x86_64-linux-gnu.so.p/checks.pyx.c
[2/3] ccache cc -Ichecks.cpython-311-x86_64-linux-gnu.so.p -I. -I../../../../numpy/numpy/core/tests/examples/cython -I/tmp/numpy/numpy/core/include -I/usr/include/python3.11 -fvisibility=hidden -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -w -O0 -g -fPIC -DNPY_NO_DEPRECATED_API=0 -DNPY_TARGET_VERSION=NPY_2_0_API_VERSION -MD -MQ checks.cpython-311-x86_64-linux-gnu.so.p/meson-generated_checks.pyx.c.o -MF checks.cpython-311-x86_64-linux-gnu.so.p/meson-generated_checks.pyx.c.o.d -o checks.cpython-311-x86_64-linux-gnu.so.p/meson-generated_checks.pyx.c.o -c checks.cpython-311-x86_64-linux-gnu.so.p/checks.pyx.c
[3/3] cc  -o checks.cpython-311-x86_64-linux-gnu.so checks.cpython-311-x86_64-linux-gnu.so.p/meson-generated_checks.pyx.c.o -Wl,--as-needed -Wl,--allow-shlib-undefined -shared -fPIC
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /usr/bin/ninja -v
======================================================= short test summary info =======================================================
FAILED numpy/core/tests/test_cython.py::test_conv_intp - TypeError: 'float' object cannot be interpreted as an integer
========================================================== 1 failed in 8.36s ==========================================================

Runtime information:

1.26.3
3.11.6 (main, Oct 26 2023, 14:30:39) [GCC 13.2.1 20231014]
WARNING: `threadpoolctl` not found in system! Install it by `pip install threadpoolctl`. Once installed, try `np.show_runtime` again for more detailed build information
[{'numpy_version': '1.26.3',
  'python': '3.11.6 (main, Oct 26 2023, 14:30:39) [GCC 13.2.1 20231014]',
  'uname': uname_result(system='Linux', node='pomiot', release='6.6.1-gentoo-dist', version='#1 SMP PREEMPT_DYNAMIC Fri Nov 10 16:04:08 -00 2023', machine='x86_64')},
 {'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
                      'found': ['SSSE3',
                                'SSE41',
                                'POPCNT',
                                'SSE42',
                                'AVX',
                                'F16C',
                                'FMA3',
                                'AVX2'],
                      'not_found': ['AVX512F',
                                    'AVX512CD',
                                    'AVX512_KNL',
                                    'AVX512_KNM',
                                    'AVX512_SKX',
                                    'AVX512_CLX',
                                    'AVX512_CNL',
                                    'AVX512_ICL',
                                    'AVX512_SPR']}}]
None

Context for the issue:

No response

The text was updated successfully, but these errors were encountered:

seberg · 2023-11-13T13:13:50Z

Beats me, somehow it looks like Cython doesn't find our __init__.pyd file for the inplace builds setup here. I have no idea how it manages to find an older one.

mattip · 2023-11-13T13:47:51Z

The mechanism used by Cython is to

look in the user-defined include paths
look along sys.path
look in Cython/Include.

Maybe somehow the inplace build is not putting the __init__.pxd in a numpy that can be found via sys.path? In the worst case, you can add a user-defined include.

seberg · 2023-11-13T14:27:33Z

Yeah, you are right, the meson setup for the editable path just breaks this. It effecitvely adds NumPy as the numpy-editable.pth file. So unless Cython would explicitly import NumPy, it has no chance of finding the correct file in the editable build.

mgorny · 2023-11-13T16:47:18Z

I don't think this is specific to that. I've provided the pip install -e . as an easy reproducer but I have seen the same issue in the Gentoo build where we use a venv with --system-site-packages.

seberg · 2023-11-13T17:27:49Z

Our tests pass, normal tests pass all with cython 3. Yes I am sure it is specific to picking up the wrong __init__.pyd

mattip · 2023-11-13T18:21:10Z

The correct __init__.pyd should be found if any([os.path.exists(p + "/numpy/__init__.pxd") for p in sys.path]) returns True. Otherwise something is off.

seberg · 2023-11-13T20:01:51Z

Just to proof my point:

PYTHONPATH=$(pwd) python -m pytest numpy/core/tests/test_cython.py::test_conv_intp

works fine since inserting the current directory fixes things. Part of the problem might be that Cython doesn't realize that the empty path '' has to translate to '' there?

In either case, happy for suggestions on how to band-aid it. Maybe one can xfail the test ideally on some condition related to such a bad environment.

mattip · 2023-11-13T20:10:02Z

One bandaid would be to add a local -I. command line argument to the (edit: cython command in the) test?

This papers over numpygh-25135, it doesn't fix it, since we are finding the right NumPy, Cython should be finding the right pyd files also (whether this is a NumPy or a Cython issue).

mattip · 2023-11-15T05:14:05Z

@mgorny Could you retry after #25141?

mgorny · 2023-11-15T06:37:00Z

@mgorny Could you retry after #25141?

That seems to fix it for me (on top of 1.26.2), both in venv and in the Gentoo ebuild. Thanks!

seberg · 2023-11-15T08:44:51Z

Opened a Cython issue, not that I think it is super high priority, since it seems somehow related to in-place build.

One thing I hadn't thought of before: There are zipped installs, I think. The Cython machinery probably also fails to pick it up there? That might be a bigger issue!

mattip · 2023-11-15T09:02:48Z

We are talking about building a cython c-extension using the numpy c-api (once built the __init__.pxd is not needed at runtime), and requiring a conventional numpy installation for a build setup seems like a reasonable request.

seberg · 2023-11-15T09:13:38Z

Hmmm, not sure if we even find the right headers for a zipped install, also. In which case it would be moot (even if it exists).

So yeah, I am not sure there is anything to improve here really, you may just not be able to ship NumPy that way. (Since users may want to do local cython builds, not just build wheels.)

rgommers · 2023-11-15T12:38:03Z

This is likely hard to fix. Perhaps Cython can special-case it in its discovery and cimport mechanism, but I doubt it. The fundamental issue is that with out-of-place builds, there is a split between the source tree (numpy/.... in the repo) and the build directory which contains built extensions and generated headers/code. This is glued together via Python import hooks to make import numpy work in an editable install. Other tools than the Python import machinery cannot understand such a split layout.

tl;dr if you need to build against numpy, use a regular install. This also implies for development work, e.g. you use a scikit-learn editable install on top of a numpy editable install, that will have problems.

requiring a conventional numpy installation for a build setup seems like a reasonable request.

agreed

mattip · 2023-11-15T12:59:53Z

Maybe we should document this conclusion? At some point I would like to encourage Cython to drop their numpy pxd files.

rgommers · 2023-11-15T13:20:32Z

Maybe we should document this conclusion? At some point I would like to encourage Cython to drop their numpy pxd files.

Yes, I think we should. Let's take that along in all the build docs that need writing/updating. mesonbuild/meson-python#429 is a similar issue.

I think this can be closed now that the workaround in gh-25141 has been merged?

seberg · 2023-11-15T14:22:51Z

Sounds like there isn't somthing we can easily/reasonably do about it, so yeah, lets close it.

This papers over numpygh-25135, it doesn't fix it, since we are finding the right NumPy, Cython should be finding the right pyd files also (whether this is a NumPy or a Cython issue).

mgorny added the 00 - Bug label Nov 13, 2023

charris added this to the 1.26.3 release milestone Nov 13, 2023

seberg changed the title ~~BUG: 1.26.2 test regression: numpy/core/tests/test_cython.py::test_conv_intp - TypeError: 'float' object cannot be interpreted as an integer~~ BUG: Inplace builds (more?) do not install their __init__.pyd well enough Nov 14, 2023

seberg mentioned this issue Nov 14, 2023

TST: Explicitly pass NumPy path to cython during tests (also speed them up) #25141

Merged

seberg mentioned this issue Nov 15, 2023

[BUG] Cython doesn't understand inplace build mechanics when searching for __init__.pyd cython/cython#5820

Open

rgommers changed the title ~~BUG: Inplace builds (more?) do not install their __init__.pyd well enough~~ BUG: Inplace builds (more?) do not install their __init__.pxd well enough Nov 15, 2023

seberg closed this as completed Nov 15, 2023

charris mentioned this issue Nov 20, 2023

TST: Explicitly pass NumPy path to cython during tests (also speed them up) #25203

Merged

Uh oh!

BUG: Inplace builds (more?) do not install their __init__.pxd well enough #25135

BUG: Inplace builds (more?) do not install their __init__.pxd well enough #25135

Comments

mgorny commented Nov 13, 2023

Describe the issue:

Reproduce the code example:

Error message:

Runtime information:

Context for the issue:

seberg commented Nov 13, 2023

Uh oh!

mattip commented Nov 13, 2023

Uh oh!

seberg commented Nov 13, 2023

Uh oh!

mgorny commented Nov 13, 2023

Uh oh!

seberg commented Nov 13, 2023

Uh oh!

mattip commented Nov 13, 2023

Uh oh!

seberg commented Nov 13, 2023

Uh oh!

mattip commented Nov 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattip commented Nov 15, 2023

Uh oh!

mgorny commented Nov 15, 2023

Uh oh!

seberg commented Nov 15, 2023

Uh oh!

mattip commented Nov 15, 2023

Uh oh!

seberg commented Nov 15, 2023

Uh oh!

rgommers commented Nov 15, 2023

Uh oh!

mattip commented Nov 15, 2023

Uh oh!

rgommers commented Nov 15, 2023

Uh oh!

seberg commented Nov 15, 2023

Uh oh!

BUG: Inplace builds (more?) do not install their `init.pxd` well enough #25135

BUG: Inplace builds (more?) do not install their `init.pxd` well enough #25135

mattip commented Nov 13, 2023 •

edited

Loading