BLD: migrate build backend from setuptools to meson-python - #2886

The bug in meson is already patched on the dev branch, so I'll just use that (through uv) for now.
Next up: reproducing api_gen and defs.pyx + def.pxd

Progress !
I think I'm really close to a working proof of concept, however I'm still struggling a bit with internal dependencies declarations. Right now my build fails with two consecutive errors:

• h5py.h5r fails to build because h5py.defs is not found
• h5py.defs fails to build because of a relative import (which was fine for h5py.h5r ??)

Interestingly, this is the order they show up, and it's supposedly the reason why that first error exists at all; meson shouldn't even attempt to build h5r before defs, but maybe I'm doing it wrong (api_dep).

[6/10] Compiling Cython source /Users/clm/.cache/uv/sdists-v9/.tmpj20Djw/h5py-3.16.0/h5py/h5r.pyx
FAILED: [code=1] h5py.h5r.abi3.so.p/h5py/h5r.pyx.c 
cython -M --fast-fail -3 /Users/clm/.cache/uv/sdists-v9/.tmpj20Djw/h5py-3.16.0/h5py/h5r.pyx -o h5py.h5r.abi3.so.p/h5py/h5r.pyx.c

Error compiling Cython file:
------------------------------------------------------------
...
# Copyright 2008-2013 Andrew Collette and contributors
#
# License:  Standard 3-clause BSD; see "license.txt" for full license terms
#           and contributor agreement.

from .defs cimport *
^
------------------------------------------------------------
/Users/clm/.cache/uv/sdists-v9/.tmpj20Djw/h5py-3.16.0/h5py/_objects.pxd:10:0: 'h5py/defs.pxd' not found
[7/10] Compiling Cython source defs.pyx
FAILED: [code=1] h5py.h5r.abi3.so.p/defs.pyx.c 
cython -M --fast-fail -3 defs.pyx -o h5py.h5r.abi3.so.p/defs.pyx.c

Error compiling Cython file:
------------------------------------------------------------
...
#
# Warning: this file is auto-generated from api_gen.py. DO NOT EDIT!
#

from .api_types_hdf5 cimport *
^
------------------------------------------------------------
defs.pxd:5:0: relative cimport from non-package directory is not allowed

You should also add cython_args: ['-I' + meson.current_source_dir(), '-I' + meson.current_build_dir()] where relevant because all generated files happen out of the source directory, which means cython needs to know where some files are rather than finding it literally next to the compiler and/or the input.

I understand the problem but the proposed solution doesn't seem to change anything visible. I still have the issue that defs.pxd's build isn't even attempted before h5r.pyx's, even though the latter requires the former.

The non relative imports thing may be solved by telling cython that builddir/h5py is a genuine package, through the means of import('fs').copyfile('__init__.py')

I don't think this worked either 😬

Probably what you want in general is to force ordering by adding the output variable of the custom target as an additional source file for the extension module which needs the pxd. This is roughly identical to the case of C generated headers.

A useful experiment is to see if the build succeeds after running ninja -k0 a few times in a row. If it starts passing, then that means things do compile, they are just missing "order dependencies" to ensure parallelism works.

first time trying to use the ninja CLI directly

❯ ninja -f build/build.ninja -k0
[1/8] Generating config with a custom command
FAILED: [code=1] h5config.json template_substitutions.json
/Users/clm/.cache/uv/archive-v0/v7nB4s-JX5JO4g8e/bin/python build_helpers/config.py -o .
Traceback (most recent call last):
  File "/Users/clm/dev/gh/h5py/h5py/build_helpers/config.py", line 11, in <module>
    import numpy as np
ModuleNotFoundError: No module named 'numpy'
ninja: build stopped: cannot make progress due to previous errors.

okay I already don't know how to get pass this (I did activate my venv, and numpy is definitely installed) 😅

Probably what you want in general is to force ordering by adding the output variable of the custom target as an additional source file for the extension module which needs the pxd.

Already tried that, and it doesn't work either. Wouldn't it be redundant with the deps argument anyway ?
I'm wondering if the issue isn't just that I'm not sure where api_tgt' outputs are actually built. They should be added the the h5py subdir, and my assumption was that it'd happen automatically as the custom_target call lives in h5py/meson.build.