Fixes relative to RPATH handling - #788

This should fix RPATH handling for good.

The only thing not included is automatic translation of the $ORIGIN anchor in install_rpath arguments to @loader_path on macOS. It would not be hard to implement, however, it would be a deviation from what Meson implements. I am not sure it is a good idea to implement it.

There is one case in which this may break projects that work now: when libraries or modules require setting an RPATH to dynamically link to a library installed in the Python install path (with something like install_dir: py.get_install_dir() / 'package') and that work now for how because meson-python does not remove build RPATH entries added by Meson and the source layout matches the install layout. This was the case for example for SciPy 1.15, see #724 (comment) and previous discussion. The comments there indicated that, for what SciPy is concerned, it should be fine to break this now. Note that SciPy 1.15 used install_rpath arguments that are implemented here, thus it would be fine, but it uses the $ORIGIN anchor on macOS too, and that does not work. See above. The behavior changes only with Meson 1.9.0 or later (older Meson versions did not export the required metadata), thus projects that pin the Meson version are fine.

@rgommers I think I added test cases for all scenarios we discussed. It would be great if you could test with packages that may be affected and that do not pin the meson-python version to any released version.

I think it is about time that we straighten out our RPATH story, thus tentatively marking this for the next release. However, it still needs a decision regarding the handling of the anchors in install_rpath, see comments above. I'll work on testing downstream projects.

@rgommers I want to add extended tests in CI that test building SciPy (and maybe some other projects later). I had a quick look at how to do it and I didn't find what the preferred way of building SciPy is these days. Should I use pixi, spin, or something else? Or is downloading the source distribution and installing with pip or uv the way to go? The latter seems the easiest but pip does not have a way to install the build dependencies for a package and disabling build isolation is by far the simplest to test a local version of meson-python.

Hmm, are you sure - that's a really heavy build (10-15 min), and it's nontrivial to set up because of the BLAS dependency. Or are you thinking as a one-off or some sort of extended test suite?

We do test meson-python's main branch in SciPy CI, but only on Linux I believe: https://github.com/scipy/scipy/blob/c18ebd1c4871ff1a04c397247815d859eec54de5/.github/workflows/linux.yml#L183

And that doesn't test all RPATH rewriting issues. I think what we probably need for more coverage is a macOS SciPy job against main?

Re building:

• If we'd want it in CI here, then I'd say that a regular Ubuntu job where the system dependencies are installed with apt is probably best. And then a regular pip install scipy --no-binary scipy. There no job that can be copied 1:1, but something like this job
• pixi is by far the easiest for local development I think, if you clone the scipy repo and replace the meson-python dependency in pixi.toml with a path dependency to the local clone of meson-python.

I did plan on getting back to this PR after the release. It'd be great to get this merged soon.

I was thinking to add an extended test suite gated on all the existing CI jobs to complete successfully. However, I didn't expect SciPy to take 15 min to compile, it is a long time since I had to do it. Do you think it is worth having something like this in our CI?

Does ccache not help here? I know that scipy takes a while because it has 1200 compile edges, with quite a bit of C++. On my laptop (znver5, -j16) it takes

$ time FEATURES=ccache ebuild scipy-1.17.1.ebuild clean install
real	4m50.092s
user	37m28.643s
sys		2m10.715s

But on a second run, it completes significantly faster:

real	0m37.002s
user	3m35.543s
sys		0m23.999s

(Needing to compile all of highs despite a system package existing for it, surely is not helping compile times. Compiling that package from source with emerge -1 =sci-mathematics/highs-1.14.0-r3 takes 48 seconds all on its own.)

I was thinking to add an extended test suite gated on all the existing CI jobs to complete successfully. However, I didn't expect SciPy to take 15 min to compile, it is a long time since I had to do it. Do you think it is worth having something like this in our CI?

I'm inclined to say no here. I think there's a lot of value in our CI being simple and very fast. We'd make the completion time way longer, and the setup quite a bit more maintenance-intensive. I'd much rather add more CI jobs in SciPy's CI that use meson-python's main branch on different platforms.

I do like the idea of better integration testing though. I think we can (a) push more of our large users to test our main branch, and (b) have the kind of CI machinery you have in mind here in a separate repo, so it's easier to test at scale. I think https://github.com/rgommers/external-deps-build/ does a lot of what you have in mind here. We can clone that, tweak it a bit so it can be used to test PRs/branches, and populate it with a lot of packages that use meson-python. WDYT?

Does ccache not help here?

Yes it does, and we use it in SciPy - but that comes with its own extra work; two job steps per build, cache management, having to run CI jobs on main to populate the cache for PRs.