ENH: start to use OpenBLAS from a wheel #24749

mattip · 2023-09-19T18:48:05Z

Follow on for MacPython/openblas-libs#87 which creates OpenBLAS wheels, available as scipy-openblas32 and scipy-openblas64. The wheel provides facilities for generating a openblas.pc file and for rewriting the _distributor_init.py file to first import the project, which will inject the dll/so into the process namespace and make the interfaces available. In order to use the wheel you must do one of these to build NumPy:

pip install scipy-openblas32 -r build_requirements.txt
spin build --with-scipy-openblas=32

or

pip install scipy-openblas64 -r build_requirements.txt
spin build --with-scipy-openblas=64

You can also use the non-spin pip install route for the 32-bit interfaces:

pip install scipy-openblas32 spin
spin config-openblas --with-scipy-openblas=32
export PKG_CONFIG_PATH=".openblas"
pip install .

I tried to modify most of the builds ~~and wheel builds~~ to use one of these workflows. ~~One problem left to solve is specifying the runtime dependency for the wheels: which scipy-openblas should each platform use?~~

rgommers

Great to see this moving forward. It looks like this PR will remove a lot of cruft.

One problem left to solve is specifying the runtime dependency for the wheels: which scipy-openblas should each platform use?

Probably best to make 1:1 replacements, and not change which of LP64/ILP64 is used in each of the CI jobs.

doc/source/user/building.rst

rgommers · 2023-09-21T10:31:22Z

doc/source/user/building.rst

+There are two variants of the wheel, where possible we reccomend using the
+one with 64-bit integer interfaces, which allow for larger arrays::
+
+    pip install scipy-openblas64 spin


Use of spin does not belong on this page, that should appear only on the pages aimed at contributors.

Hmm. Above the user is directed to use pip install -e . --no-build-isolation. How woul d you suggest they get the needed modifications before running that:

extract a properly configured openblas.pc from the scipy-openblas package

point PKG_CONFIG_PATH to it

modify _distributor_init.py

I guess I could move the functionality I added to spin to do these things into to scipy-openblas, and then this would become

pip install scipy-openblas64 python -m scipy-openblas64 setup-repo="." export PKG_CONFIG_PATH=${PWD}/.openblas # bash $env:PKG_CONFIG_PATH="$pwd/.openblas" # powershell pip install . -v -Csetup-args="--vsenv" -Csetup-args="-Duse-ilp64=true" -Csetup-args="-Dblas-symbol-suffix=64_"

Is that preferable?

Another idea: push all this into numpy/meson.build, when configured with -Dblas=scipy-openblas32 or -Dblas=scipy-openblas64. It could take care of everything: creating the openblas.pc file, adding cflags, setting PKG_CONFIG_PATH, modifying _distributor_init.py.

How would you suggest they get the needed modifications before running that:

They don't, that really isn't all that relevant here. BLAS/LAPACK remain external and generic build time dependencies. The wheel is for us to depend on, not for them to mess with directly.

Another idea: push all this into numpy/meson.build,

I'm not sure it's possible to set PKG_CONFIG_PATH from within Meson.

Either way, there may be an issue here in that the produced wheel will be cached by pip and will contain RPATH entries to a specific Python environment. That is going to break as soon as the user tries to install numpy into another environment.

I have to give it some more thought, but I think this is a little tricky, and perhaps a distraction from the main purpose of this PR right now. We want these wheels used by numpy and scipy wheels and in CI. I'd be inclined to scope this first PR to only that, and not user-facing changes.

Indeed, and we cannot add the scipy-openblasX dependency to the sdist without breaking builds from source on platforms for which we don't have openblas wheels.

We may be able to fix this with a metapackage or a do-nothing scipy-openblasX sdist, but it needs some further hack like that. Or a proper solution, like adding a "pypi provider" concept for external dependencies to https://peps.python.org/pep-0725/.

Or could we combine both 32 bit and 64 into a single wheel, along with dummy do-nothing code?

Ahh, now I get, sorry for being dense. If we add an ad-hoc dependency on, say, the manylinux2014 wheels for scipy-openblas64, and a dependency on, say, the 32-bit windows wheels for scipy-openblas32, and a dependency on, say, no openblas for macosx-arm64, then there is no way to reconcile all that for a single dependency for the sdist.

I guess another path would be to have a numpy[openblas64] wheel variant, but explaining pip install numpy[openblas64] seems like a nightmare.

Or could we combine both 32 bit and 64 into a single wheel, along with dummy do-nothing code?

Hmm. The problem is of course the wheel size: users would be downloading at least one useless large shared object. That is why I made two wheels.

PEP 725

I think the scipy-openblas wheels fall under the rejected category of external dependencies which are also packaged on PyPI

I think the scipy-openblas wheels fall under the rejected category of external dependencies which are also packaged on PyPI

Well I put it in that category because it seemed too niche - the PEP is still draft, so I can easily reverse that decision:)

the manylinux2014 wheels for scipy-openblas64, and a dependency on, say, the 32-bit windows wheels for scipy-openblas32, and a dependency on, say, no openblas for macosx-arm64, then there is no way to reconcile all that for a single dependency for the sdist.

I think we don't need any 32-bit wheel for numpy at least (scipy is a different story), so it's a little easier. We could have platform selectors perhaps, determining whether we want scipy-openblas64 or nothing.

pyproject.toml

rgommers · 2023-09-21T10:40:36Z

azure-steps-windows.yml

    If ( Test-Path env:DISABLE_BLAS ) {
        python -m pip install . -v -Csetup-args="--vsenv" -Csetup-args="-Dblas=none" -Csetup-args="-Dlapack=none" -Csetup-args="-Dallow-noblas=true"
    }
    elseif ( Test-Path env:NPY_USE_BLAS_ILP64 ) {
-        python -m pip install . -v -Csetup-args="--vsenv" -Csetup-args="-Duse-ilp64=true" -Csetup-args="-Dblas-symbol-suffix=64_"
+        python -m pip install scipy-openblas64
+        spin build --with-scipy-openblas=64


Moving pip invocations to spin should be a short-term thing, it's fine with me on Azure because that's so hard to debug. But can you please comment out the original line instead of removing it?

More importantly: I don't think this will work, because you removed the other CLI flags and they are needed. Can you instead do:

python -m pip install scipy-openblas64 export PKG_CONFIG_PATH=... # whatever is the right path here python -m pip install . -v -Csetup-args="--vsenv" -Csetup-args="-Duse-ilp64=true" -Csetup-args="-Dblas-symbol-suffix=64_"

?

spin build will add the meson flags when invoked with --with-scipy-openblas=64 (in addition to creating the openblas.pc file, modifying _distributor_init.py, and adding the proper PKG_CONFIG_PATH).

azure-pipelines.yml

mattip · 2023-09-21T12:38:49Z

.spin/cmds.py

+        ctx.params["meson_args"] += (
+            "-Dblas-symbol-suffix=64_",
+            "-Duse-ilp64=true",
+        )


Here is where spin build adds the meson arguments for 64-bit openblas builds

mattip · 2023-09-26T11:22:38Z

I rebased off main and reduced the scope of the PR to CI build/test runs. We can decide how/if to use the scipy-openblas wheels as a runtime dependency in a future PR.

rgommers · 2023-09-26T12:27:03Z

All looks about right to me, now it's "only" a matter of making CI happy.

mattip · 2023-09-27T15:58:14Z

It is too easy to run tests locally, which modify _distributor_init.py, and then mistakenly commit that changed file.

rgommers · 2023-09-27T16:42:06Z

Yeah it's never a good idea to modify files that are under version control. If you can't avoid a change, maybe change _distributor_init.py to the below?

# It's okay to replace this content:
try:
    # This `_local` file doesn't exist in VCS, so easier to modify for local development
    import ._distributor_init_local
except ImportError:
    pass

mattip · 2023-09-27T18:33:33Z

musllinux requires the ctypes.RTLD_GLOBAL flag, and should not link with libopenblas_python. Both of those changes need changes to the scipy-openblas wheels.

mattip · 2023-09-27T18:34:05Z

azure-pipelines.yml

            python3 -m pip install -r test_requirements.txt && \
            echo CFLAGS \$CFLAGS && \
-            python3 -m pip install -v . && \
+            spin openblas-config --with-scipy-openblas=32 && \


Suggested change

spin openblas-config --with-scipy-openblas=32 && \

spin openblas-config --with-scipy-openblas=32 && \

Suggested change

spin openblas-config --with-scipy-openblas=32 && \

spin config-openblas --with-scipy-openblas=32 && \

mattip · 2023-09-29T08:39:52Z

I will close this PR and start over, with the goal of accepting a blas=scipy-openblas option as suggested in #24827. The scipy-openblas.pc file now adds the needed cflags for ILP64 builds, so no extra command line flags are needed.

PKG_CONFIG_PATH=$PWD/.openblas python -m pip install . \
    --no-build-isolation -Csetup-args=-Dblas=scipy-openblas

An enhancement would be to search basedir/.openblas for the pkgconfig file.

github-actions bot added the 01 - Enhancement label Sep 19, 2023

rgommers reviewed Sep 21, 2023

View reviewed changes

mattip commented Sep 21, 2023

View reviewed changes

mattip force-pushed the openblas-wheel branch from 4f60be0 to b68dd54 Compare September 26, 2023 11:21

ENH: use OpenBLAS from a wheel

46439e5

mattip force-pushed the openblas-wheel branch 4 times, most recently from 948c1f2 to 9d07277 Compare September 27, 2023 15:41

CI fixes

ca639f0

mattip force-pushed the openblas-wheel branch from 9d07277 to ca639f0 Compare September 27, 2023 15:57

mattip commented Sep 27, 2023

View reviewed changes

mattip mentioned this pull request Sep 28, 2023

BUG: meson detection of openblas builds an executable without using pkg-config --static --libs #24827

Closed

mattip closed this Sep 29, 2023

rgommers mentioned this pull request Oct 4, 2023

BLD: use scipy-openblas wheel #24839

Merged

mattip mentioned this pull request Oct 4, 2023

ENH, BLD: use wheels to get BLAS implementations #24857

Open

mattip mentioned this pull request Oct 23, 2024

Shared library paths multi-build/multibuild#365

Closed

	spin openblas-config --with-scipy-openblas=32 && \
	spin openblas-config --with-scipy-openblas=32 && \

	spin openblas-config --with-scipy-openblas=32 && \
	spin config-openblas --with-scipy-openblas=32 && \

Uh oh!

ENH: start to use OpenBLAS from a wheel #24749

ENH: start to use OpenBLAS from a wheel #24749

Uh oh!

Conversation

mattip commented Sep 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rgommers left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattip commented Sep 26, 2023

Uh oh!

rgommers commented Sep 26, 2023

Uh oh!

mattip commented Sep 27, 2023

Uh oh!

rgommers commented Sep 27, 2023

Uh oh!

mattip commented Sep 27, 2023 • edited by rgommers Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattip commented Sep 29, 2023

Uh oh!

Uh oh!

mattip commented Sep 19, 2023 •

edited

Loading

mattip commented Sep 27, 2023 •

edited by rgommers

Loading