BLD: Build scikit-learn nightlies using NumPy nightlies #27735
Conversation
|
The idea here is that in the future you can use Numpy 2.0 to build a wheel that works with Numpy 1.x (1.25? lower? higher?) and that we can simulate this/start preparing for this by today using the Numpy nightly as a stand in for Numpy 2.0? |
The lower bound won't really change (well, we should bump it slowly probably).
More importantly, NumPy can actually start using that fact. Which allows certain ABI breaks, mainly through forcing some API changes (not that I would expect this to matter in practice for sklearn, but we will see). |
Currently the minimum required version of Numpy is 1.17.3 for users of scikit-learn. So for now we need the wheel to work with quite an old version.
Ok, this is a good reason to get this merged then! |
|
The backcompat in NumPy defaults to as far back as is plausible; as long you don't build stable API wheels (a single wheel for all Python 3.x versions). If you ever need to, you can override the default, but more likely, if things break, NumPy got it wrong. |
|
Do I understand correctly that the main goal of this PR is to build wheels against numpy development version and to test the wheels against the numpy latest release? |
The first, but it is important. We don't even want to test the wheels against the NumPy dev version. This shouldn't, and that is good! You are already testing against fully NumPy dev in the normal tests. |
Is there a way to do that by default (for users who build scikit-learn from source or for our cibuildwheel CI when we do releases) but remove that upper bound in our nightly builds? |
I don't understand this. We are testing against NumPy dev only in our "scipy-dev" CI entry. All other CI configs (e.g. the "normal" ones that run to check PRs and the ones that run when building the wheels run against released versions of NumPy). |
Yes, but it will it make users life easier? Users won't use NumPy 2 pre-releases anyway, unless they opt-in explicitly in some form. And if they want it, you are making it harder for them. Now release versions are probably better off with the upper pin, so if you prefer it to make sure that the release version has it then I can do that. It needs a bunch of extra branching to use no-build-isolation (or the weird hack of using
Yeah, and I think it both builds and tests with our nightly wheels. OTOH, the nightly wheels are build with 2.0dev now but I think also tested before upload. And that test does not use our nightlies here. |
|
I did a review of #27041. Once merged, please update this PR and trigger the cibuildwheel CI into this PR by pushing a commit with |
For those Python versions, just using NumPy defaults to the same backwards compatibility constraints as oldest-support-numpy.
NumPy 2.0 needs building with NumPy 2.0 (very soon) to allow some larger changes (technically, this is already necessary). Thus, nightlies must be build with NumPy 2.0 (it is better to not test them with it, which this does). This takes the simplified route of just using any pre-releases during nightly builds. If that fails (say cython is broken for a bit) then this could be adapted to explicitly install numpy and using `--no-build-isolation` instead. (Might require a bit of care to not use the same env when testing, not sure)
8b1b73f to
d1b29c1
Compare
Done, I am not sure what |
|
The wheel did not get built in the previous commit, it looks like you need |
|
FWIW, I did try on my fork in main to build the nightlies and chron trigger and it seemed OK. |
|
Merging this one, thanks! In the future, if we realise that installing the development all our dependencies when building the wheels fails too often, we can always look at installing only the development Numpy development version. |
There are two commits here:
>=1.25, except on Python <3.9 because NumPy 1.25 isn't compatible with that. (For new releases prior to NumPy 2.0, it would be good to add<2.0probably.)(one could also make use
<=2.0.0.dev0as a requirement. That is odd, but explicitly allows the nightlies, while rejecting a 2.0 release if it came up in an actual released version.)I moved the
"$CIRRUS_CRON" == "nightly"logic to avoid additional duplication. I tested it locally (in parts), and it build with 2.0 nightlies (failing, since my other PR is needed).This requires gh-27041 to be merged, or the build will fail!