Thanks to visit codestin.com
Credit goes to github.com

Skip to content

BLD: vendor meson-python to make the Windows builds with SIMD work #24396

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Aug 11, 2023

Conversation

rgommers
Copy link
Member

Follow-up to gh-24379. See the end of that PR for discussion on why that PR alone was not enough on Windows.

Tested together with SIMD support in gh-24395, and looks good. I plan to merge this asap and then rebase gh-23096 on top.

This also cleans up the Windows GHA CI jobs, they were pretty messy. meson as a standalone executable command cannot be used anymore as long as we vendor it.

Note that this is a bit messy. I tried to vendor spin's `meson.py`
separately, but it's not possible to do so cleanly as far as I can
tell. The import machinery is unhappy bouncing around between
`bin/spin`, `.spin/cmds.py and `.spin/meson.py`. So it was either
folding the `spin/cmds/meson.py` content all into cmds.py, or vendor all
of the spin package. This seems to work.
@rgommers rgommers added component: build component: CI Meson Items related to the introduction of Meson as the new build system for NumPy labels Aug 11, 2023
@rgommers rgommers added this to the 2.0.0 release milestone Aug 11, 2023
@rgommers rgommers added the 09 - Backport-Candidate PRs tagged should be backported label Aug 11, 2023
@github-actions github-actions bot added the 36 - Build Build related PR label Aug 11, 2023
@rgommers
Copy link
Member Author

This is good to go. @charris, @mattip, @seberg, or @stefanv anyone want to have a look, or give a thumbs up here to merge? I'd like to get things unblocked for the 1.26.0 pre-release ideally by the end of today.

@charris
Copy link
Member

charris commented Aug 11, 2023

LGTM. It will help with the backports if they are done in the correct order, so having that documented somewhere is desirable. Will all the linked OpenBLAS libraries all be 32 bits?

@rgommers
Copy link
Member Author

LGTM. It will help with the backports if they are done in the correct order, so having that documented somewhere is desirable.

Thanks for having a look. And sure, I will post a comment with an ordered list once we're ready to do them all.

Will all the linked OpenBLAS libraries all be 32 bits?

No, that's just for these couple of jobs here I think. All the wheel builds are using 64-bit OpenBLAS, so there is no change compared to 1.25.0 I believe.

@rgommers rgommers merged commit 5697016 into numpy:main Aug 11, 2023
@rgommers rgommers deleted the vendored-mesonpython branch August 11, 2023 15:39
@rgommers
Copy link
Member Author

In it goes then. On to gh-23096

@stefanv
Copy link
Contributor

stefanv commented Aug 11, 2023

This went in less than an hour after I was pinged to review, so unfortunately I didn't have time to give feedback. I think vendoring the whole of spin is almost certainly the wrong way to go, since it vastly complicates maintaining what is supposed to be a lean cmds.py.

I'll see what I can do to clear things up in a follow-up PR.

@stefanv
Copy link
Contributor

stefanv commented Aug 11, 2023

Just to understand the context here, why are we vendoring meson instead of installing it from, e.g., github?

@rgommers
Copy link
Member Author

I think vendoring the whole of spin is almost certainly the wrong way to go

I'm pretty sure that it is not. I burned 1.5 days on suggestions to avoid it, without success. The problem is that both meson-python and spin call subproces.run(['meson', ...]) and that will not work on Windows with the vendored Meson we now have. See #24379 (comment) and following comments.

Just to understand the context here, why are we vendoring meson instead of installing it from, e.g., github?

Because we need mesonbuild/meson#11307 to add SIMD support, and that's a large PR that isn't in a released Meson version. Since Meson by design cannot be extended by a third-party plugin or some such thing, we vendor it until our changes have been upstreamed and released.

@rgommers
Copy link
Member Author

Note that this is temporary, it will go away. So no worries about a long-term split of spin.

If you want the vendored content to go away asap, then there must be a way in spin itself to set the meson_cli executable as done in this PR. This is the key bit:

# The numpy-vendored version of Meson
meson_cli = [sys.executable,
             str(pathlib.Path(__file__).parent.parent.resolve() /
                 'vendor

That may be a nice feature to have in case this happens again.

I had a really hard time with debugging this stuff by the way. spin swallows tracebacks, so all I got were one-liners like:

abort: 'meson' is not a recognized command

or some such thing. Have you seen that too? You should be able to reproduce this by changing the 'meson' string to 'nonexisting'` in any command.

@stefanv
Copy link
Contributor

stefanv commented Aug 11, 2023

I'll get a spin PR ready ASAP; I'm sorry to hear you lost time on this :/

@rgommers
Copy link
Member Author

LGTM. It will help with the backports if they are done in the correct order, so having that documented somewhere is desirable.

@charris I think we're ready for the backports now, in this order:

I'd suggest then adding an xfail on the 1.26.x branch for gh-24399. At that point, we should have everything working, and we can add release notes and tag a pre-release. I suggest making it a beta rather than an RC, because it's highly likely that we'll have to make further tweaks. It also is a flag to downstream perhaps that they should actually pay attention to their build logs.

@charris
Copy link
Member

charris commented Aug 12, 2023

I think we're ready for the backports now, in this order:

Great, let's see how it goes.

@charris
Copy link
Member

charris commented Aug 13, 2023

The backports went in without much trouble.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
36 - Build Build related PR component: build component: CI Meson Items related to the introduction of Meson as the new build system for NumPy
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants