Thanks to visit codestin.com
Credit goes to github.com

Skip to content

CI: Upload nighlighties to new location #23792

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 24, 2023
Merged

Conversation

jarrodmillman
Copy link
Member

No description provided.

@charris
Copy link
Member

charris commented May 22, 2023

Should post to the mailing list, and also make a case for the change. My first guestion is "Why?".

@seberg seberg marked this pull request as ready for review May 23, 2023 00:05
@rossbar
Copy link
Contributor

rossbar commented May 23, 2023

Should post to the mailing list, and also make a case for the change. My first guestion is "Why?".

This is being worked on during the scientific Python summit - the main motivation is to move to a better name and more visibly-community-owned location, i.e. "scipy" -> "scientific python". See also networkx/networkx#6700. This will only affect CI and @bsipocz et. al. will handle all the downstream updates.

@martinfleis
Copy link

A bit of context - the Scientific Python Developer Summit is happening this week. A part of the effort is to ensure that Scientific Python core packages follow the upcoming SPEC documents and support their implementation across the ecosystem. One of the specific changes is a move of the nightly wheels from pypi.anaconda.org/scipy-wheels-nightly which is planned to be closed to pypi.anaconda.org/scientific-python-nightly-wheels. That is what this PR does.

Once all of the packages that currently use scipy-wheels-nightly switch to scientific-python-nightly-wheels, we will use a bot to open PRs in downstream repositories to automatically change the index for nightlies across the ecosystem to minimise breakage.

One more step will need to happen here though. The upload requires an upload token stored as a repository secret (as you have right now). However, you will need to generate a new token for the new anaconda organisation. Let us know who is managing this at the moment so we can give you a proper access to the new repo so you can generate a new token, unless @jarrodmillman already added you.

The same change is happening in the other packages as well.

@charris
Copy link
Member

charris commented May 23, 2023

@martinfleis I suspect that this is more related to promoting scientific python and bringing everything under that umbrella. Why not, for instance, make it numfocus site instead :) What about the staging repositories? It feels like meddling to start with the implementation rather than the argument.

@mattip
Copy link
Member

mattip commented May 23, 2023

If I understand the intent correctly, the idea is to move the archive to a new group with a better name, without any change in policy, making this is a cosmetic change. Is that correct?

Here is the page for the owners of the old location. Among them I recognize @bsipocz and @tylerjereddy (maybe more)? who are listed as participants in the summit. If indeed the intent is cosmetic, they can invite all the other owners to the new org.

@rgommers
Copy link
Member

Once all of the packages that currently use scipy-wheels-nightly switch to scientific-python-nightly-wheels, we will use a bot to open PRs in downstream repositories to automatically change the index for nightlies across the ecosystem to minimise breakage.

We have encouraged all downstream package authors, of both public and private packages, to use these nightlies (see, e.g., https://numpy.org/devdocs/dev/depending_on_numpy.html#testing-against-the-numpy-main-branch-or-pre-releases) for years now. I don't see how you're going to be able to open PRs to the repo of every consumer. This feels a bit disruptive for little gain.

@bsipocz
Copy link
Member

bsipocz commented May 23, 2023

I don't see how you're going to be able to open PRs to the repo of every consumer

right, we won't be able to find the private packages

@mattip
Copy link
Member

mattip commented May 23, 2023

You could use both indexes for a few years, until we see the download statistics drop off

@seberg
Copy link
Member

seberg commented May 23, 2023

We could upload both for a bit, the thing I would worry is about someone using an outdated channel for months. Maybe we can prevent that e.g. by uploading an "invalid" wheel that prints out a message?

If this helps with getting a lot more things into one location (and we want that), say astropy, Scikit-HEP, ... Than maybe the disruption is fine? I doubt that many use the channel overall and most of them opt-in to be alpha testers.

And since many would be added soonish it would be the last good time to do cosmetic changes, if it is just cosmetic (if those packages prefer the new name).

@tylerjereddy
Copy link
Contributor

Maybe we can prevent that e.g. by uploading an "invalid" wheel that prints out a message?

That was suggested over in SciPy at scipy/scipy#18504 (comment)

@tupui
Copy link
Contributor

tupui commented May 23, 2023

What SciPy is an is not is a real source of confusion for most of our newcomers. For folks who have been long enough the community to know the backstories, it's clear, automatic and does not matter. But for everyone else, it's strange. Moving to a new org which is not tied to a specific project is sensible and is a clear messaging (helpful for many things.)

NumFOCUS' current purpose is not really to provide infrastructure nor development tools. That's one of the role of Scientific Python though which makes it, IMHO, the perfect place to have that.

As for downstream users, pushing a broken release with instructions should do it. It's a one line change, pretty much insignificant for folks really using nightlies. If they do use nightlies, it's probably because something in NumPy is breaking them from time to time. This would just be yet another similar "breakage" with a clear resolution path.

Also stats on how much these are downloaded don't really show a lot of usage https://anaconda.org/scipy-wheels-nightly/numpy/files

tiffanyhowser

This comment was marked as spam.

@seberg seberg changed the title Upload nighlighties to new location CI: Upload nighlighties to new location May 24, 2023
@seberg
Copy link
Member

seberg commented May 24, 2023

We discussed this briefly today and I don't think anyone had serious concerns, so putting this i. @jarrodmillman and @bsipocz are planning to make sure downstream is being moved timely, if there is some fallout, it is easy to revert.

Thanks @jarrodmillman!

@seberg seberg merged commit 5d6c744 into numpy:main May 24, 2023
@seberg
Copy link
Member

seberg commented May 24, 2023

FYI, the first batch of wheels are now (almost) uploaded: https://anaconda.org/scientific-python-nightly-wheels/numpy so downstream can change their sources (assume they don't rely on other packages like SciPy that didn't upload yet).

@mattip
Copy link
Member

mattip commented May 31, 2023

We discussed this briefly today and I don't think anyone had serious concerns,

People had concerns and expressed them here. Maybe they were not serious, but neither mitigation policy was implemented:

  • upload broken wheels to the old site
  • continue to upload to both sites

and the download numbers would indicate that the communication to the downstream users was not complete, even though the NumPy documentation now points to the new site.

@martinfleis
Copy link

@bsipocz what is the state of the tool that shall auto-open PRs in downstream projects pointing to the new location?

@seberg
Copy link
Member

seberg commented May 31, 2023

Transition is indeed not complete. Right now its probably mainly pandas that needs to merge (to unblock sklearn also). SciPy needs to upload to the new url, but it sounds like that will happen today.
I assume after that @bsipocz will do the mass change very quickly, and I almost bet just those three finalizing transition may shifts the download stats to indicate a largely complete transition.

I do want the broken upload also, but it only makes sense after scipy, pandas, and sklearn are done. If that didn't happen in a week, I guess we may have to do the dual upload.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.