Do we know of people who use the scikit-learn nightlies (e.g. projects that depend on scikit-learn who are testing)? How can we let them know that the location will change?

There are similar PRs for numpy, scipy, etc. Swapping scikit-learn over before the dependencies we test against are switched over will lead to some CI failures. I think that is fine, if we know about it.

It might be worth taking a moment to ponder how to make the transition and in which order.

Also, prior to switching to a new repo, we should first put tools in place to automatically clean-up old wheels and only keep the last 5 or 10 nightly builds for each project. We started a discussion on this a while ago here:

https://discuss.scientific-python.org/t/interest-in-github-action-for-scipy-wheels-nightly-uploads-and-removals/397

We are trying to collect tools and recommendations for this in scientific-python/specs#185 (this is probably also the source of the PRs)

Do we know of people who use the scikit-learn nightlies (e.g. projects that depend on scikit-learn who are testing)? How can we let them know that the location will change?

Plan is to open a PR (we have the tooling for it) for each project we find on github using the old URL once all the core scientific python projects have moved, and produced at least one wheel to the new location

Logistically, we'll need to generate a new token for https://anaconda.org/scientific-python-nightly-wheels. For security reasons, each project in scipy-wheels-nightly have their own tokens. Also there are many maintainers on scipy-wheels-nightly that have permission to generate a new token for their project.

For scientific-python-nightly-wheels, are you thinking of using the same permission setup as scipy-wheels-nightly?

Yes, the idea is that this will be similar. @jarrodmillman already added some people but we assume that anyone who needs it will get access to generate project-specific tokens.

To address @betatim's concerns (#26417 (comment)) maybe we could upload both to the old and the new location for some time to ensure a smooth transition and leave time to downstream projects (including potentially unknown private repositories?) to adapt their own CIs?

BTW, why do we need a new channel in the first place?

BTW, why do we need a new channel in the first place?

I think one motivation for moving to a new name is branding (and with that a win for scientific python, which then helps with getting funding). Another aspect is that I think it would be nice to expand the number of packages that create nightlies and use nightlies in their CI to help catch problems earlier (aka less needs for "a quick patch release" after making a release). If we want to expand the number of projects uploading nightlies I think it makes sense to move the ownership (and responsibility) of keeping it secure, adding/removing people, etc to a place where there is a group of people taking care of it as well as making it more obvious who to contact to get access. I have to admit that I don't really know who/what takes care of the current nightlies area, I assume it is mostly Olivier.

I think the goal is to write up who to contact, etc in the SPEC I linked in a previous comment. But it still needs a bunch of work to figure out what to add to it, what the answers should be, etc.

I have to admit that I don't really know who/what takes care of the current nightlies area, I assume it is mostly Olivier.

There are a bunch of anaconda accounts with admin rights from various scipy related projects.

The current status (from scientific-python/specs#182):

• numpy has already done it CI: Upload nighlighties to new location numpy/numpy#23792
• scipy is likely to do it soon, Friday May 26 (tomorrow at the time of writing) is proposed in MAINT: upload nighlighties to new location scipy/scipy#18504 (comment)
• no answer from pandas yet Upload nightlies to new location pandas-dev/pandas#53341 and we install the development wheel from pandas in our CI

I don't think the disruption is going to be too great for scikit-learn development wheels consumers. Looking at download stats from anaconda.org, the most downloaded scikit-learn wheel has been downloaded 15 times:
https://anaconda.org/scipy-wheels-nightly/scikit-learn/files?sort=ndownloads&sort_order=desc&version=1.3.dev0

I double-checked that this count gets updated by doing pip install, although maybe the download count gets reset when we upload a new wheel with the same name as we do in scikit-learn.

As suggested in the scipy similar PR scipy/scipy#18504 (comment) we could upload a dummpy package sdist that raises an error and mentions the new URL to push people towards the new URL and avoid they use an outdated development wheel.

scientific-python/specs#195 is the latest PR I have to update SPEC4

I think if we announce the new URL on the mailing list and other places where the old URL was announced that should help catch a lot of the users that we don't know of.

Hi folks,

So here are the latest information about the migration. NumPy and SciPy have now migrated (still some small things to adjust, but some wheels are there.)

To add a new package this needs to be done (instructions have been updated on the SPEC4):

1. Tell us who needs to be responsible for nighlies for your package. And give us the anaconda handles.
2. We download from PyPi a wheel and upload it using an admin token. This will create the package.
3. We link the package to your group.
4. You can now create a personal token and upload your package.

Let me know if you need any assistance, I am happy to make calls to speed up things or we can chat on Slack/Discord.

As for the cleanup. We are discussing doing this on our side for all projects. We will keep folks in the loop.

@tupui I assume @thomasjpfan and I can continue to represent scikit-learn w.r.t. nightly wheel uploads:

• https://anaconda.org/thomasjpfan
• https://anaconda.org/ogrisel

Maybe @betatim is interested to join us?

Sounds like a plan. I am https://anaconda.org/betatim

@ogrisel done. There was also @jjerphan so you are now 3 in your group. You should all have admin access and can add/remove folks in your group (I think, let me know if you have any issues.) We don't impose anything here. We just ask admins to be mindful of security and it helps to reduce the attack surface if only a few folks have access.

If you want I can do the next step which is to download any wheel from PyPi and upload it to create the package. Then you can take over.

I think before we can switch over scikit-learn to the new location we need pandas to be uploading there as well. So it might take a few more days before we can start publishing a nightly in the new place. I guess there isn't much downside to you already uploading a wheel there? So yeah go ahead please

Done, I uploaded scikit_learn-1.2.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl. From now on you should be able to take over with a personal token.

@betatim The Pandas PR was merged pandas-dev/pandas#53341. And it looks like all the wheels are uploaded now: https://anaconda.org/scientific-python-nightly-wheels/pandas/files

You may want to take a quick look to make sure all the wheels you expect are there. Let me know if there are any issues.

I triggered a scipy-dev build and the install works fine from the new location. There are some test failures but there are also some failures on main.

I guess what is left to do is to create a token and add a secret to the scipy-dev wheel upload job?

Yes, the 3 folks above can create a personal token and update the upload job.

May I have a gentle ping on this? I would like to open PRs to downstreams that use scipy-wheels-nightly, but would like to do this once all the packages migrated, and scikit-learn is the only one remaining.

@ogrisel @betatim @thomasjpfan if one of you adds me (anaconda.org username: lesteve), I can have a look.

By the way, I am guessing there is no way to test the wheels upload works before merging this PR?

@ogrisel @betatim @thomasjpfan if one of you adds me (anaconda.org username: lesteve), I can have a look.

By the way, I am guessing there is no way to test the wheels upload works before merging this PR?

You can test locally the credentials with the Anaconda CLI. But if the current mechanism is working, there is no reason the new one is not. It's strictly the same thing, you just need to update the upload location and the token.

I generated a personal API token and stored it in SCIKIT_LEARN_NIGHTLY_UPLOAD_TOKEN for this repository. The token will expire on 1st Dec 2099.

Let's see what happens tomorrow morning.

Just a recommendation, nothing more. Besides having set the minimal permissions for the token, I would advice you to set an expiry date which is closer to a year.

Something you might want to do is to edit the page description as Anaconda only understand MD. The changes won't be affected by a new upload.

Looks like it worked fine: https://anaconda.org/scientific-python-nightly-wheels/scikit-learn/files?version=1.4.dev0