CI: Add removal of old uploads #12

matthewfeickert · 2023-05-31T03:57:01Z

Resolves #4

Remove all but the last N_LATEST_UPLOADS (defaulted to 5) package version uploads to the package index to ensure space. To do this, reply on the output form of anaconda show to be able to filter on the item delimiter character sequence for each version currently uploaded.

As an explicit example:

$ anaconda show scientific-python-nightly-wheels/xarray
Using Anaconda API: https://api.anaconda.org
Name:    xarray
Summary: 
Access:  public
Package Types:  Standard Python
Versions:
   + 2023.5.1.dev2+gd8ec3a3f
   + 2023.5.1.dev5+gf96aca45
   + 2023.5.1.dev6+g69445c62
   + 2023.5.1.dev9+g609a9016
   + 2023.5.1.dev10+gf45eb733
   + 2023.5.1.dev11+gfcb81756
   + 2023.5.1.dev12+gb319d867
   + 2023.5.1.dev14+ge3db6164

To install this package with pypi run:
     pip install -i https://pypi.anaconda.org/scientific-python-nightly-wheels/simple xarray

shows that by filtering on +

$ anaconda show scientific-python-nightly-wheels/xarray &> >(grep '+')
   + 2023.5.1.dev2+gd8ec3a3f
   + 2023.5.1.dev5+gf96aca45
   + 2023.5.1.dev6+g69445c62
   + 2023.5.1.dev9+g609a9016
   + 2023.5.1.dev10+gf45eb733
   + 2023.5.1.dev11+gfcb81756
   + 2023.5.1.dev12+gb319d867
   + 2023.5.1.dev14+ge3db6164

and then stripping ' + '

$ anaconda show scientific-python-nightly-wheels/xarray &> >(grep '+') | \
    awk '{print $2}'
2023.5.1.dev2+gd8ec3a3f
2023.5.1.dev5+gf96aca45
2023.5.1.dev6+g69445c62
2023.5.1.dev9+g609a9016
2023.5.1.dev10+gf45eb733
2023.5.1.dev11+gfcb81756
2023.5.1.dev12+gb319d867
2023.5.1.dev14+ge3db6164

one can obtain a newline separated list of all package uploads, where the most recent uploads are listed last. After stripping off the N_LATEST_UPLOADS lines that correspond to the package versions to keep for testing

$ anaconda show scientific-python-nightly-wheels/xarray &> >(grep '+') | \
    awk '{print $2}' | \
    head --lines -5 > remove-package-versions.txt
$ cat remove-package-versions.txt 
2023.5.1.dev2+gd8ec3a3f
2023.5.1.dev5+gf96aca45
2023.5.1.dev6+g69445c62

the remaining (older) package uploads can be removed with anaconda remove.

Most of this is taken from matplotlib/matplotlib#23349 (c.f. #4 (comment)).

To provide a local reproducible example of the behavior, curl the following attached Shell script into a continuumio/miniconda3:latest container with anaconda-client installed

example-pr-12.sh:

# example-pr-12.sh
#!/bin/bash

# fail on undefined variables
set -u
# Prevent pipe errors to be silenced
set -o pipefail
# Exit if any command exit as non-zero
set -e

# Input variables
ANACONDA_USER="scientific-python-nightly-wheels"
INPUT_ARTIFACTS_PATH="dist"
N_LATEST_UPLOADS=5

anaconda show "${ANACONDA_USER}" &> >(grep "${ANACONDA_USER}/") | \
    awk '{print $1}' | \
    sed 's|.*/||g' > package-names.txt

if [ -s package-names.txt ]; then
    while LANG=C IFS= read -r package_name ; do

        printf "\n# package: %s\n" "${package_name}"

        anaconda show "${ANACONDA_USER}/${package_name}" &> >(grep '+') | \
            awk '{print $2}' | \
            head --lines "-${N_LATEST_UPLOADS}" > remove-package-versions.txt

        if [ -s remove-package-versions.txt ]; then
            while LANG=C IFS= read -r package_version ; do
                echo "# Removing ${ANACONDA_USER}/${package_name}/${package_version}"
                # anaconda --token ${{ secrets.ANACONDA_TOKEN }} remove \
                #   --force \
                #   "${ANACONDA_USER}/${package_name}/${package_version}"
            done <remove-package-versions.txt
        fi

    done <package-names.txt
fi

$ docker run --rm -ti continuumio/miniconda3:latest /bin/bash
(base) root@5d1ea993cbe9:/# conda install -y anaconda-client -c conda-forge
(base) root@5d1ea993cbe9:/# apt update && apt install -y curl
(base) root@5d1ea993cbe9:/# curl -sL https://github.com/scientific-python/upload-nightly-action/files/11618429/example-pr-12.sh.txt -o example-pr-12.sh
(base) root@5d1ea993cbe9:/# bash example-pr-12.sh 

# package: ipython

# package: matplotlib

# package: networkx

# package: numpy

# package: scikit-image

# package: scikit-learn

# package: scipy

# package: xarray

(base) root@5d1ea993cbe9:/# sed -i 's/N_LATEST_UPLOADS=5/N_LATEST_UPLOADS=2/g' example-pr-12.sh  # demo purposes
(base) root@5d1ea993cbe9:/# bash example-pr-12.sh

# package: ipython

# package: matplotlib
# Removing scientific-python-nightly-wheels/matplotlib/3.6.0.dev3181+g9f17f3f851
# Removing scientific-python-nightly-wheels/matplotlib/3.6.0.dev3182+gd6bad5f684

# package: networkx

# package: numpy
# Removing scientific-python-nightly-wheels/numpy/1.25.0.dev0+1532.g5d6c744db

# package: scikit-image

# package: scikit-learn

# package: scipy

# package: xarray
# Removing scientific-python-nightly-wheels/xarray/2023.5.1.dev9+g609a9016
# Removing scientific-python-nightly-wheels/xarray/2023.5.1.dev10+gf45eb733
# Removing scientific-python-nightly-wheels/xarray/2023.5.1.dev11+gfcb81756
(base) root@5d1ea993cbe9:/#

example-pr-12.sh.txt

I'm marking this as closing Issue #4 despite that it doesn't include a time cutoff as described in #4 (comment) given that I do not know of a pip or anaconda-client API that gets upload date information from the index (Probably exists somewhere though?) and this PR resolves all other matters as far as I understand it. We can revise or extend this if desired.

action.yml

entrypoint.sh

tupui

Thank you @matthewfeickert. If that's the same approach that Matplotlib has been using successfully, then all good to me.

The only thing to me is that, IIUC this is to be done per project right? On SciPy and NumPy at least, we are not using actions to do the upload but we rely on CirrusCI. So we would need to add another action that would not upload and just check for removal? (In practice we overwrite our upload and only have a single version. But other packages might do differently.)

Should we have a nightly action on this repo that would run this for all packages?

matthewfeickert · 2023-05-31T15:05:11Z

The only thing to me is that, IIUC this is to be done per project right? On SciPy and NumPy at least, we are not using actions to do the upload but we rely on CirrusCI. So we would need to add another action that would not upload and just check for removal? (In practice we overwrite our upload and only have a single version. But other packages might do differently.)

Should we have a nightly action on this repo that would run this for all packages?

Ah I see that I have misunderstood what "centralise" meant in #4 (comment).

@tupui so for SciPy and NumPy you use CirrusCI for all the uploads to https://anaconda.org/scientific-python-nightly-wheels? If so, then yes, this would all need to be a separate action and this PR should be closed.

I think that it would be better to do a 1-action-per-repo policy, so in that case this PR should get closed and I could port the relevant logic to the new repo (I'm not very good at naming things so remove-nightly-wheels or something of that sort).

tupui · 2023-05-31T15:14:01Z

Let's continue the discussion on the issue before we make any decisions about closing etc. 😃

matthewfeickert

Given #4 (comment) please see these updated changes. I have tested this already on my fork with matplotlib (c.f. the aside in #8 (comment) if wondering why I'm doing so) and things work as expected (which is good as I'm copying an existing workflow I wrote :P).

.github/workflows/remove-wheels.yml

stefanv · 2023-05-31T19:08:02Z

Agreed that querying the index for the list of packages would make this a bit more robust against future package additions.

Aside, an alternative to sed magic:

| awk '{print $2}'

tupui

Thank you @matthewfeickert 🚀 That's good on my side. Approving as the suggestion is optional to me for now.

.github/workflows/remove-wheels.yml

tupui

Good on my side 🚀 thanks @matthewfeickert

stefanv

If left some minor bash styling feedback, but feel free to take it or leave it.

.github/workflows/remove-wheels.yml

* Remove the workflow step for querying and removing all but the last 5 nightly wheel uploads to the scientific-python-nightly-wheels Anaconda Cloud package index as this is now centrally done by the Scientific Python org for all projects that upload to https://anaconda.org/scientific-python-nightly-wheels. - c.f. scientific-python/upload-nightly-action#12

.github/workflows/remove-wheels.yml

.github/workflows/dependabot.yml

.github/workflows/remove-wheels.yml

matthewfeickert · 2023-06-08T15:31:28Z

Thanks for the review @jarrodmillman! Does your approval mean that you were able to setup the remove-old-wheels GitHub Actions environment with the Anaconda cloud API token as an environment restricted secret? If so, then I think we can merge.

* Add GitHub Actions workflow to remove old wheels from the package index each night at 1:23 UTC. Also allow for manual runs via workflow dispatch. * Remove all but the last N_LATEST_UPLOADS package version uploads to the package index to ensure space. To do this, reply on the output form of `anaconda show` to be able to filter on the item delimiter character sequence for each version currently uploaded. As an explicit example: ``` $ anaconda show scientific-python-nightly-wheels/xarray Using Anaconda API: https://api.anaconda.org Name: xarray Summary: Access: public Package Types: Standard Python Versions: + 2023.5.1.dev9+g609a9016 + 2023.5.1.dev10+gf45eb733 + 2023.5.1.dev11+gfcb81756 + 2023.5.1.dev12+gb319d867 + 2023.5.1.dev14+ge3db6164 To install this package with pypi run: pip install -i https://pypi.anaconda.org/scientific-python-nightly-wheels/simple xarray ``` shows that by filtering on '+' and then stripping ' + ' one can obtain a newline separated list of all package uploads, where the _most recent_ uploads are listed last. After stripping off the N_LATEST_UPLOADS lines that correspond to the package versions to keep for testing, the remaining (older) package uploads can be removed with `anaconda remove`.

* Use Dependabot to keep the GitHub Actions used in workflows up to date.

jarrodmillman · 2023-06-08T16:31:05Z

See #12 (comment)

.github/workflows/remove-wheels.yml

jarrodmillman · 2023-06-09T16:38:49Z

Thanks @matthewfeickert !!!

matthewfeickert · 2023-06-09T16:45:38Z

Just did a manual run with workflow dispatch and all looks good!

# package: ipython

# package: matplotlib
# Removing scientific-python-nightly-wheels/matplotlib/3.6.0.dev3181+g9f17f3f851
Using Anaconda API: https://api.anaconda.org/
# Removing scientific-python-nightly-wheels/matplotlib/3.6.0.dev3182+gd6bad5f684
Using Anaconda API: https://api.anaconda.org/
# Removing scientific-python-nightly-wheels/matplotlib/3.8.0.dev1157+g2f778fda79
Using Anaconda API: https://api.anaconda.org/
# Removing scientific-python-nightly-wheels/matplotlib/3.8.0.dev1150+g3d3a1afd6d
Using Anaconda API: https://api.anaconda.org
# Removing scientific-python-nightly-wheels/matplotlib/3.8.0.dev1192+gbb335a115c
Using Anaconda API: https://api.anaconda.org
# Removing scientific-python-nightly-wheels/matplotlib/3.8.0.dev1211+gdcb8180edc
Using Anaconda API: https://api.anaconda.org
# Removing scientific-python-nightly-wheels/matplotlib/3.8.0.dev1228+g19d93b7876
Using Anaconda API: https://api.anaconda.org

# package: networkx

# package: numpy
# Removing scientific-python-nightly-wheels/numpy/1.25.0.dev0+1532.g5d6c7[44](https://github.com/scientific-python/upload-nightly-action/actions/runs/5224131691/jobs/9431935913#step:6:45)db
Using Anaconda API: https://api.anaconda.org

# package: pandas

# package: scikit-image

# package: scikit-learn

# package: scipy

# package: statsmodels

# package: xarray
# Removing scientific-python-nightly-wheels/xarray/2023.5.1.dev9+g609a9016
Using Anaconda API: https://api.anaconda.org
# Removing scientific-python-nightly-wheels/xarray/2023.5.1.dev10+gf[45](https://github.com/scientific-python/upload-nightly-action/actions/runs/5224131691/jobs/9431935913#step:6:46)eb733
Using Anaconda API: https://api.anaconda.org/
# Removing scientific-python-nightly-wheels/xarray/2023.5.1.dev11+gfcb81756
Using Anaconda API: https://api.anaconda.org/
# Removing scientific-python-nightly-wheels/xarray/2023.5.1.dev12+gb319d867
Using Anaconda API: https://api.anaconda.org/
# Removing scientific-python-nightly-wheels/xarray/2023.5.1.dev14+ge3db6164
Using Anaconda API: https://api.anaconda.org/

Should be good to go for nightly cron jobs now and then just need to make sure that the workflow doesn't get shut off due to lack of repo activity.

stefanv · 2023-06-09T17:38:48Z

Awesome, thanks @matthewfeickert! I didn't realize inactive repos have their actions switched off. What do we need to do; a monthly empty commit?

matthewfeickert · 2023-06-09T18:27:22Z

I didn't realize inactive repos have their actions switched off. What do we need to do; a monthly empty commit?

Yeah, from https://docs.github.com/en/actions/managing-workflow-runs/disabling-and-enabling-a-workflow

Warning: To prevent unnecessary workflow runs, scheduled workflows may be disabled automatically. When a public repository is forked, scheduled workflows are disabled by default. In a public repository, scheduled workflows are automatically disabled when no repository activity has occurred in 60 days.

Though you will also get an email as a heads up and you can click a button in the email to keep the workflow alive. I'm also not 100% if "repository activity" means a Git operation or if it just means interaction with the repository in some way on GitHub.

My assumption is that as we have Dependabot turned on this won't be an issue in general, but if it becomes one then we can also just have a GHA workflow that once a month checks out a keep-alive branch, does a git commit --amend --no-edit, and --force-with-lease pushes back to the keep-alive branch. There's also various GitHub Action workflows on the GitHub Actions marketplace that do this sort of thing for you.

bsipocz · 2023-06-09T19:25:58Z

It could also push a timestamp of its last run, maybe in a separate, empty branch, or gh-pages?

matthewfeickert · 2023-06-09T19:35:25Z

It could also push a timestamp of its last run, maybe in a separate, empty branch, or gh-pages?

Yup, but then you're accumulating a daily commit history on that other branch that still lives in the repo for no real reason. Is it easy to ignore? Sure, but seems kinda unnecessary. As you point out though, you could use gh-pages as a way to throw this stuff into the void in some sense, but if for any reason in the future you want to use the gh-pages for an actual website then you need to shift things around.

Personally, I would put this problem in the "kick it down the road until it becomes a problem" bin, as for the multiple projects that I maintain that are are at this stage repos for LTS Docker images this hasn't come up much. Though if anyone has been nerdsniped by this conversation please go for it! 😉

bsipocz · 2023-06-09T19:39:46Z

I had repos where we run into this in practice, and wondered why the actions stopped working. So I would just like to avoid being on that road again :)

stefanv · 2023-06-09T20:18:51Z

One commit every 50 days on a keepalive branch isn't too bad :)

https://github.com/marketplace/actions/keepalive-workflow

bsipocz · 2023-06-09T20:22:27Z

Ahh, nice, thanks for the link. I was starting to hack something together, but it's probably much easier to just do the action. I'll open a PR for this.

* Remove the workflow step for querying and removing all but the last 5 nightly wheel uploads to the scientific-python-nightly-wheels Anaconda Cloud package index as this is now centrally done by the Scientific Python org for all projects that upload to https://anaconda.org/scientific-python-nightly-wheels. - c.f. scientific-python/upload-nightly-action#12

matthewfeickert added the enhancement New feature or request label May 31, 2023

matthewfeickert self-assigned this May 31, 2023

matthewfeickert requested review from Carreau, betatim, bsipocz and tupui May 31, 2023 03:57

matthewfeickert commented May 31, 2023

View reviewed changes

action.yml Outdated Show resolved Hide resolved

entrypoint.sh Outdated Show resolved Hide resolved

entrypoint.sh Outdated Show resolved Hide resolved

matthewfeickert commented May 31, 2023

View reviewed changes

entrypoint.sh Outdated Show resolved Hide resolved

tupui reviewed May 31, 2023

View reviewed changes

matthewfeickert mentioned this pull request May 31, 2023

Old wheels culling. #4

Closed

matthewfeickert force-pushed the feat/add-removal-of-last-n-uploads branch from 2577b71 to 56cd8fe Compare May 31, 2023 17:44

matthewfeickert changed the title ~~ENH: Add removal of old uploads~~ CI: Add removal of old uploads May 31, 2023

matthewfeickert commented May 31, 2023

View reviewed changes

.github/workflows/remove-wheels.yml Show resolved Hide resolved

.github/workflows/remove-wheels.yml Show resolved Hide resolved

.github/workflows/remove-wheels.yml Show resolved Hide resolved

.github/workflows/remove-wheels.yml Outdated Show resolved Hide resolved

matthewfeickert requested a review from tupui May 31, 2023 17:56

tupui approved these changes May 31, 2023

View reviewed changes

.github/workflows/remove-wheels.yml Outdated Show resolved Hide resolved

matthewfeickert force-pushed the feat/add-removal-of-last-n-uploads branch from 56cd8fe to 5fd637f Compare May 31, 2023 19:51

matthewfeickert requested a review from tupui May 31, 2023 20:04

tupui approved these changes May 31, 2023

View reviewed changes

stefanv approved these changes May 31, 2023

View reviewed changes

matthewfeickert mentioned this pull request Jun 1, 2023

CI: Use scientific-python/upload-nightly-action matplotlib/matplotlib#26023

Merged

matthewfeickert force-pushed the feat/add-removal-of-last-n-uploads branch from 5fd637f to d28e1de Compare June 1, 2023 05:45

matthewfeickert requested a review from stefanv June 1, 2023 05:45

matthewfeickert commented Jun 1, 2023

View reviewed changes

.github/workflows/remove-wheels.yml Show resolved Hide resolved

matthewfeickert commented Jun 1, 2023

View reviewed changes

.github/workflows/dependabot.yml Outdated Show resolved Hide resolved

matthewfeickert force-pushed the feat/add-removal-of-last-n-uploads branch 2 times, most recently from 1d24930 to c121a1e Compare June 5, 2023 21:58

matthewfeickert commented Jun 5, 2023

View reviewed changes

.github/workflows/remove-wheels.yml Show resolved Hide resolved

jarrodmillman approved these changes Jun 8, 2023

View reviewed changes

matthewfeickert force-pushed the feat/add-removal-of-last-n-uploads branch from c121a1e to 85b99a0 Compare June 8, 2023 16:00

jarrodmillman added this to the 0.2.0 milestone Jun 8, 2023

matthewfeickert added 2 commits June 8, 2023 11:23

CI: Add Dependabot config file for GitHub Actions

e964540

* Use Dependabot to keep the GitHub Actions used in workflows up to date.

matthewfeickert force-pushed the feat/add-removal-of-last-n-uploads branch from 85b99a0 to e964540 Compare June 8, 2023 16:23

jarrodmillman marked this pull request as draft June 8, 2023 16:26

stefanv approved these changes Jun 8, 2023

View reviewed changes

stefanv reviewed Jun 8, 2023

View reviewed changes

.github/workflows/remove-wheels.yml Show resolved Hide resolved

tupui marked this pull request as ready for review June 9, 2023 14:51

tupui approved these changes Jun 9, 2023

View reviewed changes

jarrodmillman merged commit c049309 into scientific-python:main Jun 9, 2023

matthewfeickert deleted the feat/add-removal-of-last-n-uploads branch June 9, 2023 16:42

matthewfeickert mentioned this pull request Jun 9, 2023

Avoid scheduled workflows becoming disabled due to lack of repo activity #17

Closed

matthewfeickert mentioned this pull request Aug 15, 2023

MNT: Use conda-lock lock file for reproducible environment #32

Merged

CI: Add removal of old uploads #12

CI: Add removal of old uploads #12

Uh oh!

Conversation

matthewfeickert commented May 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tupui left a comment

Choose a reason for hiding this comment

Uh oh!

matthewfeickert commented May 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tupui commented May 31, 2023

Uh oh!

matthewfeickert left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stefanv commented May 31, 2023

Uh oh!

tupui left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tupui left a comment

Choose a reason for hiding this comment

Uh oh!

stefanv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

matthewfeickert commented Jun 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jarrodmillman commented Jun 8, 2023

Uh oh!

Uh oh!

jarrodmillman commented Jun 9, 2023

Uh oh!

matthewfeickert commented Jun 9, 2023

Uh oh!

stefanv commented Jun 9, 2023

Uh oh!

matthewfeickert commented Jun 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bsipocz commented Jun 9, 2023

Uh oh!

matthewfeickert commented Jun 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bsipocz commented Jun 9, 2023

Uh oh!

stefanv commented Jun 9, 2023

Uh oh!

bsipocz commented Jun 9, 2023

Uh oh!

Uh oh!

matthewfeickert commented May 31, 2023 •

edited

Loading

matthewfeickert commented May 31, 2023 •

edited

Loading

matthewfeickert commented Jun 8, 2023 •

edited

Loading

matthewfeickert commented Jun 9, 2023 •

edited

Loading

matthewfeickert commented Jun 9, 2023 •

edited

Loading