Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Remove package versions older than 30 days #33

Merged
merged 2 commits into from
Aug 14, 2023

Conversation

matthewfeickert
Copy link
Member

Resolves #31

  • Add curl and jq to environment requirements.
  • Use output from the anaconda.org API to get latest upload time for each release of a package. If the upload date is older than 30 days add the package release for removal.
  • Use anaconda.org API over anaconda-client CLI API to get package versions as easier to understand.

Running this now (with the remove step commented out) gives

# package: ipython
# scientific-python-nightly-wheels/ipython/8.13.2 last uploaded on 2023-05-26
# scientific-python-nightly-wheels/ipython/8.14.0.dev0 last uploaded on 2023-05-28
# Removing scientific-python-nightly-wheels/ipython/8.13.2
# Removing scientific-python-nightly-wheels/ipython/8.14.0.dev0

# package: matplotlib

# package: networkx

# package: numpy
# scientific-python-nightly-wheels/numpy/1.25.0rc1+93.g95343a3e6 last uploaded on 
# scientific-python-nightly-wheels/numpy/1.25.0rc1+104.g19f86c318 last uploaded on 
# scientific-python-nightly-wheels/numpy/1.25.0rc1+140.g2a243e698 last uploaded on 
# scientific-python-nightly-wheels/numpy/1.25.0rc1+218.g0e5a362fd last uploaded on 
# Removing scientific-python-nightly-wheels/numpy/1.25.0rc1+93.g95343a3e6
# Removing scientific-python-nightly-wheels/numpy/1.25.0rc1+104.g19f86c318
# Removing scientific-python-nightly-wheels/numpy/1.25.0rc1+140.g2a243e698
# Removing scientific-python-nightly-wheels/numpy/1.25.0rc1+218.g0e5a362fd

# package: openblas-libs

# package: pandas
# Removing scientific-python-nightly-wheels/pandas/2.1.0.dev0+1429.gcf741a492e

# package: scikit-image
# scientific-python-nightly-wheels/scikit-image/0.21.0rc2.dev0 last uploaded on 2023-05-28
# Removing scientific-python-nightly-wheels/scikit-image/0.21.0rc2.dev0

# package: scikit-learn
# scientific-python-nightly-wheels/scikit-learn/1.2.2 last uploaded on 2023-05-31
# Removing scientific-python-nightly-wheels/scikit-learn/1.2.2

# package: scipy
# scientific-python-nightly-wheels/scipy/1.10.1 last uploaded on 
# Removing scientific-python-nightly-wheels/scipy/1.10.1

# package: statsmodels
# scientific-python-nightly-wheels/statsmodels/0.14.0 last uploaded on 2023-06-04
# scientific-python-nightly-wheels/statsmodels/0.15.0.dev30+gfdd13b464 last uploaded on 2023-06-29
# Removing scientific-python-nightly-wheels/statsmodels/0.14.0
# Removing scientific-python-nightly-wheels/statsmodels/0.15.0.dev30+gfdd13b464

# package: xarray

With regards to why the numpy v1.25.0rc1 and scipy v1.10.1 distributions are missing a last uploaded date c.f. anaconda/anaconda-client#682 (comment)

* Add curl and jq to environment requirements.
* Use output from the anaconda.org API to get latest upload time for
  each release of a package. If the upload date is older than 30 days
  add the package release for removal.
* Use anaconda.org API over anaconda-client CLI API to get package
  versions as easier to understand.
@matthewfeickert matthewfeickert added enhancement New feature or request CI labels Aug 14, 2023
@matthewfeickert matthewfeickert self-assigned this Aug 14, 2023
@matthewfeickert matthewfeickert temporarily deployed to remove-old-wheels August 14, 2023 17:10 — with GitHub Actions Inactive
Copy link
Member

@bsipocz bsipocz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, besides one possible corner case.
If it doesn't make the removal loop to error out, then all looks good to go from my side.


# Don't need to check for duplicate versions added to removal
# given nightly CRON basis.
for package_version in $(cat package-versions.txt); do
Copy link
Member

@bsipocz bsipocz Aug 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very much a corner case, but could it still produce a duplicate when the No6 is just crosses the 30-day threshold?

(but then, if removing a non-existing package in the removal loop below doesn't raise an error, then this is not a problem at all)

@matthewfeickert matthewfeickert temporarily deployed to remove-old-wheels August 14, 2023 18:43 — with GitHub Actions Inactive
Copy link
Member

@bsipocz bsipocz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Remove wheels that are older than a threshold time limit
3 participants