Codestin Search App

astrofrog · 2023-05-12T13:02:10Z

As discussed in #14756 (comment), it might be worth splitting the IERS Earth Rotation and Leap Second files (along with the URLs and ReadMe files) out into a dedicated package, so this is an experimental PR doing this. The experimental astropy-iers-data package can be found here:

https://github.com/astrofrog/astropy-iers-data

The benefits of having dedicated package are:

If people need the latest files, more recent that what we would have bundled in astropy in the past, we can tell them to simply pip/conda update the astropy-iers-data package. In environments that have restricted internet access this could also mean downloading the source/wheel from PyPI and installing it manually on those machines, which is easier than asking users to download each different file and then get astropy to point at them.
If the URL of the files breaks at short notice, we can easily release a new astropy-iers-data version and tell users to update, rather than having to do a whole astropy core package release.
The sdists and wheels for the core package will be smaller as they won't include the data files, and the repository will also not get cluttered over time with changes to these large files (both are 0.8Mb smaller with this PR)
Because we don't need to worry so much about data size, I've included the IERS A files in astropy-iers-data so that all files can now be accessed without an internet connection
We could in principle auto-update the package as often as we want - we could even release weekly or daily if we wanted (though need to check how soon we'd reach the PyPI quota)

To be clear, this will essentially have no impact on the average user because astropy-iers-data will just get pulled in as a required dependency of astropy.

The slight downside is maintenance of a separate package, but I have set it up so that it will auto-update the data files (as we did in the core package until now) via PRs and we could always enable auto-releasing each time a PR is merged to minimize overhead.

I've set up a simple CI job in astropy-iers-data which runs the astropy.utils.iers tests, though we could also simply run the full astropy test suite.

We should discuss this, but I think we will want to make sure that every time we do a release of astropy-iers-data we should also (and with an automated PR) bump the minimum version of astropy-iers-data in the core package, which will then be equivalent to updating the files here. I also think we might want to use calendar versioning for the astropy-iers-data package since the date is what is most relevant.

Things to do if we want to go ahead:

Move the astropy-iers-data package to the astropy organization
Write simple docs for the astropy-iers-data package (or remove docs/ and populate README)
Decide on automating releases, set up if needed, and do a first release
Make sure that the CI in this PR works fine
Update any docs in the core package

And in a follow-up PR:

Automate updating of minimum version pin of astropy-iers-data in astropy core -- Automate updating of minimum version pin of astropy-iers-data in astropy core #15046

github-actions · 2023-05-12T13:02:47Z

github-actions · 2023-05-12T13:02:52Z

👋 Thank you for your draft pull request! Do you know that you can use [ci skip] or [skip ci] in your commit messages to skip running continuous integration tests until you are ready?

pllim

I like +20 −22,646 ❗

taldcroft · 2023-05-15T10:12:32Z

@astrofrog - excellent, thanks for diving in to this! I am a totally swamped this week but look forward to checking this out next week.

mhvk

It is nice that this looks easy! The one overall question I have is to what extent we use astropy_iers_data in examples; my sense would be to keep it as an implementation detail, i.e., ensure that everything that is needed can be done via astropy.utils.iers. (Right now, this only affects the old-style IERS B readme, which was not really exposed; see in-line comment.)

mhvk · 2023-05-16T09:11:28Z

        ``eopc04_IAU2000.62-now``), do something like this example with an
        excerpt that is used for testing::

+            >>> from astropy_iers_data import IERS_B_IAU2000_README


Seeing this example, I think we should indeed add the readme to astropy_iers_data (it is a TODO right now), but it might also make sense to hide the package from the user here, i.e., do from astropy.utils.iers import IERS_B, IERS_B_IAU2000_README

Agreed on both points.

On second thoughts, I wonder whether, as it is mainly a backward-compatibility thing, whether the IAU2000 file should actually just remain in astropy core, since we don't have to move it and it should never need to be updated?

taldcroft

This looks pretty uncontroversial!

Auto-updating astropy-iers-data at a reasonable cadence with date-based versions seems good. Likewise keeping the astropy requirement synced up also would be a good thing.

taldcroft · 2023-05-24T09:51:29Z

        ``eopc04_IAU2000.62-now``), do something like this example with an
        excerpt that is used for testing::

+            >>> from astropy_iers_data import IERS_B_IAU2000_README


Agreed on both points.

taldcroft · 2023-05-24T10:01:14Z

This also needs a doc update:
https://docs.astropy.org/en/stable/install.html#requirements

pllim · 2023-05-24T17:19:23Z

Before this is merged, we need to do org-level stuff:

Move repo to org
Expand the repo README to provide reason why it exists and where to find more doc about it in astropy doc
Create a new team for that repo
Give correct people the access to repo and PyPI
Where does this fit on the Team page?

astrofrog · 2023-05-25T11:56:24Z

@pllim - the repo now exists: https://github.com/astropy/astropy-iers-data and I have only pushed basic infrastructure to main. I think it should have the same access teams as the core package since it is just a split out.

@taldcroft @mhvk @pllim - please take a look at astropy/astropy-iers-data#1 which is a PR adding the main content to the astropy-iers-data package. It would be good to have this reviewed properly before pushing to main, and there are a few open questions.

pllim · 2023-05-25T17:14:47Z

So who are the official maintainers for this new package? I need to know who to bug when things break there. 😉

Preferably at least 2 people who are responsive.

astrofrog · 2023-06-01T15:43:23Z

@pllim - I'm happy to be one of the maintainers

…nt working directory when using IERS_A.open/read

…tropy.utils.data

Co-authored-by: P. L. Lim <[email protected]>

astrofrog · 2023-07-03T10:23:39Z

Still waiting on conda-forge/staged-recipes#23209 at this point, it could be a little while as some other Python packages were ready for review a few weeks ago but are still waiting

astrofrog · 2023-07-03T10:24:25Z

See #15013 for a companion PR to adjust the auto-update workflow (should be merged after this one)

pllim · 2023-07-03T14:51:34Z

Why is conda-forge so slow? Do they have to do security audit or something? How long you think we have to wait?

olebole · 2023-07-03T16:33:40Z

@mhvk that is fine for me. It would also allow a simple backport of the data package.

astrofrog · 2023-07-07T09:22:06Z

@pllim - the conda-forge package is now available: https://anaconda.org/conda-forge/astropy-iers-data so I think we can proceed with this?

pllim · 2023-07-07T13:31:49Z

conda install astropy-iers-data -c conda-forge is not installing anything for me on Windows with Python 3.11. Maybe wait a few more days just to be sure?

astrofrog · 2023-07-07T13:42:10Z

Huh weird it is there on the conda forge channel - does conda complain about not finding the package?

astrofrog · 2023-07-07T13:43:02Z

See https://anaconda.org/conda-forge/astropy-iers-data/files for the files

pllim · 2023-07-07T13:57:34Z

Let me see... I was letting it run in the background. Took a very long time and now it wants to update a lot of packages. Is this normal? Why is a simple data package so demanding?

The following NEW packages will be INSTALLED:

  astropy-iers-data  conda-forge/noarch::astropy-iers-data-0.2023.7.6.22.3.16-pyhd8ed1ab_0
  libexpat           conda-forge/linux-64::libexpat-2.5.0-hcb278e6_1
  libnsl             conda-forge/linux-64::libnsl-2.0.0-h7f98852_0
  libsqlite          conda-forge/linux-64::libsqlite-3.42.0-h2797004_0
  libzlib            conda-forge/linux-64::libzlib-1.2.13-hd590300_5
  pip                conda-forge/noarch::pip-23.1.2-pyhd8ed1ab_0
  python_abi         conda-forge/linux-64::python_abi-3.11-3_cp311
  typing-extensions  conda-forge/noarch::typing-extensions-4.7.1-hd8ed1ab_0
  typing_extensions  conda-forge/noarch::typing_extensions-4.7.1-pyha770c72_0

The following packages will be UPDATED:

  bzip2                   pkgs/main::bzip2-1.0.8-h7b6447c_0 --> conda-forge::bzip2-1.0.8-h7f98852_4
  ca-certificates    pkgs/main::ca-certificates-2023.01.10~ --> conda-forge::ca-certificates-2023.5.7-hbcca054_0
  cython             pkgs/main::cython-0.29.33-py311h6a678~ --> conda-forge::cython-0.29.36-py311hb755f60_0
  filelock           pkgs/main/linux-64::filelock-3.9.0-py~ --> conda-forge/noarch::filelock-3.12.2-pyhd8ed1ab_0
  identify           pkgs/main/linux-64::identify-2.5.5-py~ --> conda-forge/noarch::identify-2.5.24-pyhd8ed1ab_0
  ld_impl_linux-64   pkgs/main::ld_impl_linux-64-2.38-h118~ --> conda-forge::ld_impl_linux-64-2.40-h41732ed_0
  libgcc-ng          pkgs/main::libgcc-ng-11.2.0-h1234567_1 --> conda-forge::libgcc-ng-13.1.0-he5830b7_0
  libgomp              pkgs/main::libgomp-11.2.0-h1234567_1 --> conda-forge::libgomp-13.1.0-he5830b7_0
  libstdcxx-ng       pkgs/main::libstdcxx-ng-11.2.0-h12345~ --> conda-forge::libstdcxx-ng-13.1.0-hfd8a6a1_0
  libuuid              pkgs/main::libuuid-1.41.5-h5eee18b_0 --> conda-forge::libuuid-2.38.1-h0b41bf4_0
  nodeenv            pkgs/main/linux-64::nodeenv-1.7.0-py3~ --> conda-forge/noarch::nodeenv-1.8.0-pyhd8ed1ab_0
  openssl              pkgs/main::openssl-1.1.1t-h7f8727e_0 --> conda-forge::openssl-3.1.1-hd590300_1
  platformdirs       pkgs/main/linux-64::platformdirs-2.5.~ --> conda-forge/noarch::platformdirs-3.8.0-pyhd8ed1ab_0
  pre-commit         pkgs/main/linux-64::pre-commit-3.2.1-~ --> conda-forge/noarch::pre-commit-3.3.3-pyha770c72_0
  python                pkgs/main::python-3.11.0-h7a1cb2a_2 --> conda-forge::python-3.11.4-hab00c5b_0_cpython
  pyyaml              pkgs/main::pyyaml-6.0-py311h5eee18b_1 --> conda-forge::pyyaml-6.0-py311hd4cff14_5
  readline               pkgs/main::readline-8.2-h5eee18b_0 --> conda-forge::readline-8.2-h8228510_1
  setuptools         pkgs/main/linux-64::setuptools-65.5.0~ --> conda-forge/noarch::setuptools-68.0.0-pyhd8ed1ab_0
  sqlite                pkgs/main::sqlite-3.40.1-h5082296_0 --> conda-forge::sqlite-3.42.0-h2c6b66d_0
  tzdata                 pkgs/main::tzdata-2022g-h04d1e81_0 --> conda-forge::tzdata-2023c-h71feb2d_0
  ukkonen            pkgs/main::ukkonen-1.0.1-py311hdb19cb~ --> conda-forge::ukkonen-1.0.1-py311h4dd048b_3
  virtualenv         pkgs/main/linux-64::virtualenv-20.17.~ --> conda-forge/noarch::virtualenv-20.23.1-pyhd8ed1ab_0
  wheel                pkgs/main::wheel-0.37.1-pyhd3eb1b0_0 --> conda-forge::wheel-0.40.0-pyhd8ed1ab_0
  yaml                     pkgs/main::yaml-0.2.5-h7b6447c_0 --> conda-forge::yaml-0.2.5-h7f98852_2
  zlib                    pkgs/main::zlib-1.2.13-h5eee18b_0 --> conda-forge::zlib-1.2.13-hd590300_5

The following packages will be SUPERSEDED by a higher-priority channel:

  _libgcc_mutex           pkgs/main::_libgcc_mutex-0.1-main --> conda-forge::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex          pkgs/main::_openmp_mutex-5.1-1_gnu --> conda-forge::_openmp_mutex-4.5-2_gnu
  certifi            pkgs/main/linux-64::certifi-2023.5.7-~ --> conda-forge/noarch::certifi-2023.5.7-pyhd8ed1ab_0
  cffi               pkgs/main::cffi-1.15.1-py311h5eee18b_3 --> conda-forge::cffi-1.15.1-py311h409f033_3
  cfgv               pkgs/main/linux-64::cfgv-3.3.1-py311h~ --> conda-forge/noarch::cfgv-3.3.1-pyhd8ed1ab_0
  distlib            pkgs/main/linux-64::distlib-0.3.6-py3~ --> conda-forge/noarch::distlib-0.3.6-pyhd8ed1ab_0
  libffi                 pkgs/main::libffi-3.4.2-h6a678d5_6 --> conda-forge::libffi-3.4.2-h7f98852_5
  ncurses                 pkgs/main::ncurses-6.4-h6a678d5_0 --> conda-forge::ncurses-6.4-hcb278e6_0
  pycparser          pkgs/main::pycparser-2.21-pyhd3eb1b0_0 --> conda-forge::pycparser-2.21-pyhd8ed1ab_0
  tk                        pkgs/main::tk-8.6.12-h1ccaba5_0 --> conda-forge::tk-8.6.12-h27826a3_0
  xz                        pkgs/main::xz-5.2.10-h5eee18b_1 --> conda-forge::xz-5.2.6-h166bdaf_0

Proceed ([y]/n)?

pllim · 2023-07-07T21:29:10Z

On a clean conda env with Python 3.11, it still pulls in other stuff but does not take as long or as many:

  added / updated specs:
    - astropy-iers-data


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    astropy-iers-data-0.2023.7.6.22.3.16|     pyhd8ed1ab_0         1.2 MB  conda-forge
    ca-certificates-2023.5.7   |       hbcca054_0         145 KB  conda-forge
    ld_impl_linux-64-2.40      |       h41732ed_0         688 KB  conda-forge
    libexpat-2.5.0             |       hcb278e6_1          76 KB  conda-forge
    libgcc-ng-13.1.0           |       he5830b7_0         758 KB  conda-forge
    libgomp-13.1.0             |       he5830b7_0         409 KB  conda-forge
    libsqlite-3.42.0           |       h2797004_0         809 KB  conda-forge
    libstdcxx-ng-13.1.0        |       hfd8a6a1_0         3.7 MB  conda-forge
    libuuid-2.38.1             |       h0b41bf4_0          33 KB  conda-forge
    libzlib-1.2.13             |       hd590300_5          60 KB  conda-forge
    ncurses-6.4                |       hcb278e6_0         860 KB  conda-forge
    openssl-3.1.1              |       hd590300_1         2.5 MB  conda-forge
    pip-23.1.2                 |     pyhd8ed1ab_0         1.3 MB  conda-forge
    python-3.11.4              |hab00c5b_0_cpython        29.3 MB  conda-forge
    readline-8.2               |       h8228510_1         275 KB  conda-forge
    setuptools-68.0.0          |     pyhd8ed1ab_0         453 KB  conda-forge
    sqlite-3.42.0              |       h2c6b66d_0         799 KB  conda-forge
    tzdata-2023c               |       h71feb2d_0         115 KB  conda-forge
    wheel-0.40.0               |     pyhd8ed1ab_0          54 KB  conda-forge
    zlib-1.2.13                |       hd590300_5          91 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        43.4 MB

The following NEW packages will be INSTALLED:

  astropy-iers-data  conda-forge/noarch::astropy-iers-data-0.2023.7.6.22.3.16-pyhd8ed1ab_0
  libexpat           conda-forge/linux-64::libexpat-2.5.0-hcb278e6_1
  libnsl             conda-forge/linux-64::libnsl-2.0.0-h7f98852_0
  libsqlite          conda-forge/linux-64::libsqlite-3.42.0-h2797004_0
  libzlib            conda-forge/linux-64::libzlib-1.2.13-hd590300_5

The following packages will be UPDATED:

  bzip2                   pkgs/main::bzip2-1.0.8-h7b6447c_0 --> conda-forge::bzip2-1.0.8-h7f98852_4
  ld_impl_linux-64   pkgs/main::ld_impl_linux-64-2.38-h118~ --> conda-forge::ld_impl_linux-64-2.40-h41732ed_0
  libgcc-ng          pkgs/main::libgcc-ng-11.2.0-h1234567_1 --> conda-forge::libgcc-ng-13.1.0-he5830b7_0
  libgomp              pkgs/main::libgomp-11.2.0-h1234567_1 --> conda-forge::libgomp-13.1.0-he5830b7_0
  libstdcxx-ng       pkgs/main::libstdcxx-ng-11.2.0-h12345~ --> conda-forge::libstdcxx-ng-13.1.0-hfd8a6a1_0
  libuuid              pkgs/main::libuuid-1.41.5-h5eee18b_0 --> conda-forge::libuuid-2.38.1-h0b41bf4_0
  openssl               pkgs/main::openssl-3.0.9-h7f8727e_0 --> conda-forge::openssl-3.1.1-hd590300_1
  readline               pkgs/main::readline-8.2-h5eee18b_0 --> conda-forge::readline-8.2-h8228510_1
  setuptools         pkgs/main/linux-64::setuptools-67.8.0~ --> conda-forge/noarch::setuptools-68.0.0-pyhd8ed1ab_0
  sqlite                pkgs/main::sqlite-3.41.2-h5eee18b_0 --> conda-forge::sqlite-3.42.0-h2c6b66d_0
  wheel              pkgs/main/linux-64::wheel-0.38.4-py31~ --> conda-forge/noarch::wheel-0.40.0-pyhd8ed1ab_0
  zlib                    pkgs/main::zlib-1.2.13-h5eee18b_0 --> conda-forge::zlib-1.2.13-hd590300_5

The following packages will be SUPERSEDED by a higher-priority channel:

  _libgcc_mutex           pkgs/main::_libgcc_mutex-0.1-main --> conda-forge::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex          pkgs/main::_openmp_mutex-5.1-1_gnu --> conda-forge::_openmp_mutex-4.5-2_gnu
  ca-certificates    pkgs/main::ca-certificates-2023.05.30~ --> conda-forge::ca-certificates-2023.5.7-hbcca054_0
  libffi                 pkgs/main::libffi-3.4.4-h6a678d5_0 --> conda-forge::libffi-3.4.2-h7f98852_5
  ncurses                 pkgs/main::ncurses-6.4-h6a678d5_0 --> conda-forge::ncurses-6.4-hcb278e6_0
  pip                pkgs/main/linux-64::pip-23.1.2-py311h~ --> conda-forge/noarch::pip-23.1.2-pyhd8ed1ab_0
  python                pkgs/main::python-3.11.4-h955ad1f_0 --> conda-forge::python-3.11.4-hab00c5b_0_cpython
  tk                        pkgs/main::tk-8.6.12-h1ccaba5_0 --> conda-forge::tk-8.6.12-h27826a3_0
  tzdata                 pkgs/main::tzdata-2023c-h04d1e81_0 --> conda-forge::tzdata-2023c-h71feb2d_0
  xz                         pkgs/main::xz-5.4.2-h5eee18b_0 --> conda-forge::xz-5.2.6-h166bdaf_0


Proceed ([y]/n)?

I see nothing amiss in https://github.com/conda-forge/astropy-iers-data-feedstock/blob/main/recipe/meta.yaml so maybe this is just conda black magic I should walk away from. I don't think this behavior blocks the PR.

pllim

Let's do a Friday evening merge on breaking changes! 😉

Thanks, all!

astrofrog added Experimental utils.iers labels May 12, 2023

github-actions Bot added installation testing labels May 12, 2023

astrofrog added the needs-discussion label May 12, 2023

astrofrog mentioned this pull request May 12, 2023

Consider making iers.Conf.auto_max_age longer and bunding IERS-A with astropy #14756

Open

pllim added this to the v6.0 milestone May 12, 2023

pllim reviewed May 12, 2023

View reviewed changes

Comment thread astropy/utils/iers/iers.py

pllim added the whatsnew-needed label May 12, 2023

astrofrog requested review from bsipocz, eteq, mhvk, pllim and taldcroft May 15, 2023 09:21

mhvk reviewed May 16, 2023

View reviewed changes

taldcroft reviewed May 24, 2023

View reviewed changes

astrofrog mentioned this pull request May 24, 2023

Create astropy-iers-data repository astropy/astropy-project#346

Closed

bsipocz removed their request for review May 24, 2023 17:48

pllim mentioned this pull request Jun 1, 2023

Update IERS Earth rotation and leap second tables #14890

Merged

astrofrog force-pushed the iers-package branch from dfd8e82 to 2f4f4a5 Compare June 1, 2023 15:27

taldcroft mentioned this pull request Jun 2, 2023

Remove ReadMe.eopc04_IAU2000 file astropy/astropy-iers-data#2

Merged

taldcroft reviewed Jun 2, 2023

View reviewed changes

Comment thread astropy/utils/iers/iers.py Outdated

astrofrog and others added 10 commits July 3, 2023 11:22

Emit a deprecation warning if finals2000A.all is present in the curre…

839077e

…nt working directory when using IERS_A.open/read

Updated documentation

55ee887

Added what's new entry

b1143a3

Fix spelling

9a98b82

Ensure IERS_A cache is clearer after tests

6a0ed9d

Add deprecation warning for accessing files that been removed with as…

68099de

…tropy.utils.data

Updated test to check for deprecation warning

e533afa

Apply suggestions from code review

e7fadf6

Co-authored-by: P. L. Lim <[email protected]>

Improve changelog wording

9635396

Add pyinstaller hook for astropy-iers-data

8cd4a69

astrofrog force-pushed the iers-package branch from aa71abd to 8cd4a69 Compare July 3, 2023 10:22

pllim approved these changes Jul 7, 2023

View reviewed changes

pllim merged commit 1e90824 into astropy:main Jul 7, 2023

pllim mentioned this pull request Jul 7, 2023

Automate updating of minimum version pin of astropy-iers-data in astropy core #15046

Closed

weaverba137 mentioned this pull request Aug 25, 2023

Move IERS utility function that is HPC-friendly from DESI to astropy #14757

Closed

pllim mentioned this pull request May 28, 2026

IERS-A data emitted IERSDegradedAccuracyWarning in astropy 5.3.3 but throws ValueError in astropy 7 or later #19823

Closed

Uh oh!

Uh oh!

Conversation

astrofrog commented May 12, 2023 • edited by pllim Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented May 12, 2023

Uh oh!

github-actions Bot commented May 12, 2023

Uh oh!

pllim left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

taldcroft commented May 15, 2023

Uh oh!

mhvk left a comment

Choose a reason for hiding this comment

Uh oh!

mhvk May 16, 2023

Choose a reason for hiding this comment

Uh oh!

taldcroft May 24, 2023

Choose a reason for hiding this comment

Uh oh!

astrofrog Jun 1, 2023

Choose a reason for hiding this comment

Uh oh!

taldcroft left a comment

Choose a reason for hiding this comment

Uh oh!

taldcroft May 24, 2023

Choose a reason for hiding this comment

Uh oh!

taldcroft commented May 24, 2023

Uh oh!

pllim commented May 24, 2023

Uh oh!

astrofrog commented May 25, 2023

Uh oh!

pllim commented May 25, 2023

Uh oh!

astrofrog commented Jun 1, 2023

Uh oh!

Uh oh!

astrofrog commented Jul 3, 2023

Uh oh!

astrofrog commented Jul 3, 2023

Uh oh!

pllim commented Jul 3, 2023

Uh oh!

olebole commented Jul 3, 2023

Uh oh!

astrofrog commented Jul 7, 2023

Uh oh!

pllim commented Jul 7, 2023

Uh oh!

astrofrog commented Jul 7, 2023

Uh oh!

astrofrog commented Jul 7, 2023

Uh oh!

pllim commented Jul 7, 2023

Uh oh!

pllim commented Jul 7, 2023

Uh oh!

pllim left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

astrofrog commented May 12, 2023 •

edited by pllim

Loading