Implement auto-downloading of IERS-A data - #4436

cc: @embray @eteq

And of course this needs to smoothly handle the case where the network turns out to be unavailable. First, use a timeout for the initial download of IERS-A. If this times out, then fall through to the standard IERS-B with a warning. (Note that in this case any extrapolation will generate an exception). Likewise if the currently cached IERS-A is too old and then the subsequent download of IERS-A fails, also fall through to IERS-B with a warning (which would also say how old the current file is).

I still feel like if someone wanted to tackle #3961 that would make this mostly moot, but this is simpler for now...

I still feel like if someone wanted to tackle #3961 that would make this mostly moot, but this is simpler for now...

The caching is the easy part of this, and if someone does #3961 then that will be quickly inserted in here.

The hard part (conceptually) is plugging a new default behavior deep into Time and Coordinates without screwing things up, while allowing for back-compatibility.

One more thing is that it wouldn't be too hard to have "on-demand" downloading which is triggered by actually trying to do a conversion that is in a time range where newer data would help. The exact algorithm for this would depend on details of the IERS bulletin files. This tighter coupling is not ideal, but on the whole probably worth doing to prevent unnecessary network access.

@mhvk @eteq - I have done a bunch of work on the auto-download of IERS data and I think this is getting very close. This PR is based on #4810 because it won't work without that.

I know this is not going to be entirely straightforward to review, but please plan to carve out some time in advance of the code freeze. Some of the logic behind the behavior is a bit intricate, but I've tried to be fairly verbose in the code comments.

Some known issues:

• Lots of coordinates fails due to trying to connect to the internet. We need to think about this.
• I was getting one doc build warning that I did not understand.
• The doc section on Auto-refresh behavior is not finished.

OK, now something strange has come up. It looks like the IERS-B values that are supplied in the finals2000A.all file (http://maia.usno.navy.mil/ser7/finals2000A.all) do not match corresponding values from the actual IERS-B file (http://hpiers.obspm.fr/iers/eop/eopc04/eopc04_IAU2000.62-now). For instance in the table below columns 2 and 3 are from IERS-B, while column 4 (UT1_UTC_B_finals) is from the UT1_UTC_B column of the IERS-A file. The last column delta is the difference. As you can see the delta value can be up to 5 times larger than then stated error on UT1-UTC, i.e. 18 microsec vs. 3.4 microsec.

In [41]: iers_b_ok
Out[41]: 
<IERS_B length=7>
  MJD    UT1_UTC  e_UT1_UTC UT1_UTC_B_finals       delta       
   d        s         s            s                 s         
float64  float64   float64      float64           float64      
------- --------- --------- ---------------- ------------------
53747.0 0.3335044  1.03e-05         0.333506 -1.60000000005e-06
53748.0 0.3337777   3.4e-06         0.333768            9.7e-06
53749.0 0.3339995   6.3e-06         0.334007 -7.49999999999e-06
53750.0 0.3340951   7.4e-06         0.334111          -1.59e-05
53751.0 0.3340757   5.5e-06         0.334096          -2.03e-05
53752.0 0.3339359   3.4e-06         0.333954          -1.81e-05
53753.0 0.3336386   1.4e-06         0.333636  2.60000000002e-06

This came up in regression testing.

724   >>> import astropy.units as u
725   >>> t = Time('2006-01-15 21:24:37.5', format='iso', scale='utc',
726   ...          location=(-155.933222*u.deg, 19.48125*u.deg), precision=6)
727   >>> t.utc.iso
728   '2006-01-15 21:24:37.500000'
729   >>> t.ut1.iso
Expected:
    '2006-01-15 21:24:37.834078'
Got:
    '2006-01-15 21:24:37.834098'

Yikes, that is not good at all! Though now that you mention it, I vaguely recall that I saw that the two columns were not identical before (it is already strange that they have different number of significant digits, though the errors you find are clearly larger than this). There may be errors in initial versions of IERS B that are corrected but not propagated to IERS A. Sigh.

So who is right? It's easy enough to use the actual available IERS-B values (that are bundled with astropy) in IERS_Auto.

IERS B is supposed to have final values.

@mhvk - see 09c02a6 for a patch. Ugh. (Skipped tests because a doctest now appears to be wanting remote data even though this addition should not have affected that. So something else to track down...).

@taldcroft - I've started looking at this. Overall, it looks good, though I want to think a bit more whether it isn't possible to make more minimal changes to IERS itself, keeping additions mostly within the IERS_Auto class.

One annoyance: now that IERS_A turns out not to have correct IERS_B values, shouldn't we auto-update IERS_B too?!

OK, I have some inkling of what's going on here: if you look at http://datacenter.iers.org/web/guest/eop you can see the original sources of the IERS_B files. There you'll see the curious fact that the official IERS B bulletin and the one that eopc04_IAU2000.62-now comes from also don't agree (although well within the errors). This is probably a smoothing effect because the whole thing comes from fitting models that depend on the data, so the small variations are probably due to wiggling polynomials and the like. I suspect the same holds for UT1-UTC.

The finals2000A file says it's updated weekly. But the bulletin B is updated on different periods and the "real" underlying bulletin A is updated at yet a different time. So my hunch is that sometimes they're not consistent because one has not been updated when the other has, and there's some wiggle room.

That's a long way of saying: the B bulletin is probably always more right. It's also unclear where/when finals2000A gets their B values anyway, so what's done here is probably the best choice.

Oh, and one related note (not particularly critical here but could be fixed at the same time): @mhvk, is there a specific reason for getting IERS B from http://hpiers.obspm.fr/iers/eop/eopc04/eopc04_IAU2000.62-now instead of http://datacenter.iers.org/eop/-/somos/5Rgv/latest/214 ? The latter seem more "official" but hard to know...

You might see something similar to #4896 with --remote-data.

@taldcroft - I think not extrapolating is better (or setting to 0 -- it really doesn't matter and we do warn!). And yes, let's make time_now private.

I think setting to 0 is bad because it makes a sudden discontinuity in just-off-the-edge. But I'm fine either way with extrapolating or not (i.e., +0 to leaving it like current master).

Extrapolating over long time scales would be bad (just imagine PM_x at JD=10000), and using the last value is closest for small extrapolations:

Good point re: long time scales. I'm convinced (just make sure to add a quick note about that in API changes, is this is a small but possibly important change for anyone who actually cares about it)

@eteq @mhvk - I think I have addressed all the comments in the last 6 commits. I have to step away for dinner but will rebase and squash a bit after that.

@taldcroft - still needs a changelog entry (including my suggestion of an "API change" mention for the changed extrapolation), but maybe you meant to do that post-rebase. With that, looks pretty solid!

@eteq - yes, I had forgotten about the changelog, but compensated now by putting in 4 separate entries!

Rebased version pushed. The commit in 274a10a might be controversial. I discovered that the hor2eq AltAz comparisons were failing again with --remote-data, and badly for the no-atmosphere case. So I actually just ran hor2eq on our CfA network (fortunately have IDL and it just ran out of the box) for a more reasonable date, namely 2000-01-01, where definitive Earth rotation values are available. These did not match at the ~5 arcsec level. At this time I noticed that if I set the time scale of obstime to UT1 then the original tests (< 5 arcsec and < 0.4 arcsec) were passing for both --remote-data and not.

So I can't say exactly what is going on here (just too long working on this), except that I think that previously the tests were "accidentally" passing by virtue of effectively zeroing out delta_ut1_utc for the 2041 test date.

Huh, strange about the hor2eq business. But I wouldn't worry too much - I noticed this in the "todo" section of hor2eq:

Difference between UTC and UT1 in determining LAST -- is this important?****

So my hunch is that hor2eq is implicitly assuming UTC=UT1, and that's why it worked in the 2041 case, just as you said. So I think what you've done here is just fine.

If the tests pass, I say we merge!

🎆

backported to 1.2.x in 7048693