ENH: include exclusive values from IERS_B in IERS_Auto (blocked by #16725) - #16070

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

• Do the proposed changes actually accomplish desired goals?
• Do the proposed changes follow the Astropy coding guidelines?
• Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
• Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
• Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
• Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
• Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
• Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
• At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

👋 Thank you for your draft pull request! Do you know that you can use [ci skip] or [skip ci] in your commit messages to skip running continuous integration tests until you are ready?

Relevant tests need --remote-data, so I think we should apply the "Extra CI" label here ?

Not really, --remote-data is in the same job as devdeps. Since devdeps has numpy failures, might have to wait for that to go away first, just to be safe.

Thanks for clarifying this @pllim !

might have to wait for that to go away first, just to be safe.

and just to make sure everyone is on the same page: as of now, the tests I've updated are failing locally, which is part of what I wanted to showcase with this draft. So, indeed, we need devdeps fixed first, but it's still going to fail (or so I hope !) here after that :)

@neutrinoceros - I've touched this code before but I'm hoping to avoid so going forward. (I struggle enough keeping up with time/table/io.ascii!). If @mhvk doesn't pipe in I can have a look.

For now I'll just rebase so we all have access to logs showing the remaining problems.

Hmm I don't understand the masked array failures in devdeps/remote-data job and in RTD.

I'm willing to bet they are related, I'll take another look in the morning.

(I never said which morning 🙈)
I finally took a look. All failures are 100% real and can be reproduced locally. I just didn't see 4/6 of them before I pushed because they are in coordinates and time tests. This PR will be my priority today.

I could fix my branch to pass tests from astropy/time but the remaining failure in astropy.coordinates appears to be real but is not directly caused by my patch. Rather, it's just revealed. See #16116

rebased onto #16120 to confirm that it unblocks this PR

Status update: this is blocked by

• BUG: MaskedNDArray.__array_ufunc__ breaks for erfa.rxp #16116 (fixed in BUG: fix support for gufuncs with more than one core dimension #16120)
• BUG: MaskedNDArray.__array_ufunc__ breaks for erfa.atioq #16123 (no fix at the time of writing)

#16123 is now fixed in #16125, so rebasing on top of it to see if we got them all 🤞🏻

I've been scratching my head for a couple hours over this one and I'm starting to feel that maybe I shot myself in the face by using table.join here instead of manually merging IERS_A and IERS_B. Instead of going in circles here, and since I'll need to rebase soon anyway, I think I'm going to take a step back for now so I can hopefully come back with a fresh perspective in a few days.

I think I'm going to take a step back for now so I can hopefully come back with a fresh perspective in a few days.

I was about to review this PR, but will wait till you say it is ready. I think a manual merge may indeed be better, since ideally we do not have masked columns at all - they'd make things slower and might cause other results to become Masked instances too. But in the meantime, we solved some bugs in Masked, so not a wasted effort by any means!

#16187 was merged a couple hours ago, so I rebased this. I'll try to get back to it soon.

Status update:
This patch changes the type of some columns in the table returned by IERS_Auto._substitute_iers_b. Namely, this makes some columns masked, which many applications of this table do not seem robust against.
The main pain point is that np.einsum is currently not implemented for MaskedNDArray: it's marked as a TODO, and I think this point is managable, so I'll try to do it in a separate PR then circle back here.

Indeed, would be good if more code could handle Masked without problems! Of course, einsum is somewhat tricky since the mask propagation is not as obvious as it is for other functions. But would be very nice to have it!

Of course, einsum is somewhat tricky since the mask propagation is not as obvious as it is for other functions.

yes, I figured as much, but only after I wrote my previous answer 😅
I'm giving it a try today, though I might need a hand from someone with a stronger algebra background. If I get stuck I'll open an issue for it.

Hi humans 👋 - this pull request hasn't had any new commits for approximately 4 months. I plan to close this in 30 days if the pull request doesn't have any new commits by then.

In lieu of a stalled pull request, please consider closing this and open an issue instead if a reminder is needed to revisit in the future. Maintainers may also choose to add keep-open label to keep this PR open but it is discouraged unless absolutely necessary.

If this PR still needs to be reviewed, as an author, you can rebase it to reset the clock.

If you believe I commented on this pull request incorrectly, please report this here.

I still have hope to finish this some day, but it's still blocked. Let me rebase the branch for good measure.

Hi humans 👋 - this pull request hasn't had any new commits for approximately 4 months. I plan to close this in 30 days if the pull request doesn't have any new commits by then.

In lieu of a stalled pull request, please consider closing this and open an issue instead if a reminder is needed to revisit in the future. Maintainers may also choose to add keep-open label to keep this PR open but it is discouraged unless absolutely necessary.

If this PR still needs to be reviewed, as an author, you can rebase it to reset the clock.

If you believe I commented on this pull request incorrectly, please report this here.

@neutrinoceros - I wonder for this PR if we shouldn't just go the by-hand route and ensure the relevant columns are not masked...