@eric-wieser maybe you can give this a quick careful review? We need to put this in within the next few days to get 1.18.4 out this months (its a weird limit due to our rackspace account used in the 1.18.x release process).

Code looks fine. It's hard to tell if this really restores exactly the behavior we care about, but it look like it does at least do that for the behavior people are somehow relying upon.

I assume what matters is this line:

https://github.com/numpy/numpy/pull/15766/files#diff-f3387afa660b73f8f37f0978193799d9L1512

Which encoded this two-step conversion.

Hmmmpf, yeah, there are definitely changes in UnicodeError, but I do not care about those.

Hard to be quite sure that there are not any changes for structured dtypes due to existing inconsistencies in the change that nobody noticed yet. I guess if we want to be on the safe side, there is no way but to revert the whole things. I will try and see if there is anything...

Can you add a test for a structured type containing strings being cast to one containing bools, just to be sure?

Ah, nevermind, str->structured never hit the old code actually since convert is 0 in that case. And I guess string -> number conversion always uses Long, float or complex, and where it did not this was a fix (i.e. longdouble).

The only thing that is a bit strange is datetimes... I cannot find a change in behaviour there, but I did not yet understand why not...

Arrg, no wonder I am confused... We have a _strided_to_strided_string_to_datetime as a second implementation for the datetime casts. So while this one was actually incorrect and accidentally fixed as well. It was never used within NumPy.
So, there is a second very subtle change here, when using datetime/timedelta with direct access of the descr->f->cast functions or through PyArray_CastScalarToCtype, PyArray_GetCastFunc, PyArray_CastScalarDirect, where the cast from string -> datetime/timedelta was accidentally fixed to be in line with NumPy...

@eric-wieser added structured string -> structured bool to the test. I am inclined to ignore that annoying point about the datetime, but we could make a release note about it. Or, I can undo more of the commit for backporting purpose just to make a point of not modifying even the most sublte thing in a point release.

Anyway, I am happy to also just revert if we have any doubts for the backport. As for master, I think this is definitely OK.

@eric-wieser Ping.

@seberg What is the argument for not backporting this?

1. I am missing some weird corner case
2. Manual string to datetime casts, in C-extensions will have their behaviour silently fixed.

I am not overly worried about that cast, it seems like a corner case of a corner case of a corner case. But if we want to be safe nothing changes in the point release, I can revert the change more fully and only keep the changes for floating point numbers (and maybe integers).

Let's play this conservative and not add anything new for the backport besides the fixes we backported in 1.18.3. I'll be branching 1.9 in about two weeks and would like to get 1.18.4 out before rackspace runs out on us.

Oh those changes are already part of that backport, but I can undo more of the backport wihout affecting the initial bufix,

I can undo more of the backport

Let's do that, it is the least risky thing to do.

Note that I am only talking about the backport.

If the easy thing is to just revert the original backport, I'm open to that also.

No its fine, I will do this now on the 1.18.x branch.

OK, see gh-16109 for the maximum possible revert approach (while retaining the original bug fix which only affects the behaviour of longdoubles).

@seberg Ping. Does this need a release note?

Added a release note, not sure if nice. It is a very subtle C-API bug fix in a sense... The datetime thing is the only change I am aware of.

Thanks Sebastian.