BUG,TST: Fix condition for whether long double is “IBM double double”.#31375
Conversation
7a76cdc to
ed5f56c
Compare
Some distributions changed the default for long double on 64-bit PowerPC (at least little-endian) from the “IBM double double” format to the IEEE 754 binary128 format. The implementation already handles that by detecting the format at build time (longdouble_format property). Before this change, the tests checked only the CPU architecture name, possibly falsely assuming that long double has the “IBM double double” format when it actually has the IEEE 754 binary128 format. After this change, the tests assume the “IBM double double” format only if the maximum value, determined by calling numpy.finfo(), is the maximum value characteristic for the “IBM double double” format. On CPU architectures other than PowerPC, the “IBM double double” format is never assumed. I found no evidence that any other CPU architecture ever used that format. On a system where long double has the “IBM double double” format, the test results don’t change: 47406 passed, 1047 skipped, 2845 deselected, 32 xfailed, 4 xpassed On a system where long double has the IEEE 754 binary128 format, the test results change from: 1 failed, 47404 passed, 1049 skipped, 2845 deselected, 32 xfailed, 4 xpassed to: 47408 passed, 1048 skipped, 2845 deselected, 32 xfailed, 2 xpassed Fixes numpy#21094.
ed5f56c to
eba6eab
Compare
seberg
left a comment
There was a problem hiding this comment.
Makes sense to me, I would prefer to tweak the definition slightly. If I have a good idea, I might also just apply that.
|
|
||
| LONG_DOUBLE_IS_IBM_DOUBLE_DOUBLE = (platform.machine().startswith("ppc") | ||
| and str(np.finfo(np.longdouble).max) | ||
| == "1.79769313486231580793728971405301e+308") |
There was a problem hiding this comment.
Would it make sense to just drop the PPC entirely? I would also slightly prefer to nmant (or even also nexp), not quite as distinct in theory, but then we don't have to trust string formatters (I don't really trust them, although I guess that one test would maybe fail if they failed)...
(A bit pattern would work as well, but one special value or two seems distinct enough, I would agree.)
There was a problem hiding this comment.
I think it should not make any difference to check for PPC or not, at least with the current way (comparing with max).
I would have expected that the string formatting code is stable/correct enough for this purpose. However, I can understand if experience suggests that it might not.
Comparing nmant and nexp would work in practice, however it doesn’t capture the characteristic fact that it is an addition of two numbers. (By the way, in general it doesn’t really make sense to specify nmant for this data type in the same sense as done for others, although I understand that it is needed for API compatibility.)
Comparing with a byte or bit pattern is a bit fiddly, as they differ on little-endian and big-endian (the order of the two numbers is the same on both, but the individual numbers have different order).
What would work is np.finfo(np.longdouble).max.as_integer_ratio() == (0x3ffffffffffffefffffffffffff << 918, 1), which is both characteristic for this data type (see the zero bit in the middle of the mantissa) and avoids string formatting.
There was a problem hiding this comment.
Sure, the as_integer_ratio seems like it should work for sure, in practice I guess also finfo(longdouble).max - finfo(double).max == something would probably also work. (I agree nmant at least is a bit shady, although I think the compiler and NumPy do define it to the size of the normalized mantissa.)
Another fair way is to use the bit pattern for something like np.longdouble(1)/10.
However, I can understand if experience suggests that it might not.
Hmmm, it might actually be completely fine and the problem was just (maybe musl) parsing strings, not printing (which we have our own code for). But either way, avoiding it seems nice.
Anyway, if you do a quick follow-up that's great, wotherwise I am happy to just put it in as is, it's a clear improvement!
There was a problem hiding this comment.
Actually we are hardcoding eps right now to float.fromhex("1p-105") in C... (The idea was to use the epsilon after normalizing the two doubles to have no gap in their mantissa bits).
So, I would think that should be good enough in practice (sure, in theory someone can create a non IEEE quad-precision that has the same mantissa, but...).
There was a problem hiding this comment.
Sorry, I missed your previous comment.
I’d prefer to compare with the maximum value, as that is unambiguously defined (whereas mantissa bits and epsilon is not quite as clear, and bit patterns differ depending on endianness). I don’t have a strong opinion on whether comparing the integer ratio or the string representation is better.
What do you think?
There was a problem hiding this comment.
yeah, sure. Just that integer ratio seems like a too large integer to work in practice :)
There was a problem hiding this comment.
Too large for what or whom? I’m not afraid of large numbers. :)
There was a problem hiding this comment.
My local python was, although, I guess it might need 80bit longdouble to hit that.
|
The Circleci error is unrelated, and should be fixed in main. |
|
OK, lets just put this in. We do the string formatting, so that part should be fine and the only assumption this bakes in is that it this can only happen on PPC; that seems a bit unnecessary to me but it was what we had before. Thanks @manueljacob. |
Some distributions changed the default for long double on 64-bit PowerPC (at least little-endian) from the “IBM double double” format to the IEEE 754 binary128 format. The implementation already handles that by detecting the format at build time (longdouble_format property). Before this change, the tests checked only the CPU architecture name, possibly falsely assuming that long double has the “IBM double double” format when it actually has the IEEE 754 binary128 format. After this change, the tests assume the “IBM double double” format only if the maximum value, determined by calling numpy.finfo(), is the maximum value characteristic for the “IBM double double” format. On CPU architectures other than PowerPC, the “IBM double double” format is never assumed. I found no evidence that any other CPU architecture ever used that format.
On a system where long double has the “IBM double double” format, the test results don’t change:
47406 passed, 1047 skipped, 2845 deselected, 32 xfailed, 4 xpassed
On a system where long double has the IEEE 754 binary128 format, the test results change from:
1 failed, 47404 passed, 1049 skipped, 2845 deselected, 32 xfailed, 4 xpassed
to:
47408 passed, 1048 skipped, 2845 deselected, 32 xfailed, 2 xpassed
Fixes #21094.
First time committer introduction
I rarely use NumPy directly. I encountered one of the previous test failures when building the NumPy package in the Nix package collection (which runs the tests).
AI Disclosure
No AI tools used