BUG: view with fieldless dtype should raise if itemsize != 0 #14393

ahaldane · 2019-08-28T16:55:42Z

Minor bugfix caused by treating the "dtype([])" structured type as if it was the "V0" flexible type.

charris · 2019-08-28T17:36:16Z

Does this need a backport?

ahaldane · 2019-08-28T17:39:34Z

This bug is so niche that a backport doesn't really seem worth the effort to me, but it would also be harmless to backport.

numpy/core/src/multiarray/getset.c

Fixes numpy#14344

eric-wieser · 2019-08-29T05:20:54Z

I think this now warrants a few more tests, since this fixed a lot of bugs.
Here's another that I think is fixed

>>> np.dtype((np.dtype([]), 10))
dtype({'names':[], 'formats':[], 'offsets':[], 'itemsize':10})  # before
dtype(([], (10,)))  # after

And another

>>> np.fromiter((() for i in range(10)), [])
ValueError: Must specify length when using variable-size data-type.  # before
array(...)  # after

eric-wieser · 2019-08-29T05:25:13Z

Probably worth a ..versionchanged in doc/source/reference/c-api/array.rst too, where this macro is documented.

eric-wieser · 2019-08-31T19:01:44Z

numpy/core/src/multiarray/arrayobject.c

        if (res == NULL && !PyErr_Occurred()) {
-            PyErr_SetString(PyExc_ValueError, "No fields found.");
+            /* these dtypes had no fields */
+            return cmp_op == Py_EQ ? Py_True : Py_False;


This change isn't correct - it doesn't produce a correctly broadcast array. Let's leave this to a future PR - the rest looks great.

Fixed by falling through to the string compare case which does the right thing here.

Edit: ~~Oh wait, I forgot about itemsize differences..~~ Never mind, it does work because V0 and V8 empty strings are equal.

ahaldane · 2019-08-31T19:04:15Z

Thanks for great reviews, as usual.

I updated PyDataType_ISUNSIZED, and went through all the places it was used to check if this introduced any bugs. It did, which I fixed, and I added appropriate tests.

I have some hesitation on doing it this way, so consider this a WIP. First, in some of the places calling ISUNSIZED the code really did want to check just elsize == 0, fields or no fields. Also, with the same logic it seems like PyDataType_ISFLEXIBLE should also be False for structured 0-size types. However, when I make that change I get a bunch of segfaults, especially in the pickling code. .

eric-wieser · 2019-08-31T19:42:25Z

numpy/core/src/multiarray/arrayobject.c

What about fieldless types with padding?

eric-wieser · 2019-08-31T22:06:43Z

numpy/core/src/multiarray/arrayobject.c

        if (res == NULL && !PyErr_Occurred()) {
-            PyErr_SetString(PyExc_ValueError, "No fields found.");
+            /* these dtypes had no fields */
+            res = PyArray_NewLikeArray(self, NPY_ANYORDER,


This doesn't broadcast with other, unless the broadcasting has already happened here

Ah, yeah. I think I've got it finally by using a multiiter.

eric-wieser · 2019-09-04T06:14:19Z

numpy/core/src/multiarray/arrayobject.c

Fixes gh-13438, I think

eric-wieser · 2019-09-04T06:17:40Z

numpy/core/src/multiarray/methods.c

It's not clear to me why this condition is needed at all, although it admittedly looks harmless.

not clear to me either. These lines previously discussed here: https://github.com/numpy/numpy/pull/12748/files#r249258808 and introduced in #12011, but not discussed there.

doc/source/reference/c-api/array.rst

eric-wieser

Looks pretty good - one unimportant question, and a doc typo. Getting the comparison fix in here should have enabled me to push some of my remaining fixes, which were untestable without it - thanks!

eric-wieser · 2019-09-04T06:22:41Z

I think an upcoming_changes/14393.c_api.rst note might be warranted about the macro behavior

charris · 2019-09-05T20:56:05Z

@ahaldane Ping.

ahaldane · 2019-09-05T20:57:14Z

About to push as you wrote that!

Fixes numpy#13438

mattip · 2019-09-08T15:06:08Z

Thanks @ahaldane and reviewers

ahaldane added 00 - Bug component: numpy._core labels Aug 28, 2019

eric-wieser reviewed Aug 28, 2019

View reviewed changes

numpy/core/src/multiarray/getset.c Outdated Show resolved Hide resolved

BUG: view with fieldless dtype should raise if itemsize != 0

847aefc

Fixes numpy#14344

ahaldane force-pushed the fix_fieldless_view_itemsize branch from d4ddfaa to 847aefc Compare August 28, 2019 17:54

BUG: PyDataType_ISUNSIZED should be False for structured types

9b740ef

eric-wieser reviewed Aug 31, 2019

View reviewed changes

numpy/core/src/multiarray/arrayobject.c Outdated

Copy link

Member

eric-wieser Aug 31, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about fieldless types with padding?

ahaldane force-pushed the fix_fieldless_view_itemsize branch 2 times, most recently from c1d3f73 to c2b2c00 Compare August 31, 2019 20:09

BUG: Fix comparison of fieldless structured arrays

7c3adfb

ahaldane force-pushed the fix_fieldless_view_itemsize branch from c2b2c00 to 7c3adfb Compare August 31, 2019 21:17

eric-wieser reviewed Aug 31, 2019

View reviewed changes

eric-wieser reviewed Sep 4, 2019

View reviewed changes

numpy/core/src/multiarray/arrayobject.c Outdated

Copy link

Member

eric-wieser Sep 4, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixes gh-13438, I think

eric-wieser reviewed Sep 4, 2019

View reviewed changes

doc/source/reference/c-api/array.rst Outdated Show resolved Hide resolved

eric-wieser approved these changes Sep 4, 2019

View reviewed changes

ahaldane force-pushed the fix_fieldless_view_itemsize branch from d33c5d2 to 5e7bca5 Compare September 5, 2019 20:57

BUG: Fix fieldless comparison broadcasting

d6f7524

Fixes numpy#13438

ahaldane force-pushed the fix_fieldless_view_itemsize branch from 5e7bca5 to d6f7524 Compare September 5, 2019 22:07

mattip merged commit f786041 into numpy:master Sep 8, 2019

mattip mentioned this pull request Sep 8, 2019

Unable to test equality of structured dtypes with no fields #13438

Closed

Uh oh!

BUG: view with fieldless dtype should raise if itemsize != 0 #14393

BUG: view with fieldless dtype should raise if itemsize != 0 #14393

Uh oh!

Conversation

ahaldane commented Aug 28, 2019

Uh oh!

charris commented Aug 28, 2019

Uh oh!

ahaldane commented Aug 28, 2019

Uh oh!

Uh oh!

eric-wieser commented Aug 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eric-wieser commented Aug 29, 2019

Uh oh!

eric-wieser Aug 31, 2019

Choose a reason for hiding this comment

Uh oh!

ahaldane Aug 31, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahaldane commented Aug 31, 2019

Uh oh!

eric-wieser Aug 31, 2019

Choose a reason for hiding this comment

Uh oh!

eric-wieser Aug 31, 2019

Choose a reason for hiding this comment

Uh oh!

ahaldane Sep 1, 2019

Choose a reason for hiding this comment

Uh oh!

eric-wieser Sep 4, 2019

Choose a reason for hiding this comment

Uh oh!

eric-wieser Sep 4, 2019

Choose a reason for hiding this comment

Uh oh!

ahaldane Sep 5, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

eric-wieser left a comment

Choose a reason for hiding this comment

Uh oh!

eric-wieser commented Sep 4, 2019

Uh oh!

charris commented Sep 5, 2019

Uh oh!

ahaldane commented Sep 5, 2019

Uh oh!

mattip commented Sep 8, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

eric-wieser commented Aug 29, 2019 •

edited

Loading

ahaldane Aug 31, 2019 •

edited

Loading