BUG: Handle subarrays in descr_to_dtype #13433

mattip · 2019-04-30T11:01:50Z

There are alternative spellings of dtype=[('c', '<f8', (2, 5))], handle the dtype=[('c', ('<f8', (5,)), (2,))] variant.

charris · 2019-04-30T14:26:44Z

The test failure was the matmul heisenbug. Restarted test.

seberg

Strange beast, nested subfields... But they do seem to work fine for all things (including being resolved at arbitrary depth when there are no fields left).

Anyway, LGTM, will merge soon.

eric-wieser · 2019-05-01T16:04:23Z

numpy/lib/format.py


    This function reverses the process, eliminating the empty padding fields.
    '''
-    if isinstance(descr, (str, dict)):
+    if isinstance(descr, (str, dict, tuple)):
        # No padding removal needed


What happens if this is a subarray of structured types?

s = np.dtype([('a', np.int8), ('b', np.int16), ('c', np.int32)], align=True) s_sub = np.dtype((s, (3,)))

I think you need to recurse for subarray types

That is an interesting case. Top level subarrays are degenerated on arrays (they are added to the dimensions of the array), cannot quickly find a way to create an array with such a dtype, but it somewhat feels like there may have been strange ways to do it.

s = np.dtype([('a', np.int8), ('b', np.int16), ('c', np.int32)], align=True) s_sub = np.dtype((s, (1,1))) arr = np.zeros(3, s_sub) print(arr.shape, arr.dtype) arr = np.ndarray(shape=3, buffer=arr, dtype=s_sub) print(arr.shape, arr.dtype)

Also, watch out for structured types like (int, [('fields', int)]) which have a non-void base

Yeah, this one is still broken (although maybe the original issue is solved and this is just another issue). Had a too shallow look at this probably, though :/.

No need for the subarray to be at the top level to hit this code-path - nest it inside a structured one.

eric-wieser

Trying to unstick my pending comment...

eric-wieser · 2019-05-02T06:50:25Z

Here's the case that this handles incorrectly:

dt = np.dtype([
    ('x', np.dtype((
        np.dtype((
            np.dtype({'names':['a','b'], 'formats':['i1','i1'], 'offsets':[0,4], 'itemsize':8}),
            (3,)
        )),
        (4,)
    )))
])
assert descr_to_dtype(dt.descr) == dt

mattip · 2019-05-02T13:33:08Z

I think we should fail to create a dtype with (int, [('fields', int)]). It does not seem to fit into any of the categories of dtypes we should parse.

In any case, its descr attribute does not provide the information to reconstruct it, so lib.format.dtype_to_descr fails. If needed, let's handle that in a different issue/PR

eric-wieser · 2019-05-02T15:46:48Z

Agreed that the non-void struct is not important. We should still support arbitrarily nested subarrays though.

mattip · 2019-05-02T15:52:07Z

I think the last commit fixed parsing nested subarrays, at least the tests with the new dtypes pass.

eric-wieser · 2019-05-02T15:45:47Z

numpy/lib/tests/test_format.py

+    np.dtype([('x', ([('a', '|i1'),
+                      ('', '|V3'),
+                      ('b', '|i1'),
+                      ('', '|V3'),


Why are you inserting these empty fields? The point of my example was that your code fails when there is unnamed padding here (fails by creating new fields, which this function's purpose is to avoid)

To hit the failing path, you need to use np.dtype({'names':['a','b'], 'formats':['i1','i1'], 'offsets':[0,4], 'itemsize':8}) as the inner type here, not [('a', '|i1'), ('', '|V3'), ('b', '|i1'), ('', '|V3')]

that passes too

The full type I use in a comment above still fails

Could you provide a complete example of a dtype that fails to roundtrip?

test added and fixed

eric-wieser · 2019-05-02T20:09:02Z

numpy/lib/tests/test_format.py

+                              'offsets':[0,4],
+                              'itemsize':8,
+                             },
+                    (3,)),


I don't think dtype(dict, tuple) is legal, which will cause an error during test collection

tests are passing

>>> np.dtype(int, "this argument is ignored") dtype('int32')

This test is ignoring the (3,) silently, which is a different bug.

mattip · 2019-05-05T23:42:46Z

I don't think the dict path can ever be hit

It seems not, removing

seberg · 2019-05-11T16:56:09Z

This stuff still confuses me a bit, but it does seem the test should cover the interesting corner cases, so can probably merge.

seberg · 2019-05-11T17:56:45Z

numpy/lib/tests/test_format.py

+            np.dtype((
+                np.dtype((
+                    np.dtype([
+                        ('a', int)


Suggested change

('a', int)

('a', int),

('b', np.dtype({'names':['a','b'],

'formats':['i1','i1'],

'offsets':[0,4],

'itemsize':8})),

Finally, this is what will make things fail...

seberg · 2019-05-11T18:07:32Z

numpy/lib/format.py

+            # subtype, will always have a shape descr[1]
+            dt = descr_to_dtype(descr[0])
+            return numpy.dtype((dt, descr[1]))
+        return numpy.dtype(descr)


Suggested change

return numpy.dtype(descr)

return np.dtype(descr_to_dtype(descr[0]), descr[1])

Is that assert correct here, since it is not a list around it, there cannot be a field name, so it must have two entries, right? (should probably not leave the assert, or doe sit get stripped on install?)

Almost looks correct, but this should be np.dtype((descr_to_dtype(descr[0]), descr[1]))

numpy/lib/format.py

charris · 2019-05-12T03:08:57Z

close/reopen

seberg · 2019-05-12T07:57:51Z

Ok, putting this in then. What I am not quite sure is whether there is some issue that should be opened here, may come back to it, but it will be a fringe issue in any case, I suppose.

BUG: handle subarrays in descr_to_dtype

666d92a

mattip added 00 - Bug 09 - Backport-Candidate PRs tagged should be backported component: numpy.dtype labels Apr 30, 2019

seberg added this to the 1.16.4 release milestone Apr 30, 2019

seberg added 06 - Regression and removed 06 - Regression labels Apr 30, 2019

seberg self-assigned this Apr 30, 2019

seberg approved these changes Apr 30, 2019

View reviewed changes

eric-wieser reviewed May 1, 2019

View reviewed changes

seberg self-requested a review May 1, 2019 16:54

eric-wieser reviewed May 2, 2019

View reviewed changes

mattip force-pushed the issue13431 branch from f8f15d8 to f0910d4 Compare May 2, 2019 18:00

eric-wieser reviewed May 2, 2019

View reviewed changes

BUG: parse more subarrays in descr_to_dtype

b90addd

mattip force-pushed the issue13431 branch from f0910d4 to b90addd Compare May 3, 2019 06:31

charris changed the title ~~BUG: handle subarrays in descr_to_dtype~~ BUG: Handle subarrays in descr_to_dtype May 11, 2019

seberg reviewed May 11, 2019

View reviewed changes

numpy/lib/format.py Outdated Show resolved Hide resolved

MAINT: remove uneeded code

bd73a15

mattip force-pushed the issue13431 branch from 697b1e4 to bd73a15 Compare May 11, 2019 23:37

charris closed this May 12, 2019

charris reopened this May 12, 2019

seberg merged commit e6227a0 into numpy:master May 12, 2019

charris mentioned this pull request May 14, 2019

BUG: Handle subarrays in descr_to_dtype #13561

Merged

charris removed the 09 - Backport-Candidate PRs tagged should be backported label May 14, 2019

charris removed this from the 1.16.4 release milestone May 14, 2019

mattip deleted the issue13431 branch June 8, 2020 06:58

	return numpy.dtype(descr)
	return np.dtype(descr_to_dtype(descr[0]), descr[1])

Uh oh!

BUG: Handle subarrays in descr_to_dtype #13433

BUG: Handle subarrays in descr_to_dtype #13433

Uh oh!

Conversation

mattip commented Apr 30, 2019

Uh oh!

charris commented Apr 30, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seberg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seberg May 1, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eric-wieser left a comment

Choose a reason for hiding this comment

Uh oh!

eric-wieser commented May 2, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattip commented May 2, 2019

Uh oh!

eric-wieser commented May 2, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattip commented May 2, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattip commented May 5, 2019

Uh oh!

seberg commented May 11, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seberg May 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

charris commented May 12, 2019

charris commented Apr 30, 2019 •

edited

Loading

seberg May 1, 2019 •

edited

Loading

eric-wieser commented May 2, 2019 •

edited

Loading

eric-wieser commented May 2, 2019 •

edited

Loading

seberg May 11, 2019 •

edited

Loading