Thanks to visit codestin.com
Credit goes to github.com

Skip to content

BUG: Handle subarrays in descr_to_dtype #13433

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 12, 2019
Merged

BUG: Handle subarrays in descr_to_dtype #13433

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
666d92a
BUG: handle subarrays in descr_to_dtype
mattip Apr 30, 2019
b90addd
BUG: parse more subarrays in descr_to_dtype
mattip May 2, 2019
bd73a15
MAINT: remove uneeded code
mattip May 5, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions numpy/lib/format.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -261,15 +261,19 @@ def dtype_to_descr(dtype):
def descr_to_dtype(descr):
'''
descr may be stored as dtype.descr, which is a list of
(name, format, [shape]) tuples. Offsets are not explicitly saved, rather
empty fields with name,format == '', '|Vn' are added as padding.
(name, format, [shape]) tuples where format may be a str or a tuple.
Offsets are not explicitly saved, rather empty fields with
name, format == '', '|Vn' are added as padding.

This function reverses the process, eliminating the empty padding fields.
'''
if isinstance(descr, (str, dict)):
if isinstance(descr, str):
# No padding removal needed
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if this is a subarray of structured types?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

s = np.dtype([('a', np.int8), ('b', np.int16), ('c', np.int32)], align=True)
s_sub = np.dtype((s, (3,)))

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you need to recurse for subarray types

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is an interesting case. Top level subarrays are degenerated on arrays (they are added to the dimensions of the array), cannot quickly find a way to create an array with such a dtype, but it somewhat feels like there may have been strange ways to do it.

Copy link
Member

@seberg seberg May 1, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

s = np.dtype([('a', np.int8), ('b', np.int16), ('c', np.int32)], align=True)
s_sub = np.dtype((s, (1,1)))
arr = np.zeros(3, s_sub)
print(arr.shape, arr.dtype)
arr = np.ndarray(shape=3, buffer=arr, dtype=s_sub)
print(arr.shape, arr.dtype)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, watch out for structured types like (int, [('fields', int)]) which have a non-void base

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this one is still broken (although maybe the original issue is solved and this is just another issue). Had a too shallow look at this probably, though :/.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for the subarray to be at the top level to hit this code-path - nest it inside a structured one.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for the subarray to be at the top level to hit this code-path - nest it inside a structured one.

return numpy.dtype(descr)

elif isinstance(descr, tuple):
# subtype, will always have a shape descr[1]
dt = descr_to_dtype(descr[0])
return numpy.dtype((dt, descr[1]))
fields = []
offset = 0
for field in descr:
Expand Down
56 changes: 56 additions & 0 deletions numpy/lib/tests/test_format.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -411,6 +411,7 @@ def teardown_module():
np.array(NbufferT, dtype=np.dtype(Ndescr).newbyteorder('<')),
np.array(PbufferT, dtype=np.dtype(Pdescr).newbyteorder('>')),
np.array(NbufferT, dtype=np.dtype(Ndescr).newbyteorder('>')),
np.zeros(1, dtype=[('c', ('<f8', (5,)), (2,))])
]


Expand Down Expand Up @@ -628,6 +629,61 @@ def test_pickle_disallow():
assert_raises(ValueError, np.save, path, np.array([None], dtype=object),
allow_pickle=False)

@pytest.mark.parametrize('dt', [
np.dtype(np.dtype([('a', np.int8),
('b', np.int16),
('c', np.int32),
], align=True),
(3,)),
np.dtype([('x', np.dtype({'names':['a','b'],
'formats':['i1','i1'],
'offsets':[0,4],
'itemsize':8,
},
(3,)),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think dtype(dict, tuple) is legal, which will cause an error during test collection

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tests are passing

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

>>> np.dtype(int, "this argument is ignored")
dtype('int32')

This test is ignoring the (3,) silently, which is a different bug.

(4,),
)]),
np.dtype([('x',
('<f8', (5,)),
(2,),
)]),
np.dtype([('x', np.dtype((
np.dtype((
np.dtype({'names':['a','b'],
'formats':['i1','i1'],
'offsets':[0,4],
'itemsize':8}),
(3,)
)),
(4,)
)))
]),
np.dtype([
('a', np.dtype((
np.dtype((
np.dtype((
np.dtype([
('a', int),
('b', np.dtype({'names':['a','b'],
'formats':['i1','i1'],
'offsets':[0,4],
'itemsize':8})),
]),
(3,),
)),
(4,),
)),
(5,),
)))
]),
])

def test_descr_to_dtype(dt):
dt1 = format.descr_to_dtype(dt.descr)
assert_equal_(dt1, dt)
arr1 = np.zeros(3, dt)
arr2 = roundtrip(arr1)
assert_array_equal(arr1, arr2)

def test_version_2_0():
f = BytesIO()
Expand Down