Thanks to visit codestin.com
Credit goes to github.com

Skip to content

BUG: fix __setitem__ for invalid dtype combinations #157

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jun 20, 2025
Merged

BUG: fix __setitem__ for invalid dtype combinations #157

merged 1 commit into from
Jun 20, 2025

Conversation

ev-br
Copy link
Member

@ev-br ev-br commented Jun 16, 2025 •
edited
Loading

Do not allow e.g.

>>> a = xp.asarray([1, 2, 3])
>>> a[0] = 3.5     # cannot type-promote a float scalar and an int array
>>> a[0] = xp.asarray(3.5)   # cannot integer and float arrays cannot be promoted together

intends to close #136

TODO:

  • add tests
  • check the downstream effect

@ev-br
Copy link
Member Author

ev-br commented Jun 16, 2025

with this branch:

$ SCIPY_ARRAY_API=1 pytest sklearn -v -n 10
...
========================================================= 33215 passed, 5622 skipped, 158 xfailed, 64 xpassed, 3979 warnings in 87.79s (0:01:27) ==========================================================

and

$ spin test -b array_api_strict -j10
...
FAILED scipy/ndimage/tests/test_filters.py::TestVectorizedFilter::test_dtype_batch_memory[array_api_strict-False-48-uint8] - ValueError: mismatched dtypes: self.dtype = array_api_strict.uint8 and other.dtype = array_api_strict.uint64
FAILED scipy/ndimage/tests/test_filters.py::TestVectorizedFilter::test_dtype_batch_memory[array_api_strict-False-48-uint16] - ValueError: mismatched dtypes: self.dtype = array_api_strict.uint16 and other.dtype = array_api_strict.uint64
FAILED scipy/ndimage/tests/test_filters.py::TestVectorizedFilter::test_dtype_batch_memory[array_api_strict-False-48-uint32] - ValueError: mismatched dtypes: self.dtype = array_api_strict.uint32 and other.dtype = array_api_strict.uint64
FAILED scipy/ndimage/tests/test_filters.py::TestVectorizedFilter::test_dtype_batch_memory[array_api_strict-False-48-int8] - ValueError: mismatched dtypes: self.dtype = array_api_strict.int8 and other.dtype = array_api_strict.int64
FAILED scipy/ndimage/tests/test_filters.py::TestVectorizedFilter::test_dtype_batch_memory[array_api_strict-False-48-int16] - ValueError: mismatched dtypes: self.dtype = array_api_strict.int16 and other.dtype = array_api_strict.int64
FAILED scipy/ndimage/tests/test_filters.py::TestVectorizedFilter::test_dtype_batch_memory[array_api_strict-False-48-int32] - ValueError: mismatched dtypes: self.dtype = array_api_strict.int32 and other.dtype = array_api_strict.int64
FAILED scipy/ndimage/tests/test_filters.py::TestVectorizedFilter::test_dtype_batch_memory[array_api_strict-False-inf-uint8] - ValueError: mismatched dtypes: self.dtype = array_api_strict.uint8 and other.dtype = array_api_strict.uint64
FAILED scipy/ndimage/tests/test_filters.py::TestVectorizedFilter::test_dtype_batch_memory[array_api_strict-False-inf-uint16] - ValueError: mismatched dtypes: self.dtype = array_api_strict.uint16 and other.dtype = array_api_strict.uint64
FAILED scipy/ndimage/tests/test_filters.py::TestVectorizedFilter::test_dtype_batch_memory[array_api_strict-False-inf-uint32] - ValueError: mismatched dtypes: self.dtype = array_api_strict.uint32 and other.dtype = array_api_strict.uint64
FAILED scipy/ndimage/tests/test_filters.py::TestVectorizedFilter::test_dtype_batch_memory[array_api_strict-False-inf-int8] - ValueError: mismatched dtypes: self.dtype = array_api_strict.int8 and other.dtype = array_api_strict.int64
FAILED scipy/ndimage/tests/test_filters.py::TestVectorizedFilter::test_dtype_batch_memory[array_api_strict-False-inf-int16] - ValueError: mismatched dtypes: self.dtype = array_api_strict.int16 and other.dtype = array_api_strict.int64
FAILED scipy/ndimage/tests/test_filters.py::TestVectorizedFilter::test_dtype_batch_memory[array_api_strict-False-inf-int32] - ValueError: mismatched dtypes: self.dtype = array_api_strict.int32 and other.dtype = array_api_strict.int64
=========================================================== 12 failed, 60147 passed, 12209 skipped, 217 xfailed, 10 xpassed in 86.96s (0:01:26) ===========================================================

Which looks like a scipy bug with an inplace assignment of uint64 into a uint8 array. WDYT @mdhaber?

> /home/ev-br/repos/scipy/build-install/usr/lib/python3.12/site-packages/scipy/ndimage/_filters.py(197)wrapped_function()
-> output[i:i2, ...] = footprinted_function(xp.asarray(view[i:i2]),
(Pdb) l
192  	                # can also look at the shape to support non-scalar elements.)
193  	                temp = footprinted_function(xp.asarray(view[i:i2]), **kwargs)
194  	                output = xp.empty(view.shape[:-n_axes], dtype=temp.dtype)
195  	                output[i:i2, ...] = temp
196  	            else:
197  ->	                output[i:i2, ...] = footprinted_function(xp.asarray(view[i:i2]),
198  	                                                         **kwargs)
199  	        return output
200  	
201  	    return (input, wrapped_function, size, mode, cval, origin,
202  	            working_axes, axes, n_axes, n_batch, xp)
(Pdb) p output.dtype
array_api_strict.uint8
(Pdb) p view.dtype
dtype('uint8')
(Pdb) p kwargs
{'axis': (-1,)}
(Pdb) p footprinted_function(xp.asarray(view[i:i2]), **kwargs).dtype
array_api_strict.uint64
(Pdb) p xp.asarray(view[i:i2]).dtype
array_api_strict.uint8

@ev-br
Copy link
Member Author

ev-br commented Jun 16, 2025

Huh. The scipy issue is actually this:

(Pdb) np.sum(np.arange(3, dtype=np.int8)).dtype
dtype('int64')
(Pdb) np.sum(np.arange(3, dtype=np.uint8)).dtype
dtype('uint64')

which is inherited by array-api-strict:

(Pdb) import array_api_strict as xp
(Pdb) xp.sum(xp.arange(3, dtype=xp.uint8)).dtype
array_api_strict.uint64

And this is actually deliberate, both in numpy (https://numpy.org/doc/2.1/reference/generated/numpy.sum.html) and array-api-strict (https://data-apis.org/array-api/2024.12/API_specification/generated/array_api.sum.html#sum).

So it does look like an issue in vectorized_filter indeed.

ev-br added a commit to ev-br/scipy that referenced this pull request Jun 18, 2025
@ev-br
TST: ndimage: avoid implicit dtype conversions in vectorized_filter t…
672c3fb 
…ests

This tests does this:  `vectorized_filter(input, xp.sum, output)`, with output
being `xp.empty_like(input)`. This is problematic for short integers because
- the output of xp.sum(int8_array) is int64 (the default int dtype), and
- internally, vectorized_filter does roughly,
`output[some_indices] = xp.sum(input[some_indices])`

So what happens here is that the output of the `sum` in the r.h.s. gets silently
downcast to the dtype of the l.h.s. by `__setitem__`.

This kind of sort of works in many array libraries, but is technically unspecified
by the Array API standard. And indeed this starts failing with array-api-strict
in data-apis/array-api-strict#157

Thus, a test only change to make downcasting explicit.
@ev-br ev-br mentioned this pull request Jun 18, 2025
Merged
@ev-br
BUG: fix __setitem__ for invalid dtype combinations
c223e37 
Do not allow e.g.

>>> a = xp.asarray([1, 2, 3])
>>> a[0] = 3.5     # cannot type-promote a float scalar and an int array
>>> a[0] = xp.asarray(3.5)   # cannot integer and float arrays cannot be promoted together
@ev-br ev-br force-pushed the stricter_setitem branch from 9049acc to c223e37 Compare June 20, 2025 07:17
@ev-br ev-br merged commit 51c2924 into data-apis:main Jun 20, 2025
17 checks passed
@ev-br ev-br added this to the 2.5 milestone Jun 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
2.5
Development

Successfully merging this pull request may close these issues.

__setitem__ should disallow invalid type promotions
1 participant
@ev-br