-
-
Notifications
You must be signed in to change notification settings - Fork 10.8k
MAINT: Remove unsafe unions and ABCs from return-annotations #18885
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
0D arrays | ||
~~~~~~~~~ | ||
|
||
During runtime numpy aggressively casts any passed 0D arrays into their | ||
corresponding `~numpy.generic` instance. Until the introduction of shape | ||
typing (see :pep:`646`) it is unfortunately not possible to make the | ||
necessary distinction between 0D and >0D arrays. While thus not strictly | ||
correct, all operations are that can potentially perform a 0D-array -> scalar | ||
cast are currently annotated as exclusively returning an `ndarray`. | ||
|
||
If it is known in advance that an operation _will_ perform a | ||
0D-array -> scalar cast, then one can consider manually remedying the | ||
situation with either `typing.cast` or a ``# type: ignore`` comment. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TLDR: The new return annotations will now just be somewhat inconvenient for 0D arrays, rather than 0D and ND arrays.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also a second question: are there any parts of the numpy documentation that describe 0D arrays and/or numpy's aggressive 0D-to-scalar casting? If so, then it might be useful the place a link here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The casts are scattered through the C code as PyArray_Return
. The policy, such as it is, is documented as
.. c:function:: PyObject* PyArray_Return(PyArrayObject* arr)
This function steals a reference to *arr*.
This function checks to see if *arr* is a 0-dimensional array and,
if so, returns the appropriate array scalar. It should be used
whenever 0-dimensional arrays could be returned to Python.
With the current tests system we cannot reasonably enforce E501 (maximum line length)
E704 (multiple statements on one line (def)) is a style rule not prescribed by PEP8. Furthermore, because it demands a function body it is needlessly inconvenient for static type checking, i.e. situation where this is no function body.
Any idea why pycodestyle is still checking |
Maybe it is picking up |
Thanks Bas. The long lines are no worse than before. |
Running pycodestyle locally with the config file does seem to work, so I suspect (and hope) it's the former. |
closes #18305.
Per the title, this PR removes unsafe unions and abstract baseclasses from the return annotations,
e.g. functions that currently have one of the following patterns.
The Problem
The problem with returning a
Union
(or, almost equivalently, an abstract-ish baseclass such asgeneric
)is that any and all operations performed on a union must be compatible with all of its members.
For example, operations that are exclusive to either
np.float64
andnp.ndarray
are thus not allowedto be executed by
np.float64 | np.ndarray
, unless the union is narrowed down via an explicitisinstance
check a priori.
While returning a union thus adds some a form of simplicity to the annotations (as we don't have to
distinguish between 0D and ND array-likes), the
Union
type is simply not suited for what we're tryingto describe here (xref python/mypy#1693). This would be a different story
for a hypothetical
UnsafeUnion
type, one where operations must be compatible with any memberof the union. Such type does not exist though.
The Solution
The solutions implemented herein fall in either one of the following two categories:
Any
. This is a simple, but non-ideal solution, as it removes any typesafety for objects returned by aforementioned functions. Nevertheless, there is a group of functions
that still needs to-be updated for dtype-support anyway (e.g. those in
np.core.fromnumeric
),so setting their return to
Any
is by no means the worst thing that can happen.A second group of functions is those where the use of
Any
is simply a necasity, as the output type is.For example, determined by the value of string literals (see
np.einsum
). As a silver lining: the einsumproblem in particular does seem like it can be resolved with relative ease via a future mypy plugin.
for the unsafe-union issue; it has been applied to the more recently annotated modules such as
np.lib.ufunclike
.There is however one important caveat here: as we currently lack shape-support (Typing support for shapes #16544) it is
currently impossible to distinguish between 0D and ND ndarrays, and thus we are unable to describe
the 0D-to-scalar casting that numpy aggressively performs on 0D arrays. While this should change
once PEP 646 is live, in the mean time users will have to settle for a
typing.cast
call or a# type: ignore
comment if it is known in advance that 0D-to-scalar cast will be performed.