DEP: deprecate np.set_numeric_ops and friends #11916

mattip · 2018-09-09T07:29:33Z

The third part of NEP 15. Deprecate np.set_numeric_ops, PyArray_SetNumericOps, PyArray_GetNumericOps.

The second part, "we should use some new, private API to set up ndarray.__add__ and friends" is still TBD.

numpy/core/_add_newdocs.py

mattip · 2018-09-09T10:46:46Z

Codecov failures since newly deprecated PyArray_GetNumericOps is not tested.

numpy/core/src/multiarray/number.c

mattip · 2018-09-17T18:37:02Z

I think codecov is comparing this pr to some other base changeset. The diff is showing changes I did not make to distutils and random.

tylerjereddy · 2018-09-17T18:40:54Z

Your link is for "changes," not the actual diff, which correctly shows which lines in this PR are not covered. I tend to ignore the "changes" tab, as previously discussed elsewhere.

mattip · 2018-09-21T10:19:35Z

rebased and deprecation test added to help code coverage

mattip · 2018-09-22T17:18:48Z

Closed/reopened to restart crashed appveyor tests

numpy/core/src/multiarray/multiarraymodule.c

numpy/core/tests/test_deprecations.py

mattip · 2018-09-22T21:49:35Z

Should squash before merging

mattip · 2018-09-23T04:42:14Z

squashed to a single commit

eric-wieser · 2018-09-23T15:46:55Z

numpy/core/_add_newdocs.py

@@ -1537,6 +1537,10 @@ def luf(lamdaexpr, *args, **kwargs):

    Set numerical operators for array objects.

+    .. deprecated:: 1.16
+
+        Use :c:func:`PyUFunc_ReplaceLoopBySignature` to change ufunc behaviour.


Perhaps we should also suggest creating a subclass with an overloaded __array_ufunc__ here, for python-only users who were replacing operators.

This reads badly. Maybe

For the general case, use :c:func:`PyUFunc_ReplaceLoopBySignature`. For ndarray subclasses, define the ``__array_ufunc__`` method and override the relevant ufunc.

eric-wieser · 2018-09-26T07:41:41Z

This closes #10141, right?

eric-wieser · 2018-09-26T07:42:52Z

It doesn't look like you added a deprecation notice to the C api docs

Needs a C docs change too

mattip · 2018-09-28T13:31:18Z

C API deprecation added. Also added a commit to remove mention of the deprecated set_numeric_ops which closes #10141

mattip · 2018-10-09T10:36:31Z

ping

mattip · 2018-10-12T08:19:14Z

rebased to fix release note merge confilicts

mattip · 2018-10-16T04:59:07Z

ping

doc/release/1.16.0-notes.rst

eric-wieser · 2018-10-16T05:38:24Z

I feel like we have a compile-time mechanism for deprecating C Apis?

At a guess, perhaps @charris or @njsmith would know more.

mattip · 2018-10-16T06:46:50Z

@eric-wieser indeed I did not increment the NPY_API_VERSION for the deprecation. Fixed.

eric-wieser · 2018-10-16T13:14:26Z

numpy/core/src/multiarray/number.c

+  Get dictionary showing number functions that all arrays will use
+*/
+NPY_NO_EXPORT PyObject *
+PyArray_GetNumericOps(void)


Should be wrapped in

#if !defined(NPY_NO_DEPRECATED_API) || \ (NPY_NO_DEPRECATED_API < NPY_1_16_API_VERSION) #endif

That is only for headers, to allow access to old API macros. The C files always start with this define, which will defeat the check you suggest
#define NPY_NO_DEPRECATED_API NPY_API_VERSION. NPY_API_VERSION is hard-coded from C_API_VERSION in core/setup_common.py , see core/setup.py where the magic happens.

We don't really have a way to actually remove or disable deprecated API functions, since the convention is that the PyArray_API structure built in __multiarray_api.h by core/code_generators/generate_array_api.py from the list in core.code_generators/numpy_api.py can only grow, never shrink, to allow backward compatibility.

This is somewhat talked about in the documentation from the user's perspective and in the 'VERSIONING SUPPORT' docstring in setup_common.py, but if you think of hook where I can hang more documentation let me know.

Oops, good point.

Can we move PyArray_GetNumericOps to a macro that hides the C api macro?

We could eventually turn the warning into an exception. Not sure if I understand which "C api macro" you mean, there are so many.

At some point we could remove things from the PyArray_API structure by changing our major version, which would infer no backward API compatibility.

I mean this macro:

#define PyArray_GetNumericOps \ (*(PyObject * (*)(void)) \ PyArray_API[41])

Perhaps we should leave this as an open question, and just take this PR as it currently stands

charris · 2018-10-16T16:04:17Z

Not sure that we should increment the API version, the API hash remains the same and nothing has been added or removed, nor do we remove deprecated API functions -- there are a couple. So the API hasn't changed, we just recommend that some interface functions should not be used. If we want to disable the function, I don't think we have a policy for that.

eric-wieser · 2018-10-16T16:16:16Z

nor do we remove deprecated API functions

If that's the case, why does NPY_NO_DEPRECATED_API exist?

mattip · 2018-10-16T17:01:17Z

Those who wish to access PyArrayObject as per pre-1.7-NumPy can set it to NPY_1_6_API_VERSION and then PyArrayObject is exposed as a full C struct.

charris · 2018-10-16T17:30:51Z

Those who wish to access PyArrayObject as per pre-1.7-NumPy

Yep, there was a move to hide the structure internals behind functions so that we would be freed up to change the structures in the future, but that is incomplete and has been stalled for some time.

mattip · 2018-10-16T19:53:52Z

Note PR #11175 changed the size of PyUFuncObject. Since that is a publically-facing API we incremented NPY_API_VERSION even though the hash did not change.

charris · 2018-10-16T20:02:02Z

But that change is not backwards compatible, i.e., code that uses the new attributes will not run on older NumPy. Is that the case for code compiled against 1.16?

mattip · 2018-10-27T19:49:36Z

The only difference in this PR is the deprecation warning that will be emitted on 1.16 if the deprecated C functions are used. Does that not justify incrementing the NPY_API_VERSION?

eric-wieser · 2018-10-27T23:03:39Z

Those who wish to access PyArrayObject as per pre-1.7-NumPy can set it to NPY_1_6_API_VERSION and then PyArrayObject is exposed as a full C struct.

Conversely, I think that users should be able to opt into no longer having PyArray_GetNumericOps available by defining it to NPY_1_16_API_VERSION - which would indeed require an api version increment.

njsmith · 2018-10-27T23:36:19Z

If that's the case, why does NPY_NO_DEPRECATED_API exist?

Honestly, as far as I've ever been able to figure out, there is no good reason for it to exist.

Yep, there was a move to hide the structure internals behind functions so that we would be freed up to change the structures in the future, but that is incomplete and has been stalled for some time.

It was also done incorrectly... the internals were moved from macros/direct access to inline functions, but inline functions are just like macros: they both expose the structure internals as part of the ABI. And we never had a plan for how to actually remove the old APIs. And I'm not even sure what the point is... I don't think anything on our roadmap actually benefits from hiding the internals of PyArrayObject.

Also, I'm pretty sure that all of the C compilers we support have native support for issuing deprecation warnings, so I'm not sure why we need to invent our own system?

eric-wieser · 2018-10-28T00:19:04Z

the internals were moved from macros/direct access to inline functions

Are you sure the inline functions aren't inline only when used within multiarray? I thought I saw something like that somewhere, but perhaps I'm imagining it.

njsmith · 2018-10-28T02:22:59Z

I see code in ndarraytypes.h to switch them between macros and inline functions depending on NPY_NO_DEPRECATED_API, but not to convert them into actual API functions? I could be missing something...

mattip · 2018-10-29T20:09:03Z

The code currently uses functions in the headers which will probably be inlined to be identical to the macros.

convert them into actual API functions ...

This would incur a performance penalty since the call could no longer be inlined, and would cross a shared-object border. Whether that penalty is significant or not depends on the use case. We could adopt a policy where the first call caches a C-compatible representation of the hidden, possibly non-C-representable field, and functions like reshape or resize invalidate the cache, but as you say "I don't think anything on our roadmap actually benefits from hiding the internals of PyArrayObject"

Besides the 1.7 attribute hiding API changes, we used NPY_API_VERSION for the RESOLVEWRITEBACKIFCOPY deprecations, and now for these changes.

Also, PR #12235 fixes the definition of the NPY_1_XX_API_VERSION macros, and is waiting for review

Perhaps we could merge this, and if a change to the API versioning policy is needed open a new issue for that.

stefanv · 2018-11-06T19:48:34Z

This is probably good to go? It looks fairly uncontroversial.

mattip · 2018-11-10T00:45:45Z

reworded documentation

charris · 2018-11-10T02:07:27Z

Thanks Matti.

eric-wieser reviewed Sep 9, 2018

View reviewed changes

numpy/core/_add_newdocs.py Outdated Show resolved Hide resolved

eric-wieser reviewed Sep 10, 2018

View reviewed changes

numpy/core/src/multiarray/number.c Show resolved Hide resolved

eric-wieser reviewed Sep 10, 2018

View reviewed changes

numpy/core/src/multiarray/number.c Outdated Show resolved Hide resolved

charris added component: numpy._core 07 - Deprecation labels Sep 14, 2018

mattip force-pushed the deprecate-set_numeric_ops branch from 53613cf to e6b3911 Compare September 21, 2018 10:15

mattip closed this Sep 22, 2018

mattip reopened this Sep 22, 2018

eric-wieser reviewed Sep 22, 2018

View reviewed changes

numpy/core/src/multiarray/multiarraymodule.c Show resolved Hide resolved

eric-wieser reviewed Sep 22, 2018

View reviewed changes

numpy/core/tests/test_deprecations.py Show resolved Hide resolved

eric-wieser previously approved these changes Sep 22, 2018

View reviewed changes

mattip force-pushed the deprecate-set_numeric_ops branch from dae5b0e to 823ed3e Compare September 23, 2018 04:41

eric-wieser reviewed Sep 23, 2018

View reviewed changes

mattip force-pushed the deprecate-set_numeric_ops branch from 823ed3e to e20fd11 Compare September 25, 2018 07:19

mattip force-pushed the deprecate-set_numeric_ops branch from e20fd11 to 11fa5ca Compare September 28, 2018 13:16

mattip force-pushed the deprecate-set_numeric_ops branch from 94723cb to 0a21fa2 Compare October 12, 2018 08:18

eric-wieser reviewed Oct 16, 2018

View reviewed changes

doc/release/1.16.0-notes.rst Show resolved Hide resolved

eric-wieser reviewed Oct 16, 2018

View reviewed changes

DEP: deprecate np.set_numeric_ops and friends

73c5a5a

mattip added this to the 1.16.0 release milestone Oct 21, 2018

mattip force-pushed the deprecate-set_numeric_ops branch from 07e9486 to 73c5a5a Compare October 21, 2018 15:28

DOC: reword from review

08a2e45

charris merged commit d0e2f1a into numpy:master Nov 10, 2018

mattip mentioned this pull request Nov 13, 2018

NEP: move nep 15 from accepted to final #12377

Merged

mattip deleted the deprecate-set_numeric_ops branch March 4, 2019 13:18

merraksh mentioned this pull request Apr 6, 2020

avoid ValueError when overriding eq pandas-dev/pandas#22611

Closed

Uh oh!

DEP: deprecate np.set_numeric_ops and friends #11916

DEP: deprecate np.set_numeric_ops and friends #11916

Uh oh!

Conversation

mattip commented Sep 9, 2018

Uh oh!

Uh oh!

mattip commented Sep 9, 2018

Uh oh!

Uh oh!

Uh oh!

mattip commented Sep 17, 2018

Uh oh!

tylerjereddy commented Sep 17, 2018

Uh oh!

mattip commented Sep 21, 2018

Uh oh!

mattip commented Sep 22, 2018

Uh oh!

Uh oh!

Uh oh!

mattip commented Sep 22, 2018

Uh oh!

mattip commented Sep 23, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eric-wieser commented Sep 26, 2018

Uh oh!

eric-wieser commented Sep 26, 2018

Uh oh!

mattip commented Sep 28, 2018

Uh oh!

mattip commented Oct 9, 2018

Uh oh!

mattip commented Oct 12, 2018

Uh oh!

mattip commented Oct 16, 2018

Uh oh!

Uh oh!

eric-wieser commented Oct 16, 2018

Uh oh!

mattip commented Oct 16, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

charris commented Oct 16, 2018

Uh oh!

eric-wieser commented Oct 16, 2018

Uh oh!

mattip commented Oct 16, 2018

Uh oh!

charris commented Oct 16, 2018

Uh oh!

mattip commented Oct 16, 2018

Uh oh!

charris commented Oct 16, 2018

Uh oh!

mattip commented Oct 27, 2018

Uh oh!

eric-wieser commented Oct 27, 2018

Uh oh!

njsmith commented Oct 27, 2018

Uh oh!

eric-wieser commented Oct 28, 2018