BUG/ENH: Fix use of ndpointer in return values #12431

eric-wieser · 2018-11-21T03:20:39Z

This:

fixes a regression in 1.15, where it became impossible to set the return value of a cdll function to an ndpointer.
removes ndpointer.__array_interface__, which was being ignored anyway in favor of the PEP3118 buffer protocol
adds ndpointer.contents to recover the lost functionality, while staying in line with the ctypes behavior
removes another instance of descr, which enables overlapping fields to be returned from C functions (such as unions).
Fixes a long-term bug where using ndpointer as a return type without specifying both type and dtype would produce an object array containing a single ndpointer. Now the ndpointer is returned directly.

This relates to (but does not fix) gh-12421, and likely fixes toinsson/pyrealsense#82

eric-wieser · 2018-11-21T03:21:31Z

doc/release/1.16.0-notes.rst

+This matches the ``.contents`` member of normal ctypes arrays, and can be used
+to construct an ``np.array`` around the pointers contents.
+
+This replaces ``np.array(some_nd_pointer)``, which stopped working in 1.15.


On the one hand, we could try to restore this - on the other, it didn't make a massive amount of sense anyway.

eric-wieser · 2018-11-21T03:29:20Z

numpy/ctypeslib.py

+        full_dtype = _dtype((self._dtype_, self._shape_))
+        full_ctype = ctypes.c_char * full_dtype.itemsize
+        buffer = ctypes.cast(self, ctypes.POINTER(full_ctype)).contents
+        return frombuffer(buffer, dtype=full_dtype).squeeze(axis=0)


This is sort of frustrating - there ought to be a better way to construct an array from a pointer without having to go through an intermediate byte array.

That's why the full_ctype ?

Yeah, the goal of full_ctypes is to get an object supporting the buffer protocol that we can pass to frombuffer. Perhaps ndarray.from_address(addr, shape, strides, dtype) ought to be a (dangerous) part of the C api?

Interesting thought. I'd mostly be concerned if memory or speed was a problem with the current approach.

Memory shouldn't be a concern - this doesn't make any copies of the underlying data - at most we're churning around a bunch of metadata. _CPointer.contents returns a reference, not a copy.

numpy/ctypeslib.py

numpy/tests/test_ctypeslib.py

charris · 2018-11-21T22:18:12Z

numpy/tests/test_ctypeslib.py

    if hasattr(sys, 'gettotalrefcount'):
        try:
            cdll = load_library('_multiarray_umath_d', np.core._multiarray_umath.__file__)
        except OSError:
            pass
+        try:
+            test_cdll = load_library('_multiarray_tests', np.core._multiarray_tests.__file__)


Long lines here and above.

Missed this one. It's not clear to me why load_library is needed here - the tests in cpython just use ctypes.CDLL(np.core._multiarray_tests.__file__), and presumably work just fine.

Are you looking to fix this, or are you going to try the cpython approach and remove an argument?

Oops, load_library is in the public interface, forgot about that.

Given the lines in this section were long before this patch, I'd be inclined leave things as they are, and not spend more CI cycles on whitespace changes :).

In a future patch I might try using CDLL directly here and seeing if any CI breaks.

charris · 2018-11-21T22:29:47Z

Couple of style nits. Otherwise LGTM. Is there any reason someone might have been using the private _ndptr class?

eric-wieser · 2018-11-21T22:32:53Z

@charris:

Is there any reason someone might have been using the private _ndptr class?

I assume you mean using it by name, rather than using it as the return value of ndpointer?

I had a quick look at whether splitting off _concrete_ndptr from _ndptr would cause trouble - the use case that came to mind was #6239, but even that goes through the public API.

If you're still worried, I could rename _ndptr to _abstract_ndptr, and then leave _concrete_ndptr at the name _ndptr?

This: * fixes a regression in 1.15, where it became impossible to set the return value of a cdll function to an ndpointer. * removes ndpointer.__array_interface__, which was being ignored anyway in favor of the PEP3118 buffer protocol * adds `ndpointer.contents` to recover the lost functionality, while staying in line with the ctypes behavior * removes another instance of `descr`, which enables overlapping fields to be returned from C functions (such as unions). * Fixes a long-term bug where using ndpointer as a return type without specifying both type and dtype would produce an object array containing a single `ndpointer`. Now the ndpointer is returned directly. This relates to numpygh-12421, and likely fixes toinsson/pyrealsense#82

eric-wieser · 2018-11-21T22:35:58Z

Style nits addressed - let me know if any remain

charris · 2018-11-21T23:16:12Z

I'm not particularly worried, just trying to game might go wrong, I was mostly thinking to the missing __array_interface__. Let's give it a shot and see what happens (yeah, I'm really not temperamentally suited to the release manager role :)

eric-wieser · 2018-11-21T23:18:04Z

I'd be surprised if anyone is using ndpointer.__array_interface__ directly. In the event that they are, and we break them - then using ndpointer.contents.__array_interface__ will work more effectively in its place

A bigger question is whether we still need to fix gh-12421 - perhaps we have the priority of __array_interface__ and PEP3118 flipped.

Should we be viewing PEP3118 as a replacement for __array_interface__, or __array_interface__ as an extension of PEP3118?

charris · 2018-11-22T00:18:11Z

Should we be viewing PEP3118 as a replacement for array_interface, or array_interface as an extension of PEP3118?

I don't really have an opinion as I haven't used the ctypes functionality. The new contents method works well on the python side the problem is that it isn't backwards compatible. Is it difficult to fix __array_interface__ ? I'm not clear on where the 'P' came from in #12421.

eric-wieser · 2018-11-22T00:24:35Z

The P has always been there - it's the PEP3118 buffer format for c_void_p, which is the base class of ndptr.

What changed in 1.15 is that instead of ignoring unsupported or corrupt PEP3118 formats and trying something else, we raise an exception. Previously that something else was __array_interface__.

charris · 2018-11-22T00:42:43Z

Hmm, from what you say #12421 should be a "won't fix". Is there a backwards compatible workaround?

eric-wieser · 2018-11-22T00:46:12Z

Depends what we want to be compatible with. If the goal is simply to make np.array(ndptr) work, ~~then defining:~~

~~def __array__(self); warn("use .contents instead") return self.contents~~

~~will work, I think.~~ - edit: this does not work.

eric-wieser · 2018-11-22T00:49:10Z

from what you say #12421 should be a "won't fix"

Correct, that's the option I'm leaning towards. I suspect no one actually cares anyway - the only downstream issue I found is resolved by this PR.

charris · 2018-11-22T00:58:12Z

I was thinking about __array__ also, but it looks it is reserved for instances of ndarray.

charris · 2018-11-22T01:10:50Z

What I was asking though, was whether frombuffer or similar could be used.

eric-wieser · 2018-11-22T01:45:49Z

What I was asking though, was whether frombuffer or similar could be used.

Used to achieve what? #10882 made it impossible to override np.array(some_nd_pointer), I think - but at the same time, I've not seen any uses of that in the wild (beyond the one removed in this patch) anyway

charris · 2018-11-22T02:29:12Z

Used to achieve what?

Can't say, as the the actual code was that led to the issue isn't known. The example is artificial, after all the array is already available :) So what is the actual problem downstream, and can it be worked around.
If toinsson/pyrealsense#82 is fixed, is there anything else to worry about?

eric-wieser · 2018-11-22T02:48:04Z

@charris: Ask who? By "the issue", are you referring to toinsson/pyrealsense#82? Because if so, the cause is using ndpointer as a ctypes return value, which this patch fixes and tests.

charris · 2018-11-22T03:34:25Z

Yes, the question was about toinsson/pyrealsense#82 and concerns whether or not they can work around the issue for the versions of NumPy that were failing.

eric-wieser · 2018-11-22T03:45:17Z

In that case, yes, they can work around it by copying my .contents implementation, and declaring the return type a c_void_p to prevent ndpointer getting in the way.

The alternative would be for us to backport this PR to 1.15.x, but that might not be worth it at this point

charris · 2018-11-22T03:54:47Z

Well, let's get this in. Thanks Eric.

charris · 2018-11-22T03:57:37Z

I was concerned about a workaround that would not only fix the problem, but work for all earlier numpy versions so they didn't end up with version dependent code. Sorry that it has taken me a bit of trouble to correctly express the concern. Anyway, sounds like that is possible.

eric-wieser · 2018-11-22T04:01:27Z

but work for all earlier numpy versions so they didn't end up with version dependent code

Yes, I believe that frombuffer has worked on 1d ctypes arrays of uint8 on all numpy versions - so they should be able to copy .contents and use it in all numpy versions.

vadimkantorov · 2020-07-27T19:16:23Z

I have ctypes objects that implement both buffer protocol and __array_interface__ (to expose some inner field on a higher-level object). It seems that in this case at least NumPy 1.17.4 always prefers the Buffer Protocol at np.asarray(...) call. Is there a way to force it to use __array_interface__? This issue seems to discuss precedence of PEP3118 and __array_interface__, so I hope my question is relevant here

eric-wieser · 2020-07-27T20:17:18Z

I suspect the buffer protocol takes precedence, but you might find that __array__ has higher precedence than both.

eric-wieser added 00 - Bug 01 - Enhancement 06 - Regression component: numpy._core labels Nov 21, 2018

eric-wieser added this to the 1.16.0 release milestone Nov 21, 2018

eric-wieser commented Nov 21, 2018

View reviewed changes

charris reviewed Nov 21, 2018

View reviewed changes

numpy/ctypeslib.py Outdated Show resolved Hide resolved

charris reviewed Nov 21, 2018

View reviewed changes

numpy/tests/test_ctypeslib.py Outdated Show resolved Hide resolved

charris reviewed Nov 21, 2018

View reviewed changes

numpy/tests/test_ctypeslib.py Outdated Show resolved Hide resolved

charris reviewed Nov 21, 2018

View reviewed changes

eric-wieser force-pushed the ndpointer-return branch from 1056f0d to 73322a0 Compare November 21, 2018 22:35

charris merged commit f01924f into numpy:master Nov 22, 2018

navjotk mentioned this pull request Dec 3, 2018

numpy 1.15 currently breaks Devito devitocodes/devito#608

Closed

eric-wieser mentioned this pull request Apr 15, 2019

BUG: Conversion from ndpointer to array is broken in 1.15 #12421

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG/ENH: Fix use of ndpointer in return values #12431

BUG/ENH: Fix use of ndpointer in return values #12431

eric-wieser commented Nov 21, 2018 •

edited

Loading

eric-wieser Nov 21, 2018

eric-wieser Nov 21, 2018 •

edited

Loading

charris Nov 21, 2018

eric-wieser Nov 21, 2018 •

edited

Loading

charris Nov 21, 2018

eric-wieser Nov 21, 2018 •

edited

Loading

charris Nov 21, 2018

eric-wieser Nov 21, 2018

charris Nov 21, 2018

charris Nov 21, 2018

eric-wieser Nov 21, 2018

charris commented Nov 21, 2018

eric-wieser commented Nov 21, 2018 •

edited

Loading

eric-wieser commented Nov 21, 2018

charris commented Nov 21, 2018

eric-wieser commented Nov 21, 2018 •

edited

Loading

charris commented Nov 22, 2018

eric-wieser commented Nov 22, 2018

charris commented Nov 22, 2018

eric-wieser commented Nov 22, 2018 •

edited

Loading

eric-wieser commented Nov 22, 2018

charris commented Nov 22, 2018 •

edited

Loading

charris commented Nov 22, 2018

eric-wieser commented Nov 22, 2018 •

edited

Loading

charris commented Nov 22, 2018 •

edited

Loading

eric-wieser commented Nov 22, 2018

charris commented Nov 22, 2018

eric-wieser commented Nov 22, 2018 •

edited

Loading

charris commented Nov 22, 2018

charris commented Nov 22, 2018

eric-wieser commented Nov 22, 2018

vadimkantorov commented Jul 27, 2020 •

edited

Loading

eric-wieser commented Jul 27, 2020

BUG/ENH: Fix use of ndpointer in return values #12431

BUG/ENH: Fix use of ndpointer in return values #12431

Conversation

eric-wieser commented Nov 21, 2018 • edited Loading

Choose a reason for hiding this comment

eric-wieser Nov 21, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eric-wieser Nov 21, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eric-wieser Nov 21, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

charris commented Nov 21, 2018

eric-wieser commented Nov 21, 2018 • edited Loading

eric-wieser commented Nov 21, 2018

charris commented Nov 21, 2018

eric-wieser commented Nov 21, 2018 • edited Loading

charris commented Nov 22, 2018

eric-wieser commented Nov 22, 2018

charris commented Nov 22, 2018

eric-wieser commented Nov 22, 2018 • edited Loading

eric-wieser commented Nov 22, 2018

charris commented Nov 22, 2018 • edited Loading

charris commented Nov 22, 2018

eric-wieser commented Nov 22, 2018 • edited Loading

charris commented Nov 22, 2018 • edited Loading

eric-wieser commented Nov 22, 2018

charris commented Nov 22, 2018

eric-wieser commented Nov 22, 2018 • edited Loading

charris commented Nov 22, 2018

charris commented Nov 22, 2018

eric-wieser commented Nov 22, 2018

vadimkantorov commented Jul 27, 2020 • edited Loading

eric-wieser commented Jul 27, 2020

eric-wieser commented Nov 21, 2018 •

edited

Loading

eric-wieser Nov 21, 2018 •

edited

Loading

eric-wieser Nov 21, 2018 •

edited

Loading

eric-wieser Nov 21, 2018 •

edited

Loading

eric-wieser commented Nov 21, 2018 •

edited

Loading

eric-wieser commented Nov 21, 2018 •

edited

Loading

eric-wieser commented Nov 22, 2018 •

edited

Loading

charris commented Nov 22, 2018 •

edited

Loading

eric-wieser commented Nov 22, 2018 •

edited

Loading

charris commented Nov 22, 2018 •

edited

Loading

eric-wieser commented Nov 22, 2018 •

edited

Loading

vadimkantorov commented Jul 27, 2020 •

edited

Loading