Thanks to visit codestin.com
Credit goes to github.com

Skip to content

BUG: gh-10882 may have broken ctypes compatibility in older versions of python #11150

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
eric-wieser opened this issue May 24, 2018 · 18 comments
Closed

Comments

@eric-wieser
Copy link
Member

eric-wieser commented May 24, 2018

#10882 changed the behavior of np.array from.

  • Try to get a PEP3118 buffer interface
    • Try to convert the buffer interface into an array
    • If the buffer interface is not correct, try __array_interface__ instead
    • sometimes call abort() due to bad error silencing (ctypeslib.as_array aborts #6741)
  • If there's no buffer interface, try __array_interface__ instead

To

  • Try to get a PEP3118 memory view
    • Try to convert the buffer interface into an array
    • If the buffer interface is not correct, fail and tell the user that something is wrong
  • If there's no buffer interface, try __array_interface__ instead

This means that objects which incorrectly implement the buffer interface now fail loudly, rather than silently-or-with-abort().

Unfortunately, prior to python/cpython#31, (before 2.7.14 / 3.6.3 / 3.7.0) ctypes did not implement the buffer interface correctly for integer types (the format and itemsize didn't match)

Do we have a minimum supported patch version of python? If so, should we enable tests for it so that this doesn't happen again?

@mattip
Copy link
Member

mattip commented May 25, 2018

If the nightly wheels build cleanly we can close this.

@eric-wieser
Copy link
Member Author

That assumes that this was tested...

@mattip
Copy link
Member

mattip commented May 31, 2018

I am confused as to what the user-experienced problem is. From the cpython link, the following fails:

import ctypes, numpy
numpy.asarray(memoryview(ctypes.c_long(42)))

on an older python, whether I use HEAD or numpy 1.11.

Edit: The error changed from a ValueError to a RuntimeError

@mattip
Copy link
Member

mattip commented Jun 5, 2018

@eric-wieser could you clarify if the problem is a different exception type or something deeper?

@eric-wieser
Copy link
Member Author

eric-wieser commented Jun 6, 2018

@mattip: I think the case that matters is np.asarray(ctypes.c_long(42)). Previously that worked because we secretly added and used ctypes.c_long.__array_interface__. The referenced PR always prefers the buffer interface (ie, calling memoryview) to using __array_interface__

@eric-wieser
Copy link
Member Author

eric-wieser commented Jun 6, 2018

Turns out my WSL has python 2.7.12, so I'm all set to test this:

>>> sys.version_info
'2.7.12 (default, Dec  4 2017, 14:50:18) \n[GCC 5.4.0 20160609]'
>>> bad = ctypes.c_long(42)
>>> ctypes.sizeof(bad)   # this failure requires 64-bit long (ie, not windows native)
8

# 1.14
>>> np.asarray(bad)
RuntimeWarning: Item size computed from the PEP 3118 buffer format string does not match the actual item size.
array(42)

# master
>>> np.asarray(bad)
RuntimeError: Item size computed from the PEP 3118 buffer format string does not match the actual item size.

@charris
Copy link
Member

charris commented Jun 7, 2018

@eric-wieser We can close this now?

@eric-wieser
Copy link
Member Author

Not unless things have changed since I ran my test above (which shows it failing)

@charris
Copy link
Member

charris commented Jun 7, 2018

Hmm. I vote that we document the required versions, with a strong suggest that folks keep current. Given that the problems are fixed upstream that seems reasonable. What about python 3.5?

EDIT: I'm going to propose dropping 3.4 in the 1.16 release.

@eric-wieser
Copy link
Member Author

In case it changes your mind, ubuntu 16.04 ships with python 2.7.12, which exhibits the ctypes bug

@eric-wieser
Copy link
Member Author

I could try tweaking #10882 to have a special case for ctypes objects,

@charris
Copy link
Member

charris commented Jun 7, 2018

Sure, give it a shot, but don't spend a lot of time on it.

eric-wieser added a commit to eric-wieser/numpy that referenced this issue Jun 8, 2018
…ype(ctype_obj))

Fixes numpygh-10528
Fixes numpygh-10978
Fixes numpygh-11150

A warning is thrown when ctypes misbehaves, in order to not hide issues that need fixing upstream
@eric-wieser
Copy link
Member Author

eric-wieser commented Nov 19, 2018

Reopening - this is a real issue for ndpointer, which provides:

  • A valid but unsupported PEP3118 interface
  • __array_interface__

After this change, we now reject the PEP3118 interface, and never check __array_interface__.

This causes toinsson/pyrealsense#82, which unfortunately they never raised upstream.

#12380 has made the error message even more cryptic, so we really ought to fix this for 1.16.

@eric-wieser eric-wieser reopened this Nov 19, 2018
@mattip
Copy link
Member

mattip commented Nov 26, 2018

The follow-on issue #12421 was left open but tentatively declared "won't fix", does that mean this issue is also "won't fix", or at least can be removed from the 1.16 milestone?

@eric-wieser
Copy link
Member Author

Yeah, this can be removed from the milestone

@eric-wieser eric-wieser removed this from the 1.16.0 release milestone Nov 26, 2018
@vadimkantorov
Copy link

I also stumbled on this: I have unrelated buffer interface (inherited from ctypes impl) and my custom array_interface that I'd like to be used

@mattip
Copy link
Member

mattip commented Mar 31, 2023

@eric-wieser we have dropped support for python<3.8, I think we can close this?

@seberg
Copy link
Member

seberg commented Aug 9, 2024

Agreed with Matti, seems like we can close it with old Python versions long EOL and unsupported.

@seberg seberg closed this as completed Aug 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants