TST: Extract a helper function to test for reference cycles #10891
Conversation
e47992b to
9241f57
Compare
|
An example of this in use: In [1]: from numpy.testing import assert_no_gc_cycles
In [2]: def make_array_ctype(shape):
...: import ctypes
...: ct = ctypes.c_uint8
...: for i in shape:
...: ct = i * ct
...:
In [3]: assert_no_gc_cycles(make_array_ctype, (1,))
AssertionError: Reference cycles were found when calling make_array_ctype: 7 objects were collected, of which 6 are shown below:
tuple object with id=2822255556536:
(<class '_ctypes.Array'>,)
PyCArrayType object with id=2822226500408:
<class '__main__.c_ubyte_Array_1'>
getset_descriptor object with id=2822252062256:
<attribute '__dict__' of 'c_ubyte_Array_1' objects>
getset_descriptor object with id=2822252062184:
<attribute '__weakref__' of 'c_ubyte_Array_1' objects>
tuple object with id=2822243712440:
(<class '__main__.c_ubyte_Array_1'>,
<class '_ctypes.Array'>,
<class '_ctypes._CData'>,
<class 'object'>)
StgDict object with id=2822226211928:
{'__dict__': <attribute '__dict__' of 'c_ubyte_Array_1' objects>,
'__doc__': None,
'__module__': '__main__',
'__weakref__': <attribute '__weakref__' of 'c_ubyte_Array_1' objects>,
'_length_': 1,
'_type_': <class 'ctypes.c_ubyte'>}Not clear to me if this is a ctypes bug, or if it is by design |
|
This assumes a single gc.collect() call is enough to make the next
gc.collect()==0, but is that really guaranteed? Should this call
gc.collect() multiple times first to flush things out?
|
This also means we can now test that our test is actually able to detect the type of failure we expect Trying to give myself some tools to debug the failure at https://github.com/numpy/numpy/pull/10882/files#r180813166
An example output for the test added in the previous commit is:
AssertionError: Reference cycles were found when calling make_cycle: 1 objects were collected, of which 1 are shown below:
list object with id=2279664872136:
[<Recursion on list with id=2279664872136>,
<Recursion on list with id=2279664872136>]
9241f57 to
d21ec05
Compare
|
@pv: I don't know. My goal here was just to encapsulate the test we're already using, and add some extra diagnostics. |
|
After a rethink: spot-on diagnosis of the problem I had in #10882. This test should indeed be collecting multiple times. I'll add another commit. |
|
Python 2.7 is being difficult. |
|
Hmm. Can you think of a reliable test that prevents the garbage collector from ever finishing collecting? If not, should I just change the test to suppress RuntimeError rather than actually test for it? |
It's not always possible to guarantee this, so also adds a test to verify that we don't hang
96323e3 to
3ff0c5c
Compare
|
Seems that sometimes the GC doesn't even try to finalize the objects, which avoids the pathological case anyway. I've detected this with a This now passes on 2.7 on my machine, and hopefully will do the same on the travis 3.6 failure. (edit: it does) |
|
Anything else needed here? Would |
|
LGTM, @pv any comments? Doesn't exactly decrease the amount of code, but is worth having something debugged and expected to work. |
|
Thanks Eric. |
|
Hmm, needs a release note as it adds a new function to |
MAINT: Remove workaround for gh-10891
This also means we can now test that our test is actually able to detect the type of failure we expect
Trying to give myself some tools to debug the failure at https://github.com/numpy/numpy/pull/10882/files#r180813166