@pv wrote on 2010-06-27

Can you attach a minimal test case that exhibits this crash?

Your array is actually an object array (the gdb backtrace shows this). How do you actually load it from the text file?

What platform (64-bit linux), etc?

@pv wrote on 2010-06-27

Ok, thanks, I managed to reproduce it now:

import numpy as np
print np.__version__, np.__file__
x = np.random.rand(42364).astype(object)
np.any(x)

Title changed from segfault in any() on large array with duplicate values to segfault in any() on large object array by @pv on 2010-06-27

trac user glub wrote on 2010-06-27

Sorry for leaving out the example.

This happened on 64-bit Fedora 13, python 2.6.4.

trac user jpeel wrote on 2010-12-29

The limit seems to be 9999 objects. 10000 segfaults.

trac user jpeel wrote on 2010-12-29

Well, I'm flummoxed, but the only significance of the number 10000 is that is when the BUFFER_UFUNCLOOP method is used. The problem, I believe, has something to do with the casting of the object to a boolean such that the result turns out to be an invalid object. Then, since the cast object is invalid, there is a segfault when Py_XINCREF is used on it.

Attachment added by trac user jpeel on 2011-01-04: 0001-BF-added-fix-for-segfault-when-using-any-on-large-ar.patch

trac user jpeel wrote on 2011-01-04

I've found the problem. The beginning of the process for when the buffer is used (# of objects >= 10000) is as follows. The first object of the array is copied and then cast as a Bool. However, if either the input array contains objects or the output will be of type object, then loop->obj was set to 1 in construct_reduce() and that signals that the cast object (in this case to a Bool) should be INCREFed. However, since the object is a Bool rather than a PyObject, a segfault occurs. This same problem doesn't happen with smaller arrays because in that case the first object is copied without being cast and then INCREFed. Since the object is a PyObject, there isn't a problem.

The patch that I submitted just removes the lines that INCREF when loop->obj is set and the array is large enough to trigger buffering. The only potential problem with doing this is if a ufunc generates an object as output, but I don't see that really as a possibility. Does anyone have any problem with this fix? The alternative is to make different loop->obj flags: one for when the input is an object and one for when the output is an object.

Pretty sure this is fixed. This bug was reported before the ufunc machinery was reworked to use the nditer. I am unable to reproduce with WinXP/numpy 1.6.2 or current master on linux64.

I don't see a segfault, but this is fishy.

In [1]: x = np.random.rand(42364).astype(object)

In [2]: np.any(x)
Out[2]: 0.8834972689258551

Should probably return a boolean. Same with all.

I'll open another issue for that.