[numpy-1.19.3 , py3.8.5, Linux Ubuntu Env] Hstack method -> Unable to allocate memory , 180 KiB #17684

MarkBel · 2020-10-30T15:16:29Z

The error has occured when using hstack, with array with shape (700, 33) and data type float64. The problem is not regarding the system as $ echo 1 > /proc/sys/vm/overcommit_memory
This will enable "always overcommit" mode, and you'll find that indeed the system will allow you to make the allocation no matter how large it is (within 64-bit memory addressing at least). The usage just required 180 KiB of space, so it can not be the root cause of the problem.

Reproducing code example:

import numpy as np
np.hstack(columns_from_the_dataframe)

Error message:

Traceback (most recent call last):
  File "<__array_function__ internals>", line 5, in hstack
  File "/usr/local/lib/python3.8/dist-packages/numnumpy-1.19.3py/core/shape_base.py", line 346, in hstack
    return _nx.concatenate(arrs, 1)
  File "<__array_function__ internals>", line 5, in concatenate
MemoryError: Unable to allocate 180. KiB for an array with shape (700, 33) and data type float64

NumPy/Python version information:

numpy-1.19.3

The text was updated successfully, but these errors were encountered:

eric-wieser · 2020-10-30T15:29:45Z

Here's the code that emits that error:

numpy/numpy/core/src/multiarray/ctors.c

Lines 809 to 834 in 08f9eeb

    
               if (data == NULL) { 
        
                   /* 
        
                    * Allocate something even for zero-space arrays 
        
                    * e.g. shape=(0,) -- otherwise buffer exposure 
        
                    * (a.data) doesn't work as it should. 
        
                    * Could probably just allocate a few bytes here. -- Chuck 
        
                    */ 
        
                   if (nbytes == 0) { 
        
                       nbytes = descr->elsize ? descr->elsize : 1; 
        
                   } 
        
                   /* 
        
                    * It is bad to have uninitialized OBJECT pointers 
        
                    * which could also be sub-fields of a VOID array 
        
                    */ 
        
                   if (zeroed || PyDataType_FLAGCHK(descr, NPY_NEEDS_INIT)) { 
        
                       data = npy_alloc_cache_zero(nbytes); 
        
                   } 
        
                   else { 
        
                       data = npy_alloc_cache(nbytes); 
        
                   } 
        
                   if (data == NULL) { 
        
                       raise_memory_error(fa->nd, fa->dimensions, descr); 
        
                       goto fail; 
        
                   } 
        
                   fa->flags |= NPY_ARRAY_OWNDATA; 
        
               }

There are only two ways that npy_alloc_cache can fail:

malloc returns null
Our cache is somehow corrupt

Does this error happen every time, or is it intermittent?

seberg · 2020-10-30T15:49:23Z

This is a bit scary. Did this issue appear with 1.19.3 and not with 1.19.2?

Our cache could corrupt, but it seems to me the main way that might happen is if we free an array without holding the GIL by accident? It would be interesting if you could provide a more complete example? I am guessing here that the np.hstack call is not enough to produce this failure (e.g. say you are loading the dataframe using pickle)?

charris · 2020-10-30T16:12:41Z

Did this issue appear with 1.19.3 and not with 1.19.2?

I'd really like to know that also, as 1.19.4 is in preparation. Is the issue repeatable?

MarkBel · 2020-10-30T16:17:46Z

@charris @seberg No issue as for 1.19.2, np.hstack works fine and smooth. Just checked with the code and 1.19.2 np version.

seberg · 2020-10-30T16:31:06Z

Uff, there are very few changes aside from the OpenBLAS one so it is a bit hard to imagine what might be going on, unless it has to do with the wheel rather than the changes? Or it is some existing race condition that just didn't show up before?

Is there any chance of being able to give a full reproducing example? Running it in gdb may also be nice, but I admit I do not have much hope that it will show anything useful.

charris · 2020-10-30T16:43:58Z

This leaves me in a quandary :( I suppose the safe thing to do would be to put out a 1.19.4rc1.

charris · 2020-10-30T16:47:18Z

unless it has to do with the wheel rather than the changes

Scary. We believe we are using newer docker images for the manylinux releases.

charris · 2020-10-30T18:48:40Z

@MarkBel Could you try installing 1.19.4 from the staging repo?

python -mpip install -i https://pypi.anaconda.org/multibuild-wheels-staging/simple numpy

I'd like to get 1.19.4 tested before uploading to PyPI.

MarkBel · 2020-11-01T15:18:04Z

@MarkBel Could you try installing 1.19.4 from the staging repo?
python -mpip install -i https://pypi.anaconda.org/multibuild-wheels-staging/simple numpy
I'd like to get 1.19.4 tested before uploading to PyPI.

I will do my best tmr and give you the feedback back.

charris · 2020-11-01T16:17:44Z

Thanks!

MarkBel · 2020-11-02T07:53:48Z

@charris Coming back with the positive feedback, I have just upgraded numpy to 1.19.4 from the staging repo as you suggested and no issue has occurred regarding memory allocation, here we go!

charris · 2020-11-02T15:22:18Z

@MarkBel Thanks.

AnkitSnh2 · 2020-12-17T00:22:46Z

I still see the issue by calling

S = np.cumsum(image)

logs:

Traceback (most recent call last):
  File "main.py", line 158, in <module>
    tf.app.run()
  File "C:\Users\***\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "C:\Users\***\AppData\Local\Programs\Python\Python37\lib\site-packages\absl\app.py", line 303, in run
    _run_main(main, args)
  File "C:\Users\***\AppData\Local\Programs\Python\Python37\lib\site-packages\absl\app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "main.py", line 102, in main
    tmp = integral_image(train_image[idx])
  File "C:\Users\***\
***\Desktop\images\Research Project\JAAD\haar\viola-jones-algo-master\detection\utils\integ_img.py", line 27, in integral_image
    S = np.cumsum(image)
  File "<__array_function__ internals>", line 6, in cumsum
  File "C:\Users\***\AppData\Local\Programs\Python\Python37\lib\site-packages\numpy\core\fromnumeric.py", line 2483, in cumsum
    return _wrapfunc(a, 'cumsum', axis=axis, dtype=dtype, out=out)
  File "C:\Users\***\AppData\Local\Programs\Python\Python37\lib\site-packages\numpy\core\fromnumeric.py", line 58, in _wrapfunc
    return bound(*args, **kwds)
MemoryError: Unable to allocate 23.7 MiB for an array with shape (6220800,) and data type uint32

python -c "import numpy; print(numpy.version.version)"
1.19.4

seberg · 2020-12-17T01:24:14Z

Would anyone be able to create a minimal, shareable example? Otherwise it is very hard to dig into it to see what is going on. There are few changes in 1.19.3 but it seems a bit like there might be a subtle but serious bug somewhere, and any starting point to find it would be very useful.

AnkitSnh2 · 2020-12-17T09:55:52Z

I tried reproing the issue, but it gives no issue with single image file. Can this be the problem because program tries to allocate more memory than available in stack? In my case I have to calculate cumsum of around 1400 images.

mattip · 2020-12-17T10:13:30Z

If each image is 23.7MB, then you would need (1400 * 23.7) / 1024 = 32.4 GB to just hold the array. So I think @AnkitSnh2's issue is very different from the one in the beginning of this issue, which seems to have been solved by changing OpenBLAS. Closing. @AnkitSnh2 if you still cannot work out what is going on, please open a new issue and detail exactly what you are doing.

mattip mentioned this issue Nov 2, 2020

maybe a problem with 0.3.12 and NumPy? OpenMathLib/OpenBLAS#2970

Closed

mattip mentioned this issue Nov 12, 2020

BLD: use BUFFERSIZE=20 in OpenBLAS #17759

Merged

charris mentioned this issue Dec 4, 2020

BLD: use BUFFERSIZE=20 in OpenBLAS #17924

Merged

mattip closed this as completed Dec 17, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[numpy-1.19.3 , py3.8.5, Linux Ubuntu Env] Hstack method -> Unable to allocate memory , 180 KiB #17684

[numpy-1.19.3 , py3.8.5, Linux Ubuntu Env] Hstack method -> Unable to allocate memory , 180 KiB #17684

MarkBel commented Oct 30, 2020 •

edited by eric-wieser

Loading

eric-wieser commented Oct 30, 2020 •

edited

Loading

Uh oh!

seberg commented Oct 30, 2020

Uh oh!

charris commented Oct 30, 2020

Uh oh!

MarkBel commented Oct 30, 2020 •

edited

Loading

Uh oh!

seberg commented Oct 30, 2020

Uh oh!

charris commented Oct 30, 2020

Uh oh!

charris commented Oct 30, 2020

Uh oh!

charris commented Oct 30, 2020

Uh oh!

MarkBel commented Nov 1, 2020

Uh oh!

charris commented Nov 1, 2020

Uh oh!

MarkBel commented Nov 2, 2020

Uh oh!

charris commented Nov 2, 2020

Uh oh!

AnkitSnh2 commented Dec 17, 2020 •

edited by eric-wieser

Loading

Uh oh!

seberg commented Dec 17, 2020

Uh oh!

AnkitSnh2 commented Dec 17, 2020

Uh oh!

mattip commented Dec 17, 2020 •

edited

Loading

Uh oh!

Uh oh!

[numpy-1.19.3 , py3.8.5, Linux Ubuntu Env] Hstack method -> Unable to allocate memory , 180 KiB #17684

[numpy-1.19.3 , py3.8.5, Linux Ubuntu Env] Hstack method -> Unable to allocate memory , 180 KiB #17684

Comments

MarkBel commented Oct 30, 2020 • edited by eric-wieser Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reproducing code example:

Error message:

NumPy/Python version information:

eric-wieser commented Oct 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seberg commented Oct 30, 2020

Uh oh!

charris commented Oct 30, 2020

Uh oh!

MarkBel commented Oct 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seberg commented Oct 30, 2020

Uh oh!

charris commented Oct 30, 2020

Uh oh!

charris commented Oct 30, 2020

Uh oh!

charris commented Oct 30, 2020

Uh oh!

MarkBel commented Nov 1, 2020

Uh oh!

charris commented Nov 1, 2020

Uh oh!

MarkBel commented Nov 2, 2020

Uh oh!

charris commented Nov 2, 2020

Uh oh!

AnkitSnh2 commented Dec 17, 2020 • edited by eric-wieser Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seberg commented Dec 17, 2020

Uh oh!

AnkitSnh2 commented Dec 17, 2020

Uh oh!

mattip commented Dec 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MarkBel commented Oct 30, 2020 •

edited by eric-wieser

Loading

eric-wieser commented Oct 30, 2020 •

edited

Loading

MarkBel commented Oct 30, 2020 •

edited

Loading

AnkitSnh2 commented Dec 17, 2020 •

edited by eric-wieser

Loading

mattip commented Dec 17, 2020 •

edited

Loading