Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Memory leak in function is_scalar_with_conversion (a static analysis report with a POC) #19376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Snape3058 opened this issue Jun 29, 2021 · 15 comments · Fixed by #19403
Closed

Memory leak in function is_scalar_with_conversion (a static analysis report with a POC) #19376

Snape3058 opened this issue Jun 29, 2021 · 15 comments · Fixed by #19403
Labels
00 - Bug

Comments

@Snape3058
Copy link

Reproducing code example:

import operator
import sys
import numpy as np

class Bar(str):
    index = 500
    def __index__(self):
        return self.index

a = np.array([1, 2])
b = Bar('1')

bef_ref = sys.getrefcount(b.index)  # b.index->ob_refcnt == 4

for i in range(1000):
    try:
        operator.pow(a, b)
    except:
        pass

assert sys.getrefcount(b.index) == bef_ref  # b.index->ob_refcnt == 1004

As the reference count linearly increases with the number of invocations, which indicates a memory leak.

Error message:

The path provided by the static analyzer is as follows.

  1. In condition branch starts from here.

    numpy/numpy/core/src/multiarray/number.c

    Line 422 in 04ab04d

    else if (PyIndex_Check(o2)) {

  2. A new reference returned from PyNumber_Index and pointed to be value.

    numpy/numpy/core/src/multiarray/number.c

    Line 423 in 04ab04d

    PyObject* value = PyNumber_Index(o2);

  3. Assume value is not NULL.

    numpy/numpy/core/src/multiarray/number.c

    Line 425 in 04ab04d

    if (value == NULL) {

  4. Call PyLong_AsSsize_t with value, which does not steal the reference.

    numpy/numpy/core/src/multiarray/number.c

    Line 431 in 04ab04d

    val = PyLong_AsSsize_t(value);

  5. Function returns on the normal branch. (the error branch is also the same case)

    numpy/numpy/core/src/multiarray/number.c

    Line 437 in 04ab04d

    return NPY_INTPOS_SCALAR;

  6. The new reference pointed to by value leaks without decreasing its refcnt.

NumPy/Python version information:

Static analysis carried out on commit 04ab04d.
@Snape3058 Snape3058 changed the title Protential memory leak in function is_scalar_with_conversion (a static analysis report) Memory leak in function is_scalar_with_conversion (a static analysis report with POC) Jun 29, 2021
@Snape3058 Snape3058 changed the title Memory leak in function is_scalar_with_conversion (a static analysis report with POC) Memory leak in function is_scalar_with_conversion (a static analysis report with a POC) Jun 29, 2021
@theoniko
Copy link
Contributor

theoniko commented Jul 2, 2021

Hello @Snape3058, @seberg,

I was trying to understand this issue and i think that based on https://stackoverflow.com/questions/24444667/what-is-the-purpose-of-py-decref-and-py-incref that a Py_DECREF(value) after val = PyLong_AsSsize_t(value); would solve the issue. Could you please maybe give more hints because i have not contribute in numpy project before?

@seberg
Copy link
Member

seberg commented Jul 2, 2021

This is the Python C-API. Basically, you "just" need to count how often you "own" an object, and make sure you own it 0 times when you are done – but I know it can be very confusing when you start with the Python C-API.
You incref if you "own" an object (once for each time you "own" it), and decref when you are done. It is roughly equivalent to:

new_variable = variable  # same as Py_INCREF(variable)
del new_variable  # same as Py_DECREF(variable)

Most functions return new references, so you do not have to call INCREF (you already own it), but you have to del it.

One thing if you fix this is to please also add the reproducer as a test. It is OK if the test just runs the bad code path: There are tools to do reference count leak checking, we do not have to test it explicitly in the test suite. (It would be fine to do that, but I think it is not worth the trouble probably)

@theoniko theoniko mentioned this issue Jul 3, 2021
Merged
@theoniko
Copy link
Contributor

theoniko commented Jul 4, 2021 •
edited
Loading

Hello @seberg,

I opened #19403 and i checked that it passes the assert from code snippet above.
My only problem at the moment is that i cannot run the unit tests locally. I created the virtual environment using conda commans:

  • conda env create -f environment.yml
  • conda activate numpy-dev

Running for exampe python runtests.py -t numpy/core/tests though does not work for me(see build.log). Could you please advice how to proceed or whom to ask? I tried to ask about it in stack overflow but i did not get any reply. Also my os is windows.
Thanks in advance.
build.log

@mattip
Copy link
Member

mattip commented Jul 4, 2021

Did you run git clean -xfdd before building?

@theoniko
Copy link
Contributor

theoniko commented Jul 4, 2021

Hello @mattip,
Yes i did but i still get the same error. Is there a slack channel to ask there as well?

@mattip
Copy link
Member

mattip commented Jul 4, 2021

I asked because the top line is "numpy/random_bounded_integers.pxd.in has not changed" which indicates your build is not clean.

@mattip
Copy link
Member

mattip commented Jul 4, 2021

On the other hand, it may just be a bug with windows + conda fortran compiler with a clean environment (before calling vsvars.bat). It is strange that your environment has LIB set but not INCLUDE. What happens if you do

set INCLUDE='c:\User'

before running runtests.py?

@theoniko
Copy link
Contributor

theoniko commented Jul 4, 2021

build.log
Now the error is RuntimeError: Broken toolchain: cannot link a simple C program

@mattip
Copy link
Member

mattip commented Jul 4, 2021

@rgommers does the environment.yml need any adjustment to properly allow discovery of compilers?

@rgommers
Copy link
Member

rgommers commented Jul 4, 2021

It shouldn't, conda activate sets all the correct paths and environment variables.

Let me copy the build log here, it's short:

...
blas_opt_info:
blas_mkl_info:
No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils
customize MSVCCompiler
  libraries mkl_rt not found in ['C:\\Users\\anaconda3\\envs\\numpy-dev\\lib', 'C:\\', 'C:\\Users\\anaconda3\\envs\\numpy-dev\\libs', 'C:\\Users\\anaconda3\\Library\\lib']
  NOT AVAILABLE

blis_info:
  libraries blis not found in ['C:\\Users\\anaconda3\\envs\\numpy-dev\\lib', 'C:\\', 'C:\\Users\\anaconda3\\envs\\numpy-dev\\libs', 'C:\\Users\\anaconda3\\Library\\lib']
  NOT AVAILABLE

openblas_info:
  libraries openblas not found in ['C:\\Users\\anaconda3\\envs\\numpy-dev\\lib', 'C:\\', 'C:\\Users\\anaconda3\\envs\\numpy-dev\\libs', 'C:\\Users\\anaconda3\\Library\\lib']
Traceback (most recent call last):
  File "C:\Users\numpy\setup.py", line 447, in <module>
    setup_package()
  File "C:\Users\numpy\setup.py", line 439, in setup_package
    setup(**metadata)
  File "C:\Users\numpy\numpy\distutils\core.py", line 135, in setup
    config = configuration()
  File "C:\Users\numpy\setup.py", line 117, in configuration
    config.add_subpackage('numpy')
  File "C:\Users\numpy\numpy\distutils\misc_util.py", line 1016, in add_subpackage
    config_list = self.get_subpackage(subpackage_name, subpackage_path,
  File "C:\Users\numpy\numpy\distutils\misc_util.py", line 982, in get_subpackage
    config = self._get_configuration_from_setup_py(
  File "C:\Users\numpy\numpy\distutils\misc_util.py", line 924, in _get_configuration_from_setup_py
    config = setup_module.configuration(*args)
  File "numpy\setup.py", line 8, in configuration
    config.add_subpackage('core')
  File "C:\Users\numpy\numpy\distutils\misc_util.py", line 1016, in add_subpackage
    config_list = self.get_subpackage(subpackage_name, subpackage_path,
  File "C:\Users\numpy\numpy\distutils\misc_util.py", line 982, in get_subpackage
    config = self._get_configuration_from_setup_py(
  File "C:\Users\numpy\numpy\distutils\misc_util.py", line 924, in _get_configuration_from_setup_py
    config = setup_module.configuration(*args)
  File "numpy\core\setup.py", line 755, in configuration
    blas_info = get_info('blas_opt', 0)
  File "C:\Users\numpy\numpy\distutils\system_info.py", line 581, in get_info
    return cl().get_info(notfound_action)
  File "C:\Users\numpy\numpy\distutils\system_info.py", line 841, in get_info
    self.calc_info()
  File "C:\Users\numpy\numpy\distutils\system_info.py", line 2026, in calc_info
    if self._calc_info(blas):
  File "C:\Users\numpy\numpy\distutils\system_info.py", line 2012, in _calc_info
    return getattr(self, '_calc_info_{}'.format(name))()
  File "C:\Users\numpy\numpy\distutils\system_info.py", line 1949, in _calc_info_openblas
    info = get_info('openblas')
  File "C:\Users\numpy\numpy\distutils\system_info.py", line 581, in get_info
    return cl().get_info(notfound_action)
  File "C:\Users\numpy\numpy\distutils\system_info.py", line 841, in get_info
    self.calc_info()
  File "C:\Users\numpy\numpy\distutils\system_info.py", line 2232, in calc_info
    info = self._calc_info()
  File "C:\Users\numpy\numpy\distutils\system_info.py", line 2203, in _calc_info
    f = new_fcompiler(c_compiler=c)
  File "C:\Users\numpy\numpy\distutils\fcompiler\__init__.py", line 874, in new_fcompiler
    load_all_fcompiler_classes()
  File "C:\Users\numpy\numpy\distutils\fcompiler\__init__.py", line 775, in load_all_fcompiler_classes
    __import__ (module_name)
  File "C:\Users\numpy\numpy\distutils\fcompiler\compaq.py", line 54, in <module>
    class CompaqVisualFCompiler(FCompiler):
  File "C:\Users\numpy\numpy\distutils\fcompiler\compaq.py", line 78, in CompaqVisualFCompiler
    m.initialize()
  File "C:\Users\numpy\numpy\distutils\msvccompiler.py", line 52, in initialize
    os.environ['include'] = _merge(environ_include, os.environ['include'])
  File "C:\Users\anaconda3\envs\numpy-dev\lib\os.py", line 679, in __getitem__
    raise KeyError(key) from None
KeyError: 'include'

It can't find a BLAS library. The problem is that numpy.distutils is bad at finding those automatically on Windows, so you need a site.cfg file with the correct paths in it. Like so: https://github.com/conda-forge/numpy-feedstock/blob/master/recipe/build.sh#L11

@rgommers
Copy link
Member

rgommers commented Jul 4, 2021

Oh wait, there's a second one, ending in

RuntimeError: Broken toolchain: cannot link a simple C program

Running from numpy source directory.
numpy/random\_bounded_integers.pxd.in has not changed
numpy/random\bit_generator.pyx has not changed
numpy/random\mtrand.pyx has not changed
numpy/random\_bounded_integers.pyx has not changed
numpy/random\_bounded_integers.pyx.in has not changed
numpy/random\_common.pyx has not changed
numpy/random\_generator.pyx has not changed
numpy/random\_mt19937.pyx has not changed
numpy/random\_pcg64.pyx has not changed
numpy/random\_philox.pyx has not changed
numpy/random\_sfc64.pyx has not changed
Cythonizing sources
blas_opt_info:
blas_mkl_info:
No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils
customize MSVCCompiler
  libraries mkl_rt not found in ['C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\lib', 'C:\\', 'C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\libs', 'C:\\Users\\nikol\\anaconda3\\Library\\lib']
  NOT AVAILABLE

blis_info:
  libraries blis not found in ['C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\lib', 'C:\\', 'C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\libs', 'C:\\Users\\nikol\\anaconda3\\Library\\lib']
  NOT AVAILABLE

openblas_info:
  libraries openblas not found in ['C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\lib', 'C:\\', 'C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\libs', 'C:\\Users\\nikol\\anaconda3\\Library\\lib']
get_default_fcompiler: matching types: '['gnu', 'intelv', 'absoft', 'compaqv', 'intelev', 'gnu95', 'g95', 'intelvem', 'intelem', 'flang']'
customize GnuFCompiler
Could not locate executable g77
Could not locate executable f77
customize IntelVisualFCompiler
Could not locate executable ifort
Could not locate executable ifl
customize AbsoftFCompiler
Could not locate executable f90
customize CompaqVisualFCompiler
Found executable C:\cyg\cygwin64\bin\DF.exe
customize IntelItaniumVisualFCompiler
Could not locate executable efl
customize Gnu95FCompiler
Could not locate executable gfortran
Could not locate executable f95
customize G95FCompiler
Could not locate executable g95
customize IntelEM64VisualFCompiler
customize IntelEM64TFCompiler
Could not locate executable efort
Could not locate executable efc
customize PGroupFlangCompiler
Found executable C:\Users\nikol\anaconda3\envs\numpy-dev\Library\bin\flang.exe
  NOT AVAILABLE

accelerate_info:
  NOT AVAILABLE

atlas_3_10_blas_threads_info:
Setting PTATLAS=ATLAS
  libraries tatlas not found in ['C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\lib', 'C:\\', 'C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\libs', 'C:\\Users\\nikol\\anaconda3\\Library\\lib']
  NOT AVAILABLE

atlas_3_10_blas_info:
  libraries satlas not found in ['C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\lib', 'C:\\', 'C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\libs', 'C:\\Users\\nikol\\anaconda3\\Library\\lib']
  NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
  libraries ptf77blas,ptcblas,atlas not found in ['C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\lib', 'C:\\', 'C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\libs', 'C:\\Users\\nikol\\anaconda3\\Library\\lib']
  NOT AVAILABLE

atlas_blas_info:
  libraries f77blas,cblas,atlas not found in ['C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\lib', 'C:\\', 'C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\libs', 'C:\\Users\\nikol\\anaconda3\\Library\\lib']
  NOT AVAILABLE

C:\Users\nikol\OneDrive\Documents\OpenProjects\Python\numpy\numpy\distutils\system_info.py:2026: UserWarning: 
    Optimized (vendor) Blas libraries are not found.
    Falls back to netlib Blas library which has worse performance.
    A better performance should be easily gained by switching
    Blas library.
  if self._calc_info(blas):
blas_info:
  libraries blas not found in ['C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\lib', 'C:\\', 'C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\libs', 'C:\\Users\\nikol\\anaconda3\\Library\\lib']
  NOT AVAILABLE

C:\Users\nikol\OneDrive\Documents\OpenProjects\Python\numpy\numpy\distutils\system_info.py:2026: UserWarning: 
    Blas (http://www.netlib.org/blas/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [blas]) or by setting
    the BLAS environment variable.
  if self._calc_info(blas):
blas_src_info:
  NOT AVAILABLE

C:\Users\nikol\OneDrive\Documents\OpenProjects\Python\numpy\numpy\distutils\system_info.py:2026: UserWarning: 
    Blas (http://www.netlib.org/blas/) sources not found.
    Directories to search for the sources can be specified in the
    numpy/distutils/site.cfg file (section [blas_src]) or by setting
    the BLAS_SRC environment variable.
  if self._calc_info(blas):
  NOT AVAILABLE

non-existing path in 'numpy\\distutils': 'site.cfg'
lapack_opt_info:
lapack_mkl_info:
  libraries mkl_rt not found in ['C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\lib', 'C:\\', 'C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\libs', 'C:\\Users\\nikol\\anaconda3\\Library\\lib']
  NOT AVAILABLE

openblas_lapack_info:
  libraries openblas not found in ['C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\lib', 'C:\\', 'C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\libs', 'C:\\Users\\nikol\\anaconda3\\Library\\lib']
get_default_fcompiler: matching types: '['gnu', 'intelv', 'absoft', 'compaqv', 'intelev', 'gnu95', 'g95', 'intelvem', 'intelem', 'flang']'
customize GnuFCompiler
customize IntelVisualFCompiler
customize AbsoftFCompiler
customize CompaqVisualFCompiler
customize IntelItaniumVisualFCompiler
customize Gnu95FCompiler
customize G95FCompiler
customize IntelEM64VisualFCompiler
customize IntelEM64TFCompiler
customize PGroupFlangCompiler
  NOT AVAILABLE

openblas_clapack_info:
  libraries openblas,lapack not found in ['C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\lib', 'C:\\', 'C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\libs', 'C:\\Users\\nikol\\anaconda3\\Library\\lib']
get_default_fcompiler: matching types: '['gnu', 'intelv', 'absoft', 'compaqv', 'intelev', 'gnu95', 'g95', 'intelvem', 'intelem', 'flang']'
customize GnuFCompiler
customize IntelVisualFCompiler
customize AbsoftFCompiler
customize CompaqVisualFCompiler
customize IntelItaniumVisualFCompiler
customize Gnu95FCompiler
customize G95FCompiler
customize IntelEM64VisualFCompiler
customize IntelEM64TFCompiler
customize PGroupFlangCompiler
  NOT AVAILABLE

flame_info:
  libraries flame not found in ['C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\lib', 'C:\\', 'C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\libs', 'C:\\Users\\nikol\\anaconda3\\Library\\lib']
  NOT AVAILABLE

atlas_3_10_threads_info:
Setting PTATLAS=ATLAS
  libraries lapack_atlas not found in C:\Users\nikol\anaconda3\envs\numpy-dev\lib
  libraries tatlas,tatlas not found in C:\Users\nikol\anaconda3\envs\numpy-dev\lib
  libraries lapack_atlas not found in C:\
  libraries tatlas,tatlas not found in C:\
  libraries lapack_atlas not found in C:\Users\nikol\anaconda3\envs\numpy-dev\libs
  libraries tatlas,tatlas not found in C:\Users\nikol\anaconda3\envs\numpy-dev\libs
  libraries lapack_atlas not found in C:\Users\nikol\anaconda3\Library\lib
  libraries tatlas,tatlas not found in C:\Users\nikol\anaconda3\Library\lib
<class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
  NOT AVAILABLE

atlas_3_10_info:
  libraries lapack_atlas not found in C:\Users\nikol\anaconda3\envs\numpy-dev\lib
  libraries satlas,satlas not found in C:\Users\nikol\anaconda3\envs\numpy-dev\lib
  libraries lapack_atlas not found in C:\
  libraries satlas,satlas not found in C:\
  libraries lapack_atlas not found in C:\Users\nikol\anaconda3\envs\numpy-dev\libs
  libraries satlas,satlas not found in C:\Users\nikol\anaconda3\envs\numpy-dev\libs
  libraries lapack_atlas not found in C:\Users\nikol\anaconda3\Library\lib
  libraries satlas,satlas not found in C:\Users\nikol\anaconda3\Library\lib
<class 'numpy.distutils.system_info.atlas_3_10_info'>
  NOT AVAILABLE

atlas_threads_info:
Setting PTATLAS=ATLAS
  libraries lapack_atlas not found in C:\Users\nikol\anaconda3\envs\numpy-dev\lib
  libraries ptf77blas,ptcblas,atlas not found in C:\Users\nikol\anaconda3\envs\numpy-dev\lib
  libraries lapack_atlas not found in C:\
  libraries ptf77blas,ptcblas,atlas not found in C:\
  libraries lapack_atlas not found in C:\Users\nikol\anaconda3\envs\numpy-dev\libs
  libraries ptf77blas,ptcblas,atlas not found in C:\Users\nikol\anaconda3\envs\numpy-dev\libs
  libraries lapack_atlas not found in C:\Users\nikol\anaconda3\Library\lib
  libraries ptf77blas,ptcblas,atlas not found in C:\Users\nikol\anaconda3\Library\lib
<class 'numpy.distutils.system_info.atlas_threads_info'>
  NOT AVAILABLE

atlas_info:
  libraries lapack_atlas not found in C:\Users\nikol\anaconda3\envs\numpy-dev\lib
  libraries f77blas,cblas,atlas not found in C:\Users\nikol\anaconda3\envs\numpy-dev\lib
  libraries lapack_atlas not found in C:\
  libraries f77blas,cblas,atlas not found in C:\
  libraries lapack_atlas not found in C:\Users\nikol\anaconda3\envs\numpy-dev\libs
  libraries f77blas,cblas,atlas not found in C:\Users\nikol\anaconda3\envs\numpy-dev\libs
  libraries lapack_atlas not found in C:\Users\nikol\anaconda3\Library\lib
  libraries f77blas,cblas,atlas not found in C:\Users\nikol\anaconda3\Library\lib
<class 'numpy.distutils.system_info.atlas_info'>
  NOT AVAILABLE

lapack_info:
  libraries lapack not found in ['C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\lib', 'C:\\', 'C:\\Users\\nikol\\anaconda3\\envs\\numpy-dev\\libs', 'C:\\Users\\nikol\\anaconda3\\Library\\lib']
  NOT AVAILABLE

C:\Users\nikol\OneDrive\Documents\OpenProjects\Python\numpy\numpy\distutils\system_info.py:1858: UserWarning: 
    Lapack (http://www.netlib.org/lapack/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [lapack]) or by setting
    the LAPACK environment variable.
  return getattr(self, '_calc_info_{}'.format(name))()
lapack_src_info:
  NOT AVAILABLE

C:\Users\nikol\OneDrive\Documents\OpenProjects\Python\numpy\numpy\distutils\system_info.py:1858: UserWarning: 
    Lapack (http://www.netlib.org/lapack/) sources not found.
    Directories to search for the sources can be specified in the
    numpy/distutils/site.cfg file (section [lapack_src]) or by setting
    the LAPACK_SRC environment variable.
  return getattr(self, '_calc_info_{}'.format(name))()
  NOT AVAILABLE

numpy_linalg_lapack_lite:
  FOUND:
    language = c
    define_macros = [('HAVE_BLAS_ILP64', None), ('BLAS_SYMBOL_SUFFIX', '64_')]

Warning: attempted relative import with no known parent package
C:\Users\nikol\anaconda3\envs\numpy-dev\lib\distutils\dist.py:274: UserWarning: Unknown distribution option: 'define_macros'
  warnings.warn(msg)
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building py_modules sources
creating build\src.win-amd64-3.9
creating build\src.win-amd64-3.9\numpy
creating build\src.win-amd64-3.9\numpy\distutils
building library "npymath" sources
LINK : fatal error LNK1104: cannot open file 'MSVCRT.lib'
Traceback (most recent call last):
  File "C:\Users\nikol\OneDrive\Documents\OpenProjects\Python\numpy\setup.py", line 447, in <module>
    setup_package()
  File "C:\Users\nikol\OneDrive\Documents\OpenProjects\Python\numpy\setup.py", line 439, in setup_package
    setup(**metadata)
  File "C:\Users\nikol\OneDrive\Documents\OpenProjects\Python\numpy\numpy\distutils\core.py", line 169, in setup
    return old_setup(**new_attr)
  File "C:\Users\nikol\anaconda3\envs\numpy-dev\lib\site-packages\setuptools\__init__.py", line 163, in setup
    return distutils.core.setup(**attrs)
  File "C:\Users\nikol\anaconda3\envs\numpy-dev\lib\distutils\core.py", line 148, in setup
    dist.run_commands()
  File "C:\Users\nikol\anaconda3\envs\numpy-dev\lib\distutils\dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "C:\Users\nikol\anaconda3\envs\numpy-dev\lib\distutils\dist.py", line 985, in run_command
    cmd_obj.run()
  File "C:\Users\nikol\OneDrive\Documents\OpenProjects\Python\numpy\numpy\distutils\command\build.py", line 61, in run
    old_build.run(self)
  File "C:\Users\nikol\anaconda3\envs\numpy-dev\lib\distutils\command\build.py", line 135, in run
    self.run_command(cmd_name)
  File "C:\Users\nikol\anaconda3\envs\numpy-dev\lib\distutils\cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "C:\Users\nikol\anaconda3\envs\numpy-dev\lib\distutils\dist.py", line 985, in run_command
    cmd_obj.run()
  File "C:\Users\nikol\OneDrive\Documents\OpenProjects\Python\numpy\numpy\distutils\command\build_src.py", line 144, in run
    self.build_sources()
  File "C:\Users\nikol\OneDrive\Documents\OpenProjects\Python\numpy\numpy\distutils\command\build_src.py", line 155, in build_sources
    self.build_library_sources(*libname_info)
  File "C:\Users\nikol\OneDrive\Documents\OpenProjects\Python\numpy\numpy\distutils\command\build_src.py", line 288, in build_library_sources
    sources = self.generate_sources(sources, (lib_name, build_info))
  File "C:\Users\nikol\OneDrive\Documents\OpenProjects\Python\numpy\numpy\distutils\command\build_src.py", line 378, in generate_sources
    source = func(extension, build_dir)
  File "numpy\core\setup.py", line 666, in get_mathlib_info
    raise RuntimeError("Broken toolchain: cannot link a simple C program")
RuntimeError: Broken toolchain: cannot link a simple C program

@theoniko you can include it directly here, and if it's long then enclose it in:

<details>

...

</details>

Much easier than letting us download build logs.

@rgommers
Copy link
Member

rgommers commented Jul 4, 2021

set INCLUDE='c:\User'

I'm not a Windows expert, but that doesn't seem healthy. There's nothing useful to include there, and perhaps causes the error?

I'd say just do what conda-forge build.bat does: https://github.com/conda-forge/numpy-feedstock/blob/master/recipe/bld.bat

@theoniko
Copy link
Contributor

theoniko commented Jul 7, 2021 •
edited
Loading

I used ubuntu VM and i was finally able to run the unit tests. I will make a new try for Windows. Is there a compiler flag to try gcc instead of VS compiler?

@theoniko
Copy link
Contributor

theoniko commented Jul 7, 2021 •
edited by mattip
Loading

Hello @rgommers, @mattip, @seberg
Thanks a lot for your help. As i mentioned above i managed to have the unit tests running in ubuntu VM but
on windows i am getting the error DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working for Python (Python used is 3.8.5). Could you please help bypass that error?

Building, see build.log...
Build OK
NumPy version 1.22.0.dev0+339.gd7cf6cd2b
NumPy relaxed strides checking option: True
NumPy CPU features:  SSE SSE2 SSE3 SSSE3* SSE41* POPCNT* SSE42* AVX* F16C* FMA3* AVX2* AVX512F? AVX512CD? AVX512_SKX? AVX512_CLX? AVX512_CNL?
Traceback (most recent call last):
  File "runtests.py", line 685, in <module>
    main(argv=sys.argv[1:])
  File "runtests.py", line 383, in main
    result = test(args.mode,
  File "C:\Users\nikol\OneDrive\Documents\OpenProjects\Python\numpy\build\testenv\Lib\site-packages\numpy\_pytesttester.py", line 197, in __call__
    code = pytest.main(pytest_args)
  File "C:\Users\nikol\anaconda3\lib\site-packages\_pytest\config\__init__.py", line 143, in main
    config = _prepareconfig(args, plugins)
  File "C:\Users\nikol\anaconda3\lib\site-packages\_pytest\config\__init__.py", line 318, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "C:\Users\nikol\anaconda3\lib\site-packages\pluggy\hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "C:\Users\nikol\anaconda3\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "C:\Users\nikol\anaconda3\lib\site-packages\pluggy\manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "C:\Users\nikol\anaconda3\lib\site-packages\pluggy\callers.py", line 203, in _multicall
    gen.send(outcome)
  File "C:\Users\nikol\anaconda3\lib\site-packages\_pytest\helpconfig.py", line 100, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "C:\Users\nikol\anaconda3\lib\site-packages\pluggy\callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "C:\Users\nikol\anaconda3\lib\site-packages\pluggy\callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "C:\Users\nikol\anaconda3\lib\site-packages\_pytest\config\__init__.py", line 1003, in pytest_cmdline_parse
    self.parse(args)
  File "C:\Users\nikol\anaconda3\lib\site-packages\_pytest\config\__init__.py", line 1283, in parse
    self._preparse(args, addopts=addopts)
  File "C:\Users\nikol\anaconda3\lib\site-packages\_pytest\config\__init__.py", line 1191, in _preparse
    self.hook.pytest_load_initial_conftests(
  File "C:\Users\nikol\anaconda3\lib\site-packages\pluggy\hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "C:\Users\nikol\anaconda3\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "C:\Users\nikol\anaconda3\lib\site-packages\pluggy\manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "C:\Users\nikol\anaconda3\lib\site-packages\pluggy\callers.py", line 208, in _multicall
    return outcome.get_result()
  File "C:\Users\nikol\anaconda3\lib\site-packages\pluggy\callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "C:\Users\nikol\anaconda3\lib\site-packages\pluggy\callers.py", line 182, in _multicall
    next(gen)  # first yield
  File "C:\Users\nikol\anaconda3\lib\site-packages\_pytest\capture.py", line 157, in pytest_load_initial_conftests
    _readline_workaround()
  File "C:\Users\nikol\anaconda3\lib\site-packages\_pytest\capture.py", line 90, in _readline_workaround
    import readline  # noqa: F401
  File "C:\Users\nikol\anaconda3\lib\site-packages\readline.py", line 34, in <module>
    rl = Readline()
  File "C:\Users\nikol\anaconda3\lib\site-packages\pyreadline\rlmain.py", line 422, in __init__
    BaseReadline.__init__(self)
  File "C:\Users\nikol\anaconda3\lib\site-packages\pyreadline\rlmain.py", line 62, in __init__
    mode.init_editing_mode(None)
  File "C:\Users\nikol\anaconda3\lib\site-packages\pyreadline\modes\emacs.py", line 633, in init_editing_mode
    self._bind_key('space',       self.self_insert)
  File "C:\Users\nikol\anaconda3\lib\site-packages\pyreadline\modes\basemode.py", line 162, in _bind_key
    if not callable(func):
  File "C:\Users\nikol\anaconda3\lib\site-packages\pyreadline\py3k_compat.py", line 8, in callable
    return isinstance(x, collections.Callable)
  File "C:\Users\nikol\anaconda3\lib\collections\__init__.py", line 49, in __getattr__
    warnings.warn("Using or importing the ABCs from 'collections' instead "

@mattip
Copy link
Member

mattip commented Jul 7, 2021

Somehow you installed pyreadline. It last released a version over 5 years ago and has not kept up. Can you remove it? Also: when commenting here, you can use the code format button in the tool bar (or enclose your copy-paste inside three back-ticks on a separate line at the beginning of the text and three more on a separate line after)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
00 - Bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

BUG: Fix memory leak in function npyiter_multi_index_set theoniko/numpy
5 participants
@seberg @rgommers @mattip @Snape3058 @theoniko