BUG: importing numpy.distutils replaces the global logger with a custom class which doesn't support kwargs #24213

oliver-s-lee · 2023-07-19T08:04:28Z

Describe the issue:

Importing numpy.distutils silently replaces the global logger returned by logging.getLogger() with the custom class from numpy.distutils.log. This is surprising in of itself, but additionally the signature of numpy.distutils.log.Log._log() is outdated/incorrect and does not support the kwargs exc_info, stack_info, stacklevel or extra. In particular, this means the exception() method of the custom logger does not work.

Reproduce the code example:

>>> import logging
>>> type(logging.getLogger())
<class 'logging.RootLogger'>
>>> import numpy.distutils
>>> type(logging.getLogger())
<class 'numpy.distutils.log.Log'>
>>> logging.getLogger().exception("Oh No")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/oliver/anaconda3/envs/python3.10.12/lib/python3.10/logging/__init__.py", line 1512, in exception
    self.error(msg, *args, exc_info=exc_info, **kwargs)
  File "/home/oliver/anaconda3/envs/python3.10.12/lib/python3.10/logging/__init__.py", line 1506, in error
    self._log(ERROR, msg, args, **kwargs)
TypeError: Log._log() got an unexpected keyword argument 'exc_info'

Error message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/oliver/anaconda3/envs/python3.10.12/lib/python3.10/logging/__init__.py", line 1512, in exception
    self.error(msg, *args, exc_info=exc_info, **kwargs)
  File "/home/oliver/anaconda3/envs/python3.10.12/lib/python3.10/logging/__init__.py", line 1506, in error
    self._log(ERROR, msg, args, **kwargs)
TypeError: Log._log() got an unexpected keyword argument 'exc_info'

Runtime information:

>>> import sys, numpy; print(numpy.__version__); print(sys.version) 
1.22.3
3.10.8 (main, Nov 24 2022, 14:13:03) [GCC 11.2.0]

Context for the issue:

Any package that imports anything from numpy.distutils will accidentally break the root logger for the whole program/script. In addition, because it's not obvious that the root logger is replaced in this way, this problem can be (and sadly was) challenging to diagnose.

For a real world example, importing pyscf also recreates this behaviour:

>>> import logging
>>> type(logging.getLogger())
<class 'logging.RootLogger'>
>>> import pyscf
>>> type(logging.getLogger())
<class 'numpy.distutils.log.Log'>
>>> try:
...     raise Exception("Bugger")
... except Exception:
...     logging.getLogger().exception("Oh No")
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
Exception: Bugger

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
  File "/home/oliver/anaconda3/envs/python3.10.12/lib/python3.10/logging/__init__.py", line 1512, in exception
    self.error(msg, *args, exc_info=exc_info, **kwargs)
  File "/home/oliver/anaconda3/envs/python3.10.12/lib/python3.10/logging/__init__.py", line 1506, in error
    self._log(ERROR, msg, args, **kwargs)
TypeError: Log._log() got an unexpected keyword argument 'exc_info'

The text was updated successfully, but these errors were encountered:

charris · 2023-07-19T14:05:23Z

I am going to close this. Distutils is very old and now deprecated, we are moving to meson for NumPy 2.0.0 and 1.26.0 builds. Python itself dropped distutils in Python 3.12.

If you have questions about the transition, feel free to continue the discussion here.

oliver-s-lee · 2023-07-20T09:56:35Z

Fair enough. For completeness and for other's reference, I should point out that this does not only affect importing numpy.distutils directly, but also crops up in other places where numpy itself imports from numpy.distutils. For example, numpy.ctypeslib.load_library() uses numpy.distutils.misc_util.get_shared_lib_extension, and hence calling load_library() will break your root logger.

charris · 2023-07-20T20:18:36Z

I should point out that this does not only affect importing numpy.distutils directly,

Is this in the main branch, or in a release? If this is still in the main branch, it will need fixing. You might try one of the nightly wheels to see if there are still problems: https://anaconda.org/scientific-python-nightly-wheels/numpy/ .

mattip · 2023-07-21T06:04:21Z

The main branch still uses numpy.distutils.misc_util.get_shared_lib_extension, which is still needed since sysconfig.get_config_vars('SHLIB_SUFFIX') still returns .so on darwin, which is wrong. It should be .dylib. So we should move that function out of distutils (maybe copy it into ctypeslib.py).

At some point after finishing #24206 (which allows removing setuptools from the test_requirements.txt pip requirements file), we should try deleting numpy.distutils to see what tests will fail because we still use it internally.

rgommers · 2023-07-22T09:26:05Z

The main branch still uses numpy.distutils.misc_util.get_shared_lib_extension, which is still needed since sysconfig.get_config_vars('SHLIB_SUFFIX') still returns .so on darwin, which is wrong. It should be .dylib. So we should move that function out of distutils (maybe copy it into ctypeslib.py).

It doesn't:

numpy/numpy/ctypeslib.py

Lines 131 to 142 in b0ff5c5

    
           # Try to load library with platform-specific name, otherwise 
        
           # default to libname.[so|dll|dylib].  Sometimes, these files are 
        
           # built erroneously on non-linux platforms. 
        
           base_ext = ".so" 
        
           if sys.platform.startswith("darwin"): 
        
               base_ext = ".dylib" 
        
           elif sys.platform.startswith("win"): 
        
               base_ext = ".dll" 
        
           libname_ext = [libname + base_ext] 
        
           so_ext = sysconfig.get_config_var("EXT_SUFFIX") 
        
           if not so_ext == base_ext: 
        
               libname_ext.insert(0, libname + so_ext)

. Also the 1.25.x branch is clean.

numpy.distutils is not installed at all on Python 3.12, and all tests pass there. So any functionality in NumPy that has tests no longer relies on numpy.distutils. It's possible that something untested slipped through the cracks, but that'll be pretty niche and likely will not be found by manually deleting the folder.

I searched the repo again, it's a bit difficult to do because of the many comments, setup.py files etc. where distutils and numpy.distutils are still mentioned/used. But I don't see anything really problematic.

I agree with closing this issue, because of the orginal reason given: numpy.disutils is deprecated and slated to disappear, so we don't really want to spend time on this kind of thing anymore.

mattip · 2023-07-23T05:56:16Z

It doesn't:
numpy.distutils is not installed at all on Python 3.12, and all tests pass there.

Got, thanks. I must have been looking at a different branch. Thanks for re-checking. Closing.

oliver-s-lee added the 00 - Bug label Jul 19, 2023

oliver-s-lee mentioned this issue Jul 19, 2023

Fix for not all input objects supporting last_line cclib/cclib#1222

Merged

charris closed this as completed Jul 19, 2023

charris reopened this Jul 20, 2023

rgommers added component: numpy.distutils 57 - Close? Issues which may be closable unless discussion continued labels Jul 22, 2023

mattip closed this as completed Jul 23, 2023

marius-baseten mentioned this issue Nov 18, 2024

Fix model wrapper logger, and better XP around chains docker image va… basetenlabs/truss#1251

Merged

harupy mentioned this issue Mar 7, 2025

[BUG] mlflow autologging resets root logger mlflow/mlflow#14887

Closed

23 tasks

lifuhuang mentioned this issue May 14, 2025

[Bug] Fix accidental logger override caused by internVL. sgl-project/sglang#6282

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

BUG: importing numpy.distutils replaces the global logger with a custom class which doesn't support kwargs #24213

BUG: importing numpy.distutils replaces the global logger with a custom class which doesn't support kwargs #24213

oliver-s-lee commented Jul 19, 2023 •

edited

Loading

charris commented Jul 19, 2023

Uh oh!

oliver-s-lee commented Jul 20, 2023

Uh oh!

charris commented Jul 20, 2023 •

edited

Loading

Uh oh!

mattip commented Jul 21, 2023 •

edited

Loading

Uh oh!

rgommers commented Jul 22, 2023

Uh oh!

mattip commented Jul 23, 2023

Uh oh!

Uh oh!

BUG: importing numpy.distutils replaces the global logger with a custom class which doesn't support kwargs #24213

BUG: importing numpy.distutils replaces the global logger with a custom class which doesn't support kwargs #24213

Comments

oliver-s-lee commented Jul 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe the issue:

Reproduce the code example:

Error message:

Runtime information:

Context for the issue:

charris commented Jul 19, 2023

Uh oh!

oliver-s-lee commented Jul 20, 2023

Uh oh!

charris commented Jul 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattip commented Jul 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rgommers commented Jul 22, 2023

Uh oh!

mattip commented Jul 23, 2023

Uh oh!

oliver-s-lee commented Jul 19, 2023 •

edited

Loading

charris commented Jul 20, 2023 •

edited

Loading

mattip commented Jul 21, 2023 •

edited

Loading