Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[Bug]: Coredump when importing matplotlib.pyplot #25823

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
john-odonovan opened this issue May 5, 2023 · 16 comments
Closed

[Bug]: Coredump when importing matplotlib.pyplot #25823

john-odonovan opened this issue May 5, 2023 · 16 comments
Labels
status: needs clarification Issues that need more information to resolve.

Comments

@john-odonovan
Copy link

Bug summary

When I execute python -c 'import matplotlib.pyplot', python will coredump. This will occur when it is generating the ~/.cache/matplotlib/fontlist-v330.json file, assuming that it was not already successfully generated.

The root cause of the problem is that the /usr/share/fonts/google-noto-emoji/NotoColorEmoji.ttf font will cause an invalid pixel size exception to be thrown from the FreeType library, the exception is not subsequently handled in the C++ layer, and python will abort.

Code for reproduction

[ -f /usr/share/fonts/google-noto-emoji/NotoColorEmoji.ttf ] \
  && /bin/rm -f ~/.cache/matplotlib/fontlist-v330.json \
  && python -c "import matplotlib.pyplot"

Actual outcome

[1] 229645 abort (core dumped) python -c

Expected outcome

No output

Additional information

It appears that this is only caused by the NotoColorEmoji font. If I update font_manager.py to skip this font, there is no issue.
This was at least working in 3.1.3. I have tried 3.4.3 and 3.6.1 and they both have the same issue. I'm using system_freetype=True in the mplsetup.py file. I have also tried compiling FreeType and this does not seem to resolve anything.

The root cause of the problem is that the /usr/share/fonts/google-noto-emoji/NotoColorEmoji.ttf font will cause an invalid pixel size exception to be thrown from the FreeType library, the exception is not subsequently handled in the C++ layer, and python will abort.

I am using RedHat 7.6, 7.3 works fine because it does not have the problematic font.

Here is a link to a similar issue using Pillow, issue. I'm not familiar with the code, but my understanding is that the correct solution would be the handle the FreeType exceptions in the ft2font_wrapper.cpp code and not propogate it into the Python interpreter.

Here is part of the stack trace.

#0 0x00007f7862cef207 in raise () from /lib64/libc.so.6
#1 0x00007f7862cf08f8 in abort () from /lib64/libc.so.6
#2 0x00007f784cfb2420 in _Unwind_SetGR (context=, index=, val=)
at /tmp/gcc-v9.3.0p3/gcc.source/libgcc/unwind-dw2.c:1442
#3 0x00007f784cfbe797 in __gcc_personality_v0 (version=, actions=, exception_class=, ue_header=0x15d9bf0,
context=0x7fff105961e0) at /tmp/gcc-v9.3.0p3/gcc.source/libgcc/unwind-c.c:231
#4 0x00007f785a69a7b3 in _Unwind_RaiseException_Phase2 () at /tmp/gcc-v9.3.0p4lnx86/gcc.source/libgcc/unwind.inc:64
#5 0x00007f785a69acf1 in _Unwind_RaiseException (exc=0x15d9bf0) at /tmp/gcc-v9.3.0p4lnx86/gcc.source/libgcc/unwind.inc:136
#6 0x00007f785a74a358 in __cxa_throw () from .../lib/64bit/libstdc++.so.6
#7 0x00007f784cfb12a1 in throw_ft_error(std::__cxx11::basic_string<char, std::char_traits, std::allocator >, int) [clone .cold] ()
from .../lib/python3.9/site-packages/matplotlib-3.6.1-py3.9-linux-x86_64.egg/matplotlib/ft2font.cpython-39-x86_64-linux-gnu.so
#8 0x00007f784cfb9868 in PyFT2Font_init(PyFT2Font*, _object*, _object*) () at /tmp/gcc-v9.3.0p3/gcc.source/libgcc/unwind-dw2.c:1442

Operating system

RedHat 7.6

Matplotlib Version

3.6.1

Matplotlib Backend

Call to getBackend also crashes (same reason as above)

Python version

3.9.16

Jupyter version

No response

Installation

None

@ksunden
Copy link
Member

ksunden commented May 6, 2023

This is somewhat surprising. I can state that my main dev machine has NotoColorEmoji installed and I do not see this.

It is true that non-scalable fonts are not supported by Matplotlib, but there is handling python-side for rejecting such fonts (though that is after calling into the ft2font cpp library).

@QuLogic
Copy link
Member

QuLogic commented May 6, 2023

system_freetype=True

But with what version of FreeType?

@QuLogic
Copy link
Member

QuLogic commented May 6, 2023

#7 0x00007f784cfb12a1 in throw_ft_error(std::__cxx11::basic_string<char, std::char_traits, std::allocator >, int) [clone .cold] ()
from .../lib/python3.9/site-packages/matplotlib-3.6.1-py3.9-linux-x86_64.egg/matplotlib/ft2font.cpython-39-x86_64-linux-gnu.so
#8 0x00007f784cfb9868 in PyFT2Font_init(PyFT2Font*, _object*, _object*) () at /tmp/gcc-v9.3.0p3/gcc.source/libgcc/unwind-dw2.c:1442

The only thing in PyFT2Font_init that I can see that calls throw_ft_error is the FT2Font constructor, but that is wrapped in the CALL_CPP_FULL macro that handles all exceptions, so this seems a system-specific problem.

@john-odonovan
Copy link
Author

The version of FreeType is 2.10.4

matplotlib.ft2font.freetype_version
'2.10.4'
matplotlib.ft2font.freetype_build_type
'system'

Here is the size of the NotoColorEmoji.ttf file for reference

$ ls -l /usr/share/fonts/google-noto-emoji/NotoColorEmoji.ttf
-rw-r--r--. 1 root root 7297112 May 3 2018 /usr/share/fonts/google-noto-emoji/NotoColorEmoji.ttf

I have checked this in 5+ different machines, and the only one which works is the 7.3 machine without the NotoColorEmoji font. All others are using RH7.6. I'll check on some other RH8 machines.

@john-odonovan
Copy link
Author

Some more debugging, but I'm not clear on how this is an issue;

In FT2Font::FT2Font, the call to FT_Set_Char_Size fails with an error code of 23. As a result throw_ft_error will be called.

    error = FT_Set_Char_Size(face, 12 * 64, 0, 72 * (unsigned int)hinting_factor, 72);
    if (error) {
        FT_Done_Face(face);
        throw_ft_error("Could not set the fontsize", error);
    }

throw_ft_error will throw a runtime exception with the following message;

"Could not set the fontsize (invalid pixel size; error code 0x17)"

    throw std::runtime_error(os.str());

FT2Font::FT2Font was called from within the CALL_CPP_FULL macro as @QuLogic pointed out, which does have a try/catch block and explicitly catches std::runtime_error (+ has a catch all as well).

    CALL_CPP_FULL(
        "FT2Font", (self->x = new FT2Font(open_args, hinting_factor, fallback_fonts)),
        Py_CLEAR(self->py_file), -1);

The only strange thing I see is that when these files are compiled, gcc is being used as opposed to g++. It looks like gcc does recognize .cpp as a C++ extension and has no issue compiling the code, but I wonder if there is some impact on exception handling from doing this? I tried forcing g++ by setting CC but this led to other issues with C code that could not be compiled by g++.

@john-odonovan
Copy link
Author

I manually compiled the ft2font related files with g++ and it made no difference, also used GCC 12.2.0 and still a crash.

@john-odonovan
Copy link
Author

I went back to try and figure out the difference between 3.1.3 which was working and 3.4.3/3.6.1, ... It looks like the issue is related to some compiler options.

If I remove the -fvisibility=hidden -flto from the compile line for the objects in the src directory (ft2font.cpp. ft2font_wrapper.cpp), then everything works fine. The old 3.1.3 build did not have these options. I'm not sure what the interaction between these options and exceptions is though.

@john-odonovan
Copy link
Author

The issue seems to be the addition of -flto to the compile line for src/ft2font.cpp. If I remove -flto from this compile line, leaving it on the others, then no crash.

@john-odonovan
Copy link
Author

Hi,

I'm not really going to go much further to see why LTO and exceptions are not playing well together. In the *setup.cfg.template file, there is an entry to disable LTO. Once I do this, the crashes in matplotlib disappear.

    if [ -f setup.cfg.template ];
    then
      sed -e 's/#tests = .*/tests = False/' \
          -e 's/#system_freetype = .*/system_freetype = True/' \
          -e 's/#enable_lto = .*/enable_lto = False/' \
          setup.cfg.template > setup.cfg
    fi
    if [ -f mplsetup.cfg.template ];
    then
      sed -e 's/#tests = .*/tests = False/' \
          -e 's/#system_freetype = .*/system_freetype = True/' \
          -e 's/#enable_lto = .*/enable_lto = False/' \
          mplsetup.cfg.template > mplsetup.cfg
    fi

If someone wants to investigate further, then just setting error = 23 in the FT2Font constructor after the call to FT_Set_Char_Size should be sufficient to replicate the problem, but remove the ~/.cache/matplotlib first to make sure that the code is called.

    error = FT_Set_Char_Size(face, 12 * 64, 0, 72 * (unsigned int)hinting_factor, 72);
    // error = 23;
    if (error) {
        FT_Done_Face(face);
        throw_ft_error("Could not set the fontsize", error);
    }

@ksunden
Copy link
Member

ksunden commented May 8, 2023

What version of GCC do you have?

@ksunden
Copy link
Member

ksunden commented May 8, 2023

Even if I put the error = 23; in, I do not get a core dump.

On import no error happens, though when trying to render text I get a ValueError since no fonts are found since all of them are rejected because of the error.

This is without changing compiler options/etc.

@john-odonovan
Copy link
Author

I'm using GCC 9.3 and I've also tried 12.2. I could try recompiling everything including Python with 12.2 and see what happens.

@ksunden
Copy link
Member

ksunden commented May 8, 2023

I'm on gcc 11.3.

When I use freetype 2.10.1 (the closest version we had SHA hash for already in setupext.py) I do not see any issue.

When I use freetype 2.10.4/2.11.1 I initially got undefined symbol: BrotliDecoderDecompress but after passing --with-brotli=no to the freetype configure, I got the same as 2.10.1. No errors.

Full Diff:
diff --git a/setupext.py b/setupext.py
index a91c681e0f..f981424ad5 100644
--- a/setupext.py
+++ b/setupext.py
@@ -176,13 +176,15 @@ _freetype_hashes = {
         '955e17244e9b38adb0c98df66abb50467312e6bb70eac07e49ce6bd1a20e809a',
     '2.10.1':
         '3a60d391fd579440561bf0e7f31af2222bc610ad6ce4d9d7bd2165bca8669110',
+    '2.10.4':
+        '5eab795ebb23ac77001cfb68b7d4d50b5d6c7469247b0b01b2c953269f658dac',
     '2.11.1':
         'f8db94d307e9c54961b39a1cc799a67d46681480696ed72ecf78d4473770f09b'
 }
 # This is the version of FreeType to use when building a local version.  It
 # must match the value in lib/matplotlib.__init__.py, and the cache path in
 # `.circleci/config.yml`.
-TESTING_VERSION_OF_FREETYPE = '2.6.1'
+TESTING_VERSION_OF_FREETYPE = '2.11.1'
 if sys.platform.startswith('win') and platform.machine() == 'ARM64':
     # older versions of freetype are not supported for win/arm64
     # Matplotlib tests will not pass
@@ -644,7 +646,7 @@ class FreeType(SetupPackage):
             configure = [
                 "./configure", "--with-zlib=no", "--with-bzip2=no",
                 "--with-png=no", "--with-harfbuzz=no", "--enable-static",
-                "--disable-shared"
+                "--disable-shared", "--with-brotli=no",
             ]
             host = sysconfig.get_config_var('HOST_GNU_TYPE')
             if host is not None:  # May be unset on PyPy.
diff --git a/src/ft2font.cpp b/src/ft2font.cpp
index 2e5d6e63b8..f56f65fabd 100644
--- a/src/ft2font.cpp
+++ b/src/ft2font.cpp
@@ -360,6 +360,7 @@ FT2Font::FT2Font(FT_Open_Args &open_args,
     hinting_factor = hinting_factor_;
 
     error = FT_Set_Char_Size(face, 12 * 64, 0, 72 * (unsigned int)hinting_factor, 72);
+    error = 23;
     if (error) {
         FT_Done_Face(face);
         throw_ft_error("Could not set the fontsize", error);

@john-odonovan
Copy link
Author

I recompiled everything with GCC 12.2.0, also checked the top-level Python build to make sure it was linked with g++ (--with-cxx-main on configure commands). I updated to use the system FreeType (2.8.0). I'm not really sure what steps I would take next. For now I will just go with enable_lto=False, to resolve the issue.

@tacaswell
Copy link
Member

Can you reproduce this with 3.7.x or main?

@QuLogic QuLogic added the status: needs clarification Issues that need more information to resolve. label Mar 27, 2024
@jklymak
Copy link
Member

jklymak commented Apr 3, 2024

I'm going to close as an obscure compiler issue, but feel free to ask for a re-open.

@jklymak jklymak closed this as not planned Won't fix, can't repro, duplicate, stale Apr 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status: needs clarification Issues that need more information to resolve.
Projects
None yet
Development

No branches or pull requests

5 participants