Exception ignored in tp_clear of: <class 'memoryview'> #110408

wg-postalgia · 2023-10-05T16:56:38Z

Bug report

Bug description:

Searching this error the only reference I find was a previous conversation where this issue was considered theoretical and not reproducible.

I made a small script which can reproduce the error.
It's really frustrating to do, because the error isn't caught by the debugger, so it's really hard to figure out what was triggering the error, since the Traceback is seemingly random. I slowly pruned my code away to leave this minimal case.

The main code iterates through a PDF file, implemented as an iterator that returns the next page of the PDF (via poppler) as an image at a given DPI. For each page, a Future is created with ProcessPoolExecutor which delays and returns.

Both the iterator through poppler AND the futures are required.

I have solved the problem in my code by removing the iterator, but spent the time to create this reproduction in case it helps someone track down this bug.

Prerequisite: https://pdf2image.readthedocs.io/en/latest/installation.html

import functools
import threading
import time
from pdf2image import convert_from_path, pdfinfo_from_path
from concurrent.futures import ProcessPoolExecutor


class Pdf2ImageIterator:

    def __init__(self, pdf_path: str, poppler_path: str, dpi_high: int):
        self._path = pdf_path
        self._dpi_high = dpi_high
        self._poppler_path = poppler_path
        self.count = pdfinfo_from_path(self._path, None, None, poppler_path=self._poppler_path)["Pages"]

    def __iter__(self):
        self._page = 1
        return self

    def __next__(self):
        try:
            page_h = convert_from_path(self._path, poppler_path=self._poppler_path, dpi=self._dpi_high,
                                        first_page=self._page, last_page=self._page)
            self._page += 1
            return page_h
        except Exception as e:
            print(f"Pdf2Images {e}")


def my_done_callback(i, image, future):
    print(f"my_done_callback on {threading.get_ident()}")


def my_future(i, image):
    print(f"my_future on {threading.get_ident()}")
    time.sleep(1)
    return 0


if __name__ == '__main__':
    ppe = ProcessPoolExecutor(None)

    poppler_path = r"C:\U<YOUR-PATH>\poppler-23.01.0\Library\bin"
    pdf = r"<PDF file with multiple pages>"
    pages = Pdf2ImageIterator(pdf, poppler_path, 200)

    for i, im in enumerate(pages):
        future = ppe.submit(my_future, i, im)
        future.add_done_callback(functools.partial(my_done_callback, i, im))

    print("Done")

Example console logs:

Exception ignored in tp_clear of: <class 'memoryview'>
Traceback (most recent call last):
  File "...\plugins\python-ce\helpers\pydev\pydevd_tracing.py", line 56, in _internal_set_trace
    filename = frame.f_back.f_code.co_filename.lower()
BufferError: memoryview has 1 exported buffer

Exception ignored in tp_clear of: <class 'memoryview'>
Traceback (most recent call last):
  File "...Programs\Python\Python310\lib\threading.py", line 568, in set
    with self._cond:
BufferError: memoryview has 1 exported buffer

Exception ignored in tp_clear of: <class 'memoryview'>
Traceback (most recent call last):
  File "...\plugins\python-ce\helpers\pydev\pydevd_tracing.py", line 56, in _internal_set_trace
    filename = frame.f_back.f_code.co_filename.lower()
BufferError: memoryview has 1 exported buffer

Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)] on win32

CPython versions tested on:

3.10

Operating systems tested on:

Windows

gaogaotiantian · 2023-10-05T17:29:47Z

Are you able to reproduce this without the debugger (using pure python script in cmdline like python yourscript.py)? I'm assuming you are using PyCharm or some other IDEs because the exception was raised from pydev.

wg-postalgia · 2023-10-05T19:11:51Z

I am using PyCharm and cannot reproduce without the debugger.
Does that mean it's a debugger bug? I'm not sure where to re-make this report to, any suggestions?
Thank you

gaogaotiantian · 2023-10-05T19:58:52Z

I believe this is a dup of #77894. I think this is still a CPython issue, the debugger should not be able to do anything to crash Python. However, without a simple repro that involves "only CPython", it's not easy to track down the issue. It requires plenty of work here to understand the problem.

wg-postalgia · 2023-10-05T20:18:32Z

Agreed, yes that's the original conversation I referenced - sorry I didn't have the correct migrated link.
Happy to close this as duplicate and leave for anyone who feels like investigating this in the future?

It's kind of fun to reproduce a bug that was only theoretical - at least now that I have a fix anyways :)

cyc1111111111 · 2023-10-24T12:39:18Z

同意，是的，这就是我引用的原始对话 - 抱歉，我没有正确的迁移链接。很高兴将其作为重复项关闭并留给任何想要将来调查此问题的人吗？

重现一个理论上的错误是很有趣的 - 至少现在我已经修复了:)

Have you solved this problem? I had the same problem

wg-postalgia · 2023-10-24T13:13:49Z

I thought my problem was solved, but it seems to be a timing issue and the problem still occurs randomly. When it does, the debugger throws the exception and all of the ProcessPool processes have to be stopped manually with Task Manager. I have stopped using the debugger which does completely resolve the issue. For debugging, you can try to set the number of CPUs to 1, that seemed to work for me.

…

On Tue, Oct 24, 2023 at 8:39 AM cyc1111111111 ***@***.***> wrote: 同意，是的，这就是我引用的原始对话 - 抱歉，我没有正确的迁移链接。很高兴将其作为重复项关闭并留给任何想要将来调查此问题的人吗？重现一个理论上的错误是很有趣的 - 至少现在我已经修复了:) Have you solved this problem? I had the same problem — Reply to this email directly, view it on GitHub <#110408 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A5L4UIAQMXTK5KMEBFUIRATYA6ZIFAVCNFSM6AAAAAA5USFJE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZXGEZDMNRRGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

ForeverFerret · 2024-01-12T01:10:12Z

ProcessPoolExecutor || multiprocessing.Pool

谢谢您! 您的回答对我同样有帮助!
经我测试, 我使用的Pycharm专业版2023.3.2版本会有这个问题, 我又接着尝试了2022.2.5和2022.1.4也都有同样的问题!
但使用VSCode调试则完全没有问题, 所以我怀疑是pycharm对dev做了特别的处理导致了问题

dnparadice · 2024-02-23T17:04:37Z

+1 this is annoying, debugger should handle multiprocessing more elegantly

tobiaswuerth · 2024-05-22T21:49:14Z

+1

Andre1122 · 2025-02-18T05:41:55Z

I recently replaced my computer and switched the CPU from Core to AMD The same code encountered this issue. Setting CPU to 1 can solve the problem. I want to know if this error is caused by AMD CPUs I am considering whether to replace the computer again.

wg-postalgia · 2025-03-07T13:35:02Z

No it happens on my "11th Gen Intel(R) Core(TM) i5-1135G7 " so not AMD only.

devdanzin · 2025-03-10T11:37:10Z

Can you reproduce the issue in main? Seems related to #127085, so it might have been fixed recently.

Also maybe #77894.

tkohegyi · 2025-04-15T14:37:45Z

I face with Exception ignored in tp_clear of: <class 'memoryview'> error many times when I run threads in python code and try to debug it:

Exception ignored in tp_clear of: <class 'memoryview'>
Traceback (most recent call last):
File "", line 1, in
BufferError: memoryview has 1 exported buffer

Process finished with exit code -1073741819 (0xC0000005)

This actually means debug is not possible.

dnparadice · 2025-04-15T14:57:49Z

This issue is resolved in Python 3.12

tkohegyi · 2025-04-15T17:14:53Z

Sorry @dnparadice, but I experienced this with Python 3.12.

dnparadice · 2025-04-15T17:35:30Z

Sorry @dnparadice, but I experienced this with Python 3.12.

That is too bad, I used to see this exception regularly and have not seen it since updating to Python 3.12.
Maybe post the code that causes the exception here.

wg-postalgia added the type-bug An unexpected behavior, bug, or error label Oct 5, 2023

iritkatriel added the interpreter-core (Objects, Python, Grammar, and Parser dirs) label Nov 28, 2023

devdanzin mentioned this issue Apr 15, 2025

Aborts from working with memoryviews and buffers across threads in free-threading build #132565

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Exception ignored in tp_clear of: <class 'memoryview'> #110408

Exception ignored in tp_clear of: <class 'memoryview'> #110408

wg-postalgia commented Oct 5, 2023 •

edited by AlexWaygood

Loading

gaogaotiantian commented Oct 5, 2023

Uh oh!

wg-postalgia commented Oct 5, 2023

Uh oh!

gaogaotiantian commented Oct 5, 2023

Uh oh!

wg-postalgia commented Oct 5, 2023

Uh oh!

cyc1111111111 commented Oct 24, 2023

Uh oh!

wg-postalgia commented Oct 24, 2023 via email

Uh oh!

ForeverFerret commented Jan 12, 2024

Uh oh!

dnparadice commented Feb 23, 2024

Uh oh!

tobiaswuerth commented May 22, 2024

Uh oh!

Andre1122 commented Feb 18, 2025

Uh oh!

wg-postalgia commented Mar 7, 2025

Uh oh!

devdanzin commented Mar 10, 2025 •

edited

Loading

Uh oh!

tkohegyi commented Apr 15, 2025

Uh oh!

dnparadice commented Apr 15, 2025

Uh oh!

tkohegyi commented Apr 15, 2025

Uh oh!

dnparadice commented Apr 15, 2025

Uh oh!

Uh oh!

Exception ignored in tp_clear of: <class 'memoryview'> #110408

Exception ignored in tp_clear of: <class 'memoryview'> #110408

Comments

wg-postalgia commented Oct 5, 2023 • edited by AlexWaygood Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

gaogaotiantian commented Oct 5, 2023

Uh oh!

wg-postalgia commented Oct 5, 2023

Uh oh!

gaogaotiantian commented Oct 5, 2023

Uh oh!

wg-postalgia commented Oct 5, 2023

Uh oh!

cyc1111111111 commented Oct 24, 2023

Uh oh!

wg-postalgia commented Oct 24, 2023 via email

Uh oh!

ForeverFerret commented Jan 12, 2024

Uh oh!

dnparadice commented Feb 23, 2024

Uh oh!

tobiaswuerth commented May 22, 2024

Uh oh!

Andre1122 commented Feb 18, 2025

Uh oh!

wg-postalgia commented Mar 7, 2025

Uh oh!

devdanzin commented Mar 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tkohegyi commented Apr 15, 2025

Uh oh!

dnparadice commented Apr 15, 2025

Uh oh!

tkohegyi commented Apr 15, 2025

Uh oh!

dnparadice commented Apr 15, 2025

Uh oh!

wg-postalgia commented Oct 5, 2023 •

edited by AlexWaygood

Loading

devdanzin commented Mar 10, 2025 •

edited

Loading