Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Possible memory leak in pybind11 migration #28866

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tacaswell opened this issue Sep 23, 2024 · 2 comments
Closed

Possible memory leak in pybind11 migration #28866

tacaswell opened this issue Sep 23, 2024 · 2 comments
Labels
Release critical For bugs that make the library unusable (segfaults, incorrect plots, etc) and major regressions.
Milestone

Comments

@tacaswell
Copy link
Member

Hi @QuLogic, I think I might have found an issue with this commit. I'm not sure if this is the right forum (or if I should create an issue?). I suspect this commit is causing a memory usage growth, even after running the garbage collector. I've only tested it on Windows 10 with python3.11. Can you reproduce this behavior? Details below.

memory_usage_test.py

import gc

from matplotlib.figure import Figure

from numpy.random import rand

import psutil



def plot(num_points):

    fig = Figure()

    ax = fig.add_subplot()

    ax.plot(rand(num_points))

    fig.clear()



def delta_mb_after_plot_and_gc(num_points):

    proc = psutil.Process()

    before_mb = proc.memory_info().rss / 1e6

    plot(num_points)

    gc.collect()

    after_mb = proc.memory_info().rss / 1e6

    delta_mb = after_mb - before_mb

    print(f"plotting {num_points} points leaks {delta_mb:.3f} MB")



def main():

    gc.disable()

    delta_mb_after_plot_and_gc(100_000)

    delta_mb_after_plot_and_gc(1_000_000)

    delta_mb_after_plot_and_gc(10_000_000)



if __name__ == "__main__":

    main()

Running it against this commit and the previous commit (I found this commit via git bisect)


(mpl_venv) PS C:\Users\justi\projects\matplotlib_ref_cycle\matplotlib> git checkout 597554db667344c4c4ad026ec1c6dd5f4c688d8f

HEAD is now at 597554db66 Add a pybind11 type caster for GCAgg and its requirements



(mpl_venv) PS C:\Users\justi\projects\matplotlib_ref_cycle\matplotlib> python ..\memory_usage_test.py

plotting 100000 points leaks 1.978 MB

plotting 1000000 points leaks 16.077 MB

plotting 10000000 points leaks 159.719 MB



(mpl_venv) PS C:\Users\justi\projects\matplotlib_ref_cycle\matplotlib> python ..\memory_usage_test.py

plotting 100000 points leaks 1.970 MB

plotting 1000000 points leaks 16.081 MB

plotting 10000000 points leaks 159.625 MB



(mpl_venv) PS C:\Users\justi\projects\matplotlib_ref_cycle\matplotlib> git checkout a47e26bd8583f0fcacc1dd83c3f2ac39b1f7a091

Previous HEAD position was 597554db66 Add a pybind11 type caster for GCAgg and its requirements

HEAD is now at a47e26bd85 Add a pybind11 type caster for agg::rgba



(mpl_venv) PS C:\Users\justi\projects\matplotlib_ref_cycle\matplotlib> python ..\memory_usage_test.py

plotting 100000 points leaks 0.303 MB

plotting 1000000 points leaks 0.041 MB

plotting 10000000 points leaks -0.279 MB



(mpl_venv) PS C:\Users\justi\projects\matplotlib_ref_cycle\matplotlib> python ..\memory_usage_test.py

plotting 100000 points leaks 0.197 MB

plotting 1000000 points leaks 0.041 MB

plotting 10000000 points leaks -0.311 MB

Environment Details


(mpl_venv) PS C:\Users\justi\projects\matplotlib_ref_cycle\matplotlib> python --version

Python 3.11.9



(mpl_venv) PS C:\Users\justi\projects\matplotlib_ref_cycle\matplotlib> [System.Environment]::OSVersion.Version

Major  Minor  Build  Revision

-----  -----  -----  --------

10     0      19045  0



(mpl_venv) PS C:\Users\justi\projects\matplotlib_ref_cycle\matplotlib> pip freeze | findstr matplotlib

-e git+https://github.com/justinjhendrick/matplotlib.git@a47e26bd8583f0fcacc1dd83c3f2ac39b1f7a091#egg=matplotlib

matplotlib-inline==0.1.7

Originally posted by @justinjhendrick in 597554d

@tacaswell tacaswell added the Release critical For bugs that make the library unusable (segfaults, incorrect plots, etc) and major regressions. label Sep 23, 2024
@tacaswell tacaswell added this to the v3.10.0 milestone Sep 23, 2024
@QuLogic
Copy link
Member

QuLogic commented Sep 23, 2024

I'm pretty sure this has to do with the extra refs added here, but as those fixed a crash, I don't want to just remove them.

However, the final cleanup in #28856 makes it so that pybind11 tracks everything for us (removing the extra explicit ref), and I believe it also fixes the leak, as your test is pretty consistently smaller:

$ python ../mpl-tests/memory_usage_test.py 
plotting 100000 points leaks 1.753 MB
plotting 1000000 points leaks 14.402 MB
plotting 10000000 points leaks 0.111 MB
$ python ../mpl-tests/memory_usage_test.py 
plotting 100000 points leaks 1.855 MB
plotting 1000000 points leaks 14.402 MB
plotting 10000000 points leaks -0.082 MB
$ python ../mpl-tests/memory_usage_test.py 
plotting 100000 points leaks 1.806 MB
plotting 1000000 points leaks 14.402 MB
plotting 10000000 points leaks 0.008 MB
$ python ../mpl-tests/memory_usage_test.py 
plotting 100000 points leaks 1.827 MB
plotting 1000000 points leaks 14.402 MB
plotting 10000000 points leaks 0.000 MB

Though I'm not sure why the increase for 1000000 still.

@ksunden
Copy link
Member

ksunden commented Oct 9, 2024

@QuLogic

My understanding is that this was fixed by #28856, which has now been merged (but didn't link to here). If that is incorrect, please feel free to reopen.

@ksunden ksunden closed this as completed Oct 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Release critical For bugs that make the library unusable (segfaults, incorrect plots, etc) and major regressions.
Projects
None yet
Development

No branches or pull requests

3 participants