Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Array object is not garbage collected #1176

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jcbsv opened this issue Jul 2, 2020 · 2 comments
Closed

Array object is not garbage collected #1176

jcbsv opened this issue Jul 2, 2020 · 2 comments

Comments

@jcbsv
Copy link

jcbsv commented Jul 2, 2020

Environment

  • Pythonnet version: 2.5.1
  • Python version: 3.8.2
  • Operating System: Windows 10 (.Net 4.7.2, 64bit)

Details

I use the technique described here to convert a Numpy ndarray to .Net Array and use the resulting array in C# code. However, the array is not garbage collected, as the Python interpreter keeps a reference on the array (Python.Runtime.CLRObject).

I have included an example below, where each step in the iteration increases the memory usage by about 8MB, since the old arrays are not garbage collected.

I have tried using the Dispose() method on the array, and forcing the Python garbage collector (as suggested in other issues), but to no avail. I have also tried 100 other things, but there is no to re-iterate all the failures here.

Not sure where to go from here. Any suggestions are welcome!

Example code

C# code Program.cs:

using System;
using Python.Runtime;

namespace ConsoleApp1
{
    class Program
    {
        private dynamic module; 
        public Program()
        {
            Environment.SetEnvironmentVariable("PYTHONPATH", Environment.CurrentDirectory, EnvironmentVariableTarget.Process);
            PythonEngine.Initialize();

            using (Py.GIL())
            {
                module = Py.Import("testmodule");
            }
        }

        ~Program()
        {
            module = null;
            PythonEngine.Shutdown();
        }

        Array Step()
        {
            Array a;
            using (Py.GIL())
            {
                a = module.step();
            }
            return a;
        }

        public void Iterate()
        {
            Array a;
            for(int i=0; i<100; i++)
            {
                a = Step();
            }
        }

        static void Main(string[] args)
        {
            Program p = new Program();
            p.Iterate();
        }
    }
}

Python code testmodule.py:

import numpy as np

import ctypes
import System
from System.Runtime.InteropServices import GCHandle, GCHandleType

#
# C# <-> Numpy conversion adopted from
# https://github.com/pythonnet/pythonnet/issues/514#issuecomment-350375105

_MAP_NP_NET = {
    np.dtype('float32'): System.Single,
    np.dtype('float64'): System.Double,
    np.dtype('int8'): System.SByte,
    np.dtype('int16'): System.Int16,
    np.dtype('int32'): System.Int32,
    np.dtype('int64'): System.Int64,
    np.dtype('uint8'): System.Byte,
    np.dtype('uint16'): System.UInt16,
    np.dtype('uint32'): System.UInt32,
    np.dtype('uint64'): System.UInt64,
    np.dtype('bool'): System.Boolean,
}

def asNetArray(npArray):
    """Given a `numpy.ndarray` returns a CLR `System.Array`.  See _MAP_NP_NET for
    the mapping of Numpy dtypes to CLR types.
    """
    dims = npArray.shape
    dtype = npArray.dtype

    if not npArray.flags.c_contiguous:
        npArray = npArray.copy(order='C')
    assert npArray.flags.c_contiguous

    try:
        if npArray.ndim == 1:
            netArray = System.Array.CreateInstance(_MAP_NP_NET[dtype], dims[0])
        elif npArray.ndim == 2:
            netArray = System.Array.CreateInstance(
                _MAP_NP_NET[dtype], dims[0], dims[1])
        elif npArray.ndim == 3:
            netArray = System.Array.CreateInstance(
                _MAP_NP_NET[dtype], dims[0], dims[1], dims[2])
        else:
            raise NotImplementedError(
                "asNetArray does not support arrays of ndim={}".format(
                    npArray.ndim))
    except KeyError:
        raise NotImplementedError(
            "asNetArray does not yet support dtype {}".format(dtype))

    destHandle = None
    try:
        # Memmove
        destHandle = GCHandle.Alloc(netArray, GCHandleType.Pinned)
        sourcePtr = npArray.__array_interface__['data'][0]
        destPtr = destHandle.AddrOfPinnedObject().ToInt64()
        ctypes.memmove(destPtr, sourcePtr, npArray.nbytes)
    finally:
        if destHandle is not None and destHandle.IsAllocated:
            destHandle.Free()
    return netArray


def step():
    a = np.random.random(1000000)
    return asNetArray(a)
@lostmsu
Copy link
Member

lostmsu commented Jul 2, 2020

I don't see any code in your example, that would necessarily cause C# garbage collector to run. .NET does not immediately deallocate objects, which have no references to them. It only does so under memory pressure.

@jcbsv
Copy link
Author

jcbsv commented Jul 2, 2020

Ok, maybe this is just due to my inexperience with the .Net garbage collector. It seemed weird to me that the program is allowed allocate many GBs of memory in chunks of 8MB to keep objects that are not referenced. The garbage collector does eventually get around to clean up.

@jcbsv jcbsv closed this as completed Jul 2, 2020
@hbldh hbldh mentioned this issue Aug 20, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lostmsu @jcbsv