Reset the type slots (split from #958) #1016

BadSingleton · 2019-12-17T18:59:12Z

On shutdown, clear the managed Types' slots.

What does this implement/fix? Explain your changes.

This implements part of #958, where the managed type slots are reset on shutdown.

Does this close any currently open issues?

Not yet.

Any other comments?

Most of the code was originally written by @amos402

Checklist

Check all those that are applicable and complete.

Make sure to include one or more tests for your change
Not really testable at the moment. Once the soft shutdown is implemented, tests can be written.
If an enhancement PR, please create docs and at best an example
Add yourself to AUTHORS
Updated the CHANGELOG

On shutdown, clear the managed Types' slots.

lostmsu · 2019-12-20T19:15:51Z

src/runtime/interop.cs

+        }
+    }
+
+    [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi)]


NIT: none of the fields are declared strings, so why does CharSet matter?

It doesn't, I'll remove it.

lostmsu · 2019-12-20T19:17:06Z

src/runtime/metatype.cs

+                SlotsHolder.ReleaseTypeSlots(PyCLRMetaType);
+            }
+            Runtime.XDecref(PyCLRMetaType);
+            PyCLRMetaType = IntPtr.Zero;


Since you are setting PyCLRMetaType to 0, I think it would be a good idea to check for it in the beginning of the method, and throw something like ObjectDisposedException

lostmsu · 2019-12-20T19:17:55Z

src/runtime/metatype.cs

@@ -22,6 +22,15 @@ public static IntPtr Initialize()
            return PyCLRMetaType;
        }

+        public static void Release()
+        {
+            if (Runtime.Refcount(PyCLRMetaType) > 1)


Not sure I understand the reasoning behind > 1 condition.
In fact, why even use Refcount? This class should have its own way to track if PyCLRMetaType is allocated.

I think the reasoning is the same as with the similar pattern in typemanager.cs#38. The only way I can see this class tracking it's allocation is to look if the PyCLRMetaType is IntPtr.Zero or not.

The logic is: If the refcount is 1, then at line 31 when decref, it'll go to zero and the entire type will be released. In that case we don't care about clearing the slots. It's a standard optimization in a ref-counting framework.

It's correct assuming that ReleaseTypeSlots doesn't also release refcounts that otherwise won't be released.

That said: Do we have coverage on line 29? I'm not sure I really understand under what circumstance it happens.

I'd put the expression into bool isLastReference (don't miss the negations) or at least add a comment.

lostmsu · 2019-12-20T19:22:47Z

src/runtime/typemanager.cs

+                IntPtr ret1 = NativeCodePage + native.Return1;
+                IntPtr ret0 = NativeCodePage + native.Return0;
+                InitializeSlot(type, ret0, "tp_traverse", false);
+                InitializeSlot(type, ret0, "tp_clear", false);


NIT: can you add parameter name to false here? (e.g. paramName: false) Otherwise it is unclear what it means.

lostmsu · 2019-12-20T19:28:02Z

src/runtime/typemanager.cs

-        /// <param name="name">Name of the method.</param>
-        static void InitializeSlot(IntPtr type, IntPtr slot, string name)
+        /// <param name="canOverride">Can override the slot when it existed</param>
+        static void InitializeSlot(IntPtr type, IntPtr slot, string name, bool canOverride = true)


Can you explain why is it OK sometimes to not overwrite a slot?

I see this is used for tp_traverse and tp_clear. What are consequences of not overwriting this slot? Would not this cause our cleanup code to simply be skipped?

I think it relates to other parts of #958 where some classes define their own tp_traverse and tp_clear. This could be removed for now

lostmsu · 2019-12-20T19:45:55Z

src/runtime/typemanager.cs

+            }
+            _alredyReset = true;
+            IntPtr tp_name = Marshal.ReadIntPtr(_type, TypeOffset.tp_name);
+            string typeName = Marshal.PtrToStringAnsi(tp_name);


Judging by the code around typemanager.cs:430, the value of tp_name can well be a Unicode string. In Python3 they clearly use UTF8.

This variable isn't used. We should remove it (and tp_name), and also remove the commented-out use at line 1055.

Just some debug code I left...

lostmsu · 2019-12-20T19:48:02Z

src/runtime/typemanager.cs

+            IntPtr tp_base = Marshal.ReadIntPtr(_type, TypeOffset.tp_base);
+            Runtime.XDecref(tp_base);
+            Marshal.WriteIntPtr(_type, TypeOffset.tp_base, IntPtr.Zero);
+
+            IntPtr tp_bases = Marshal.ReadIntPtr(_type, TypeOffset.tp_bases);
+            Runtime.XDecref(tp_bases);
+            Marshal.WriteIntPtr(_type, TypeOffset.tp_bases, IntPtr.Zero);


Is overwriting base types in a type object a defined behavior in Python?
Why do we need this?

Is overwriting base types in a type object a defined behavior in Python?

It should, both of these values are still valid if NULL.
As to why, this is the reason of this PR. On domain reload (and unload) we clobber the Python objects that points to C#, leaving empty, but still valid, Python objects that no longer refer to code that is now gone.

lostmsu · 2019-12-20T19:48:41Z

src/runtime/typemanager.cs

+            Marshal.WriteIntPtr(_type, TypeOffset.mp_length, IntPtr.Zero);
+        }
+
+        private static void OnDestruct(IntPtr ob)


Is ob a Python type here? Perhaps then name as such.

Here and below, ob should be tp

src/runtime/typemanager.cs

lostmsu · 2019-12-20T19:52:34Z

src/runtime/typemanager.cs

+            if (offset == TypeOffset.tp_clear
+                || offset == TypeOffset.tp_traverse)
+            {
+                return TypeManager.NativeCodePage + TypeManager.NativeCode.Active.Return0;


Maybe I missed a conversation. What is the story for ARM in this code?

I'm not sure if this is what you're looking for, but it seems that it was removed in #913

benoithudson · 2020-01-06T19:00:00Z

FYI: Felix is back from holiday on Monday.

benoithudson · 2020-01-14T21:02:50Z

src/runtime/typemanager.cs

@@ -311,6 +337,24 @@ internal static IntPtr WriteMethodDefSentinel(IntPtr mdef)
            return WriteMethodDef(mdef, IntPtr.Zero, IntPtr.Zero, 0, IntPtr.Zero);
        }

+        internal static void FreeMethodDef(IntPtr mdef)
+        {
+            unsafe


Have we verified this still works in Unity with default options? Default Unity settings block unsafe code. I think it's just at compile-time, so unsafe compiled code is fine, but I want to make sure.

Yes, it does, there are other uses of unsafe code in the project. (XDecref for instance)

benoithudson · 2020-01-14T21:04:05Z

src/runtime/typemanager.cs

+                    {
+                        Runtime.PyErr_Print();
+                    }
+                    FreeMethodDef(mdefAddr);


Why do we open a scope and create a new local variable that just copies a local variable that already exists?

Oh, I get it -- so that the lambda grabs the right value.

Feels like we could refactor these two scopes into a function, so it's a bit less mysterious.

benoithudson · 2020-01-14T21:21:00Z

src/runtime/typemanager.cs

-
-            InitializeSlot(type, ret0, "tp_traverse");
-            InitializeSlot(type, ret0, "tp_clear");
-            InitializeSlot(type, ret1, "tp_is_gc");


Why does this work without tp_is_gc? It used to be required (I didn't really fully understand it, but leaving it out led to crashes).

If we actually don't need it then we can remove everything about Return1 from the code.

It was probably an oversight, and I added it back.

benoithudson · 2020-01-14T21:21:56Z

src/runtime/typemanager.cs

+                    if (slotsHolder != null)
+                    {
+                        slotsHolder.Add(offset, thunkRet0);
+                    }


Why inline InitializeSlot here?

Can't find a good reason, will uninline

benoithudson · 2020-01-14T21:54:12Z

src/runtime/typemanager.cs

+                return;
+            }
+            Marshal.WriteIntPtr(type, offset, thunk.Address);
+            slotsHolder.Add(offset, thunk);


won't this throw with the default slotsHolder?

Seems like we should remove the function with one less arg, and handle null here.

benoithudson · 2020-01-14T21:58:30Z

src/runtime/typemanager.cs

+        public const string HolderKeyName = "_slots_holder";
+        public delegate void Resetor(IntPtr type, int offset);
+
+        private GCHandle _handle;


The C# world tends to either use m_ or nothing at all to mark instance variables (I think that's the python.net tradition) -- but not just a leading underscore.

benoithudson · 2020-01-14T21:59:59Z

src/runtime/typemanager.cs

+        private IntPtr _capsule;
+        private IntPtr _type;
+        private Dictionary<int, ThunkInfo> _slots = new Dictionary<int, ThunkInfo>();
+        private List<Delegate> _keepalive = new List<Delegate>();


Does this actually get used anywhere? I see Add and Clear called on it but nothing else.

If it's just a place to park a pointer to avoid getting GC'd then the KeeapAlive (!) function should document it.

benoithudson · 2020-01-14T22:02:05Z

src/runtime/typemanager.cs

+            _deallocators.Add(deallocate);
+        }
+
+        public void KeeapAlive(Delegate d)


The spelling is a bit odd?

A comment about what this does would be good.

benoithudson · 2020-01-14T22:03:27Z

src/runtime/typemanager.cs

+
+        private void ResetSlots()
+        {
+            if (_alredyReset)


typo: alreadyReset

benoithudson · 2020-01-14T22:12:12Z

src/runtime/typemanager.cs

+            Marshal.WriteIntPtr(_type, TypeOffset.tp_bases, IntPtr.Zero);
+
+            // Because __len__ may have a "default" implementation, not registered to us
+            // always explicitly clobber it. See mp_lenth_slot class


Is there a typo in mp_length_slot?

No need to handle mp_length manually here, it already included in common case.

benoithudson · 2020-01-14T22:13:22Z

src/runtime/typemanager.cs

+            else if (offset == TypeOffset.tp_dealloc)
+            {
+                // tp_free of PyTypeType is point to PyObejct_GC_Del.
+                return Marshal.ReadIntPtr(Runtime.PyTypeType, TypeOffset.tp_free);


PyObejct_GC_Del has a typo

benoithudson · 2020-01-14T22:18:55Z

src/runtime/typemanager.cs

+            return (SlotsHolder)handle.Target;
+        }
+
+        private static IntPtr GetDefaultSlot(int offset)


I think the doc here should say that we're looking for the C function to call when resetting the slot at the given offset within the type.

We might want to eventually consider having those functions throw exceptions e.g. "the underlying .NET type has been deleted"

Problem is: with domain gone, who will provide those functions?

…onnet into reset-type-slots

Manipulate SlotsHolder manually instead of Capsule mechanism

benoithudson · 2020-01-24T19:44:18Z

src/runtime/metatype.cs

@@ -11,17 +11,30 @@ namespace Python.Runtime
    internal class MetaType : ManagedType
    {
        private static IntPtr PyCLRMetaType;
-
+        private static SlotsHolder _metaSlotsHodler;


I just noticed the typo here. This is holding slots, not bitcoin.

benoithudson · 2020-01-24T19:44:59Z

src/runtime/methodwrapper.cs

@@ -13,7 +13,6 @@ internal class MethodWrapper
        public IntPtr mdef;
        public IntPtr ptr;
        private bool _disposed = false;
-


if you add this empty line back in you remove an entire file from the squash-commit.

lostmsu

Please, update to master

lostmsu · 2020-02-12T22:47:11Z

src/runtime/interop.cs

@@ -537,4 +537,13 @@ public ThunkInfo(Delegate target)
            Address = Marshal.GetFunctionPointerForDelegate(target);
        }
    }
+
+    [StructLayout(LayoutKind.Sequential)]
+    struct PyMethodDef


Please, add the following link as a comment: https://docs.python.org/3/c-api/structures.html#c.PyMethodDef

lostmsu · 2020-02-12T22:49:47Z

src/runtime/metatype.cs

@@ -22,6 +22,15 @@ public static IntPtr Initialize()
            return PyCLRMetaType;
        }

+        public static void Release()
+        {
+            if (Runtime.Refcount(PyCLRMetaType) > 1)


I'd put the expression into bool isLastReference (don't miss the negations) or at least add a comment.

lostmsu · 2020-02-12T22:52:07Z

src/runtime/typemanager.cs

-        private static Dictionary<Type, IntPtr> cache;
+        private const BindingFlags tbFlags = BindingFlags.Public | BindingFlags.Static;
+        private static readonly Dictionary<Type, IntPtr> cache = new Dictionary<Type, IntPtr>();
+        private static readonly Dictionary<IntPtr, SlotsHolder> _slotsHolders = new Dictionary<IntPtr, SlotsHolder>();

        static TypeManager()


Remove empty static constructor

lostmsu · 2020-02-12T22:56:53Z

src/runtime/typemanager.cs

            mdef = WriteMethodDefSentinel(mdef);

            Marshal.WriteIntPtr(type, TypeOffset.tp_methods, mdefStart);
+            slotsHolder.Set(TypeOffset.tp_methods, (t, offset) =>


NIT: name lambda parameter for clarity

lostmsu · 2020-02-12T22:59:35Z

src/runtime/typemanager.cs

+            return offset;
+        }
+
+        static bool IsSlotSet(IntPtr type, string name)


NIT: make an overload, taking offset, and use it here, and in InitializeSlot above

lostmsu · 2020-02-12T23:00:11Z

src/runtime/typemanager.cs

+        }
+    }
+
+    class SlotsHolder


Put this class into a separate file

lostmsu · 2020-02-12T23:04:52Z

src/runtime/typemanager.cs

+            return (SlotsHolder)handle.Target;
+        }
+
+        private static IntPtr GetDefaultSlot(int offset)


Problem is: with domain gone, who will provide those functions?

lostmsu · 2020-02-12T23:06:28Z

src/runtime/typemanager.cs

+    }
+
+
+    static class SlotHelper


Does not seem to be used anywhere

BadSingleton · 2020-02-19T17:50:41Z

Work has stopped on our side on this, so I'm closing the PR. We might get back to it eventually.

BadSingleton and others added 2 commits December 17, 2019 13:42

Reset the type slots

118c21a

On shutdown, clear the managed Types' slots.

Merge branch 'master' into reset-type-slots

953b5b5

lostmsu requested changes Dec 20, 2019

View reviewed changes

benoithudson reviewed Jan 14, 2020

View reviewed changes

Merge remote-tracking branch 'upstream/master' into reset-type-slots

515b227

benoithudson reviewed Jan 14, 2020

View reviewed changes

BadSingleton and others added 6 commits January 17, 2020 15:53

(WIP) merge and code review fixes

f5c53a1

Merge branch 'reset-type-slots' of ssh://github.com/BadSingleton/pyth…

e0960d0

…onnet into reset-type-slots

Reset a tuple for tp_bases

61dbb8d

Record mp_length slot

03674d2

Partial cherry pick of 433d0f6

e7effd8

Manipulate SlotsHolder manually instead of Capsule mechanism

Code review fixes

8baad6a

BadSingleton force-pushed the reset-type-slots branch from 4ea1a6f to 8baad6a Compare January 21, 2020 18:53

benoithudson reviewed Jan 24, 2020

View reviewed changes

lostmsu reviewed Feb 12, 2020

View reviewed changes

BadSingleton closed this Feb 19, 2020

Reset the type slots (split from #958) #1016

Reset the type slots (split from #958) #1016

Conversation

BadSingleton commented Dec 17, 2019

What does this implement/fix? Explain your changes.

Does this close any currently open issues?

Any other comments?

Checklist

lostmsu Dec 20, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lostmsu Dec 20, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benoithudson commented Jan 6, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benoithudson Jan 14, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lostmsu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BadSingleton commented Feb 19, 2020

lostmsu Dec 20, 2019 •

edited

Loading

lostmsu Dec 20, 2019 •

edited

Loading

benoithudson Jan 14, 2020 •

edited

Loading