Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Reset the type slots (split from #958) #1016

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
BadSingleton wants to merge 9 commits into from
Closed

Reset the type slots (split from #958) #1016

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
118c21a
Reset the type slots
BadSingleton Dec 17, 2019
953b5b5
Merge branch 'master' into reset-type-slots
filmor Dec 18, 2019
515b227
Merge remote-tracking branch 'upstream/master' into reset-type-slots
BadSingleton Jan 14, 2020
f5c53a1
(WIP) merge and code review fixes
BadSingleton Jan 17, 2020
e0960d0
Merge branch 'reset-type-slots' of ssh://github.com/BadSingleton/pyth…
BadSingleton Jan 17, 2020
61dbb8d
Reset a tuple for tp_bases
amos402 Dec 26, 2019
03674d2
Record mp_length slot
amos402 Dec 23, 2019
e7effd8
Partial cherry pick of 433d0f6
amos402 Jan 18, 2020
8baad6a
Code review fixes
Jan 21, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Code review fixes
  • Loading branch information
Félix Bourbonnais committed Jan 21, 2020
commit 8baad6a88a94cbbdd19ef4ab07875ede07f3eaa9
4 changes: 4 additions & 0 deletions src/runtime/metatype.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,10 @@ public static IntPtr Initialize()

public static void Release()
{
if (PyCLRMetaType == IntPtr.Zero)
{
throw new ObjectDisposedException("PyCLRMetaType");
}
if (Runtime.Refcount(PyCLRMetaType) > 1)
Copy link
Member

@lostmsu lostmsu Dec 20, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I understand the reasoning behind > 1 condition.
In fact, why even use Refcount? This class should have its own way to track if PyCLRMetaType is allocated.

Copy link
Contributor Author

@BadSingleton BadSingleton Jan 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the reasoning is the same as with the similar pattern in typemanager.cs#38. The only way I can see this class tracking it's allocation is to look if the PyCLRMetaType is IntPtr.Zero or not.

Copy link
Contributor

@benoithudson benoithudson Jan 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic is: If the refcount is 1, then at line 31 when decref, it'll go to zero and the entire type will be released. In that case we don't care about clearing the slots. It's a standard optimization in a ref-counting framework.

It's correct assuming that ReleaseTypeSlots doesn't also release refcounts that otherwise won't be released.

That said: Do we have coverage on line 29? I'm not sure I really understand under what circumstance it happens.

Copy link
Member

@lostmsu lostmsu Feb 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd put the expression into bool isLastReference (don't miss the negations) or at least add a comment.

{
_metaSlotsHodler.ResetSlots();
Expand Down
194 changes: 89 additions & 105 deletions src/runtime/typemanager.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,8 @@ internal static IntPtr CreateType(ManagedType impl, Type clrType)
if (typeof(ICollection).IsAssignableFrom(clrType) || clrType.GetInterfaces().Any(x => x.IsGenericType && x.GetGenericTypeDefinition() == typeof(ICollection<>)))
{
var method = typeof(mp_length_slot).GetMethod(nameof(mp_length_slot.mp_length));
InitializeSlot(type, TypeOffset.mp_length, method, slotsHolder);
var thunk = Interop.GetThunk(method);
InitializeSlot(type, thunk, "__len__", slotsHolder);
}

if (base_ != IntPtr.Zero)
Expand Down Expand Up @@ -354,6 +355,28 @@ internal static void FreeMethodDef(IntPtr mdef)
}
}

/// <summary>
/// Adds a deallocator for a type's method. At deallocation, the deallocator will remove the
/// method from the type's Dict and deallocate the PyMethodDef object.
/// </summary>
/// <param name="t">The type to add the deallocator to.</param>
/// <param name="mdef">The pointer to the PyMethodDef structure.</param>
/// <param name="name">The name of the slot.</param>
/// <param name="slotsHolder">The SlotsHolder holding the deallocator/.</param>
internal static void AddDeallocator(IntPtr t, IntPtr mdef, string name, SlotsHolder slotsHolder)
{
slotsHolder.AddDealloctor(() =>
{
//IntPtr t = type;
IntPtr tp_dict = Marshal.ReadIntPtr(t, TypeOffset.tp_dict);
if (Runtime.PyDict_DelItemString(tp_dict, name) != 0)
{
Runtime.PyErr_Print();
}
FreeMethodDef(mdef);
});
}

internal static IntPtr CreateMetaType(Type impl, out SlotsHolder slotsHolder)
{
// The managed metatype is functionally little different than the
Expand Down Expand Up @@ -389,54 +412,31 @@ internal static IntPtr CreateMetaType(Type impl, out SlotsHolder slotsHolder)
Debug.Assert(4 * IntPtr.Size == Marshal.SizeOf(typeof(PyMethodDef)));
IntPtr mdefStart = mdef;
ThunkInfo thunk = Interop.GetThunk(typeof(MetaType).GetMethod("__instancecheck__"), "BinaryFunc");
slotsHolder.KeeapAlive(thunk.Target);

{
IntPtr mdefAddr = mdef;
slotsHolder.AddDealloctor(() =>
{
IntPtr t = type;
IntPtr tp_dict = Marshal.ReadIntPtr(t, TypeOffset.tp_dict);
if (Runtime.PyDict_DelItemString(tp_dict, "__instancecheck__") != 0)
{
Runtime.PyErr_Print();
}
FreeMethodDef(mdefAddr);
});
}
slotsHolder.KeepAlive(thunk.Target);
// Add deallocator before writing the method def, as after WriteMethodDef, mdef
// will not have the same value.
AddDeallocator(type, mdef, "__instancecheck__", slotsHolder);
mdef = WriteMethodDef(
mdef,
"__instancecheck__",
thunk.Address
);

thunk = Interop.GetThunk(typeof(MetaType).GetMethod("__subclasscheck__"), "BinaryFunc");
slotsHolder.KeeapAlive(thunk.Target);
{
IntPtr mdefAddr = mdef;
slotsHolder.AddDealloctor(() =>
{
IntPtr t = type;
IntPtr tp_dict = Marshal.ReadIntPtr(t, TypeOffset.tp_dict);
if (Runtime.PyDict_DelItemString(tp_dict, "__subclasscheck__") != 0)
{
Runtime.PyErr_Print();
}
FreeMethodDef(mdefAddr);
});
}
slotsHolder.KeepAlive(thunk.Target);
AddDeallocator(type, mdef, "__subclasscheck__", slotsHolder);

mdef = WriteMethodDef(
mdef,
"__subclasscheck__",
thunk.Address
);

// FIXME: mdef is not used
// Pad the last field with zeroes to terminate the array
mdef = WriteMethodDefSentinel(mdef);

Marshal.WriteIntPtr(type, TypeOffset.tp_methods, mdefStart);
slotsHolder.Add(TypeOffset.tp_methods, (t, offset) =>
slotsHolder.Set(TypeOffset.tp_methods, (t, offset) =>
Copy link
Member

@lostmsu lostmsu Feb 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: name lambda parameter for clarity

{
var p = Marshal.ReadIntPtr(t, offset);
Runtime.PyMem_Free(p);
Expand Down Expand Up @@ -818,9 +818,8 @@ internal static void InitializeSlots(IntPtr type, Type impl, SlotsHolder slotsHo
// These have to be defined, though, so by default we fill these with
// static C# functions from this class.

// var ret0 = Interop.GetThunk(((Func<IntPtr, int>)Return0).Method).Address;
// var ret1 = Interop.GetThunk(((Func<IntPtr, int>)Return1).Method).Address;

IntPtr ret0 = IntPtr.Zero;
IntPtr ret1 = IntPtr.Zero;
if (native != null)
{
// If we want to support domain reload, the C# implementation
Expand All @@ -829,35 +828,18 @@ internal static void InitializeSlots(IntPtr type, Type impl, SlotsHolder slotsHo
// load them into a separate code page that is leaked
// intentionally.
InitializeNativeCodePage();
IntPtr ret1 = NativeCodePage + native.Return1;
IntPtr ret0 = NativeCodePage + native.Return0;
InitializeSlot(type, ret0, "tp_traverse", canOverride: false);
InitializeSlot(type, ret0, "tp_clear", canOverride: false);
InitializeSlot(type, ret1, "tp_is_gc", canOverride: false);
ret1 = NativeCodePage + native.Return1;
ret0 = NativeCodePage + native.Return0;
}
else
{
if (!IsSlotSet(type, "tp_traverse"))
{
var thunkRet0 = Interop.GetThunk(((Func<IntPtr, int>)Return0).Method);
var offset = GetSlotOffset("tp_traverse");
Marshal.WriteIntPtr(type, offset, thunkRet0.Address);
if (slotsHolder != null)
{
slotsHolder.Add(offset, thunkRet0);
}
}
if (!IsSlotSet(type, "tp_clear"))
{
var thunkRet0 = Interop.GetThunk(((Func<IntPtr, int>)Return0).Method);
var offset = GetSlotOffset("tp_clear");
Marshal.WriteIntPtr(type, offset, thunkRet0.Address);
if (slotsHolder != null)
{
slotsHolder.Add(offset, thunkRet0);
}
}
ret1 = Interop.GetThunk(((Func<IntPtr, int>)Return1).Method).Address;
ret0 = Interop.GetThunk(((Func<IntPtr, int>)Return0).Method).Address;
}

InitializeSlot(type, ret0, "tp_traverse");
InitializeSlot(type, ret0, "tp_clear");
InitializeSlot(type, ret1, "tp_is_gc");
}

static int Return1(IntPtr _) => 1;
Expand All @@ -873,11 +855,11 @@ internal static void InitializeSlots(IntPtr type, Type impl, SlotsHolder slotsHo
/// </summary>
/// <param name="type">Type being initialized.</param>
/// <param name="slot">Function pointer.</param>
/// <param name="canOverride">Can override the slot when it existed</param>
static void InitializeSlot(IntPtr type, IntPtr slot, string name, bool canOverride = true)
/// <param name="name">Name of the slot to initialize</param>
static void InitializeSlot(IntPtr type, IntPtr slot, string name)
{
var offset = GetSlotOffset(name);
if (!canOverride && Marshal.ReadIntPtr(type, offset) != IntPtr.Zero)
if (Marshal.ReadIntPtr(type, offset) != IntPtr.Zero)
{
return;
}
Expand All @@ -901,16 +883,6 @@ static void InitializeSlot(IntPtr type, ThunkInfo thunk, string name, SlotsHolde
}
}

static void InitializeSlot(IntPtr type, int slotOffset, MethodInfo method, SlotsHolder slotsHolder = null)
{
var thunk = Interop.GetThunk(method);
Marshal.WriteIntPtr(type, slotOffset, thunk.Address);
if (slotsHolder != null)
{
slotsHolder.Add(slotOffset, thunk);
}
}

static int GetSlotOffset(string name)
{
Type typeOffset = typeof(TypeOffset);
Expand Down Expand Up @@ -981,88 +953,100 @@ private static SlotsHolder CreateSlotsHolder(IntPtr type)

class SlotsHolder
Copy link
Member

@lostmsu lostmsu Feb 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Put this class into a separate file

{
public delegate void Resetor(IntPtr type, int offset);
/// <summary>
/// Delegate called to customize a (Python) Type slot reset
/// </summary>
/// <param name="type">The type that will have a slot reset</param>
/// <param name="offset">The offset of the slot</param>
public delegate void ResetSlotAction(IntPtr type, int offset);

private readonly IntPtr _type;
private Dictionary<int, ThunkInfo> _slots = new Dictionary<int, ThunkInfo>();
private List<Delegate> _keepalive = new List<Delegate>();
private Dictionary<int, Resetor> _customRestors = new Dictionary<int, Resetor>();
private List<Action> _deallocators = new List<Action>();
private bool _alredyReset = false;
private readonly IntPtr type;
private Dictionary<int, ThunkInfo> slots = new Dictionary<int, ThunkInfo>();
private List<Delegate> keepalive = new List<Delegate>();
private Dictionary<int, ResetSlotAction> customResetors = new Dictionary<int, ResetSlotAction>();
private List<Action> deallocators = new List<Action>();
private bool alreadyReset = false;

/// <summary>
/// Create slots holder for holding the delegate of slots and be able to reset them.
/// </summary>
/// <param name="type">Steals a reference to target type</param>
public SlotsHolder(IntPtr type)
{
_type = type;
this.type = type;
}

public void Add(int offset, ThunkInfo thunk)
{
_slots.Add(offset, thunk);
slots.Add(offset, thunk);
}

public void Add(int offset, Resetor resetor)
public void Set(int offset, ResetSlotAction resetor)
{
_customRestors[offset] = resetor;
customResetors[offset] = resetor;
}

public void AddDealloctor(Action deallocate)
{
_deallocators.Add(deallocate);
deallocators.Add(deallocate);
}

public void KeeapAlive(Delegate d)
/// <summary>
/// Add a delegate to keep it from being garbage collected.
/// </summary>
/// <param name="d">The delegate to add</param>
public void KeepAlive(Delegate d)
{
_keepalive.Add(d);
keepalive.Add(d);
}

public void ResetSlots()
{
if (_alredyReset)
if (alreadyReset)
{
return;
}
_alredyReset = true;
IntPtr tp_name = Marshal.ReadIntPtr(_type, TypeOffset.tp_name);
string typeName = Marshal.PtrToStringAnsi(tp_name);
foreach (var offset in _slots.Keys)
alreadyReset = true;
foreach (var offset in slots.Keys)
{
IntPtr ptr = GetDefaultSlot(offset);
//DebugUtil.Print($"Set slot<{TypeOffsetHelper.GetSlotNameByOffset(offset)}> to 0x{ptr.ToString("X")} at {typeName}<0x{_type}>");
Marshal.WriteIntPtr(_type, offset, ptr);
Marshal.WriteIntPtr(type, offset, ptr);
}

foreach (var action in _deallocators)
foreach (var action in deallocators)
{
action();
}

foreach (var pair in _customRestors)
foreach (var pair in customResetors)
{
int offset = pair.Key;
var resetor = pair.Value;
resetor?.Invoke(_type, offset);
resetor?.Invoke(type, offset);
}

_customRestors.Clear();
_slots.Clear();
_keepalive.Clear();
_deallocators.Clear();
customResetors.Clear();
slots.Clear();
keepalive.Clear();
deallocators.Clear();

// Custom reset
IntPtr tp_base = Marshal.ReadIntPtr(_type, TypeOffset.tp_base);
IntPtr tp_base = Marshal.ReadIntPtr(type, TypeOffset.tp_base);
Runtime.XDecref(tp_base);
Marshal.WriteIntPtr(_type, TypeOffset.tp_base, IntPtr.Zero);
Marshal.WriteIntPtr(type, TypeOffset.tp_base, IntPtr.Zero);

IntPtr tp_bases = Marshal.ReadIntPtr(_type, TypeOffset.tp_bases);
IntPtr tp_bases = Marshal.ReadIntPtr(type, TypeOffset.tp_bases);
Runtime.XDecref(tp_bases);
tp_bases = Runtime.PyTuple_New(0);
Marshal.WriteIntPtr(_type, TypeOffset.tp_bases, tp_bases);
Marshal.WriteIntPtr(type, TypeOffset.tp_bases, tp_bases);
}

/// <summary>
/// Returns the default C function pointer for the slot to reset.
/// </summary>
/// <param name="offset">The offset of the slot.</param>
/// <returns>The default C function pointer of the slot.</returns>
private static IntPtr GetDefaultSlot(int offset)
Copy link
Contributor

@benoithudson benoithudson Jan 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the doc here should say that we're looking for the C function to call when resetting the slot at the given offset within the type.

We might want to eventually consider having those functions throw exceptions e.g. "the underlying .NET type has been deleted"

Copy link
Member

@lostmsu lostmsu Feb 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Problem is: with domain gone, who will provide those functions?

{
if (offset == TypeOffset.tp_clear
Expand All @@ -1072,7 +1056,7 @@ private static IntPtr GetDefaultSlot(int offset)
}
else if (offset == TypeOffset.tp_dealloc)
{
// tp_free of PyTypeType is point to PyObejct_GC_Del.
// tp_free of PyTypeType is point to PyObject_GC_Del.
return Marshal.ReadIntPtr(Runtime.PyTypeType, TypeOffset.tp_free);
Copy link
Contributor

@benoithudson benoithudson Jan 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PyObejct_GC_Del has a typo

}
else if (offset == TypeOffset.tp_free)
Expand Down