Thanks to visit codestin.com
Credit goes to github.com

Skip to content

GH-95589: Dont crash when subclassing extension classes with multiple inheritance #96028

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Aug 17, 2022
Merged

GH-95589: Dont crash when subclassing extension classes with multiple inheritance #96028

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
1b39c88
Treat tp_weakref and tp_dictoffset like other opaque slots for multip…
markshannon Aug 16, 2022
b994446
Document Py_TPFLAGS_MANAGED_DICT and Py_TPFLAGS_MANAGED_WEAKREF in wh…
markshannon Aug 16, 2022
bca93af
Add news.
markshannon Aug 16, 2022
2ce1e9f
Fix typo
markshannon Aug 16, 2022
21b50cf
Fix typo
markshannon Aug 16, 2022
0be2b91
Fix another typo.
markshannon Aug 16, 2022
c930792
Fix tense.
markshannon Aug 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions Doc/whatsnew/3.12.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -450,6 +450,11 @@ New Features
inherit the ``Py_TPFLAGS_HAVE_VECTORCALL`` flag.
(Contributed by Petr Viktorin in :gh:`93274`.)

The :const:`Py_TPFLAGS_MANAGED_DICT` and :const:`Py_TPFLAGS_MANAGED_WEAKREF`
flags have been added. This allows extensions classes to support object
``__dict__`` and weakrefs with less bookkeeping,
using less memory and with faster access.

Porting to Python 3.12
----------------------

Expand Down Expand Up @@ -486,6 +491,18 @@ Porting to Python 3.12
:c:func:`PyUnicode_FromFormatV`.
(Contributed by Philip Georgi in :gh:`95504`.)

* Extension classes wanting to add a ``__dict__`` or weak reference slot
should use :const:`Py_TPFLAGS_MANAGED_DICT` and
:const:`Py_TPFLAGS_MANAGED_WEAKREF` instead of ``tp_dictoffset`` and
``tp_weaklistoffset``, respectively.
The use of ``tp_dictoffset`` and ``tp_weaklistoffset`` is still
supported, but does not fully support multiple inheritance
(:gh: `95589`), and performance may be worse.
Classes declaring :const:`Py_TPFLAGS_MANAGED_DICT` should call
:c:func:`_PyObject_VisitManagedDict` and :c:func:`_PyObject_ClearManagedDict`
to traverse and clear their instance's dictionaries.
To clear weakrefs, call :c:func:`PyObject_ClearWeakRefs`, as before.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the extension docs be updated to provide examples of using the new API? (Probably as a separate PR...)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, they should. I also plan to add a new file documenting the new object layout.


Deprecated
----------

Expand Down
44 changes: 33 additions & 11 deletions Lib/test/test_capi.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -677,21 +677,43 @@ def test_heaptype_with_custom_metaclass(self):

def test_multiple_inheritance_ctypes_with_weakref_or_dict(self):

class Both1(_testcapi.HeapCTypeWithWeakref, _testcapi.HeapCTypeWithDict):
with self.assertRaises(TypeError):
class Both1(_testcapi.HeapCTypeWithWeakref, _testcapi.HeapCTypeWithDict):
pass
with self.assertRaises(TypeError):
class Both2(_testcapi.HeapCTypeWithDict, _testcapi.HeapCTypeWithWeakref):
pass

def test_multiple_inheritance_ctypes_with_weakref_or_dict_and_other_builtin(self):

with self.assertRaises(TypeError):
class C1(_testcapi.HeapCTypeWithDict, list):
pass

with self.assertRaises(TypeError):
class C2(_testcapi.HeapCTypeWithWeakref, list):
pass

class C3(_testcapi.HeapCTypeWithManagedDict, list):
pass
class Both2(_testcapi.HeapCTypeWithDict, _testcapi.HeapCTypeWithWeakref):
class C4(_testcapi.HeapCTypeWithManagedWeakref, list):
pass

for cls in (_testcapi.HeapCTypeWithDict, _testcapi.HeapCTypeWithDict2,
_testcapi.HeapCTypeWithWeakref, _testcapi.HeapCTypeWithWeakref2):
for cls2 in (_testcapi.HeapCTypeWithDict, _testcapi.HeapCTypeWithDict2,
_testcapi.HeapCTypeWithWeakref, _testcapi.HeapCTypeWithWeakref2):
if cls is not cls2:
class S(cls, cls2):
pass
class B1(Both1, cls):
inst = C3()
inst.append(0)
str(inst.__dict__)

inst = C4()
inst.append(0)
str(inst.__weakref__)

for cls in (_testcapi.HeapCTypeWithManagedDict, _testcapi.HeapCTypeWithManagedWeakref):
for cls2 in (_testcapi.HeapCTypeWithDict, _testcapi.HeapCTypeWithWeakref):
class S(cls, cls2):
pass
class B1(C3, cls):
pass
class B2(Both1, cls):
class B2(C4, cls):
pass
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also assert that the mro of these classes looks correct?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't change the mro. That's determined by the C3 algorithm. https://en.wikipedia.org/wiki/C3_linearization


def test_pytype_fromspec_with_repeated_slots(self):
Expand Down
4 changes: 4 additions & 0 deletions Misc/NEWS.d/next/C API/2022-08-16-16-54-42.gh-issue-95589.6xE1ar.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Extensions classes that set ``tp_dictoffset`` and ``tp_weaklistoffset``
lose the support for multiple inheritance, but are now safe. Extension
classes should use :const:`Py_TPFLAGS_MANAGED_DICT` and
:const:`Py_TPFLAGS_MANAGED_WEAKREF` instead.
45 changes: 13 additions & 32 deletions Objects/typeobject.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2330,51 +2330,32 @@ best_base(PyObject *bases)
return base;
}

#define ADDED_FIELD_AT_OFFSET(name, offset) \
(type->tp_ ## name && (base->tp_ ##name == 0) && \
type->tp_ ## name + sizeof(PyObject *) == (offset) && \
type->tp_flags & Py_TPFLAGS_HEAPTYPE)

static int
extra_ivars(PyTypeObject *type, PyTypeObject *base)
shape_differs(PyTypeObject *t1, PyTypeObject *t2)
{
size_t t_size = type->tp_basicsize;
size_t b_size = base->tp_basicsize;

assert(t_size >= b_size); /* Else type smaller than base! */
if (type->tp_itemsize || base->tp_itemsize) {
/* If itemsize is involved, stricter rules */
return t_size != b_size ||
type->tp_itemsize != base->tp_itemsize;
}
/* Check for __dict__ and __weakrefs__ slots in either order */
if (ADDED_FIELD_AT_OFFSET(weaklistoffset, t_size)) {
t_size -= sizeof(PyObject *);
}
if ((type->tp_flags & Py_TPFLAGS_MANAGED_DICT) == 0 &&
ADDED_FIELD_AT_OFFSET(dictoffset, t_size)) {
t_size -= sizeof(PyObject *);
}
/* Check __weakrefs__ again, in case it precedes __dict__ */
if (ADDED_FIELD_AT_OFFSET(weaklistoffset, t_size)) {
t_size -= sizeof(PyObject *);
}
return t_size != b_size;
return (
t1->tp_basicsize != t2->tp_basicsize ||
t1->tp_itemsize != t2->tp_itemsize
);
}

static PyTypeObject *
solid_base(PyTypeObject *type)
{
PyTypeObject *base;

if (type->tp_base)
if (type->tp_base) {
base = solid_base(type->tp_base);
else
}
else {
base = &PyBaseObject_Type;
if (extra_ivars(type, base))
}
if (shape_differs(type, base)) {
return type;
else
}
else {
return base;
}
}

static void object_dealloc(PyObject *);
Expand Down