Unexpected behavior with `dict` as a class attribute #102648

adrinjalali · 2023-03-13T17:29:12Z

Bug report

When one puts __dict__ as a class attribute, the behavior becomes very odd:

>>> class A:
...     __dict__ = {'a': 1}
...     def __init__(self):
...         self.b = 2
... 
>>> a = A()
>>> print("a.b:", a.b)
a.b: 2
>>> print("a.__dict__", a.__dict__)
a.__dict__ {'a': 1}
>>> print("a has a", hasattr(a, 'a'))
a has a False
>>> print("a has b", hasattr(a, 'b'))
a has b True
>>> print("dir(a)", dir(a))
dir(a) ['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'a']
>>> print("a.a", a.a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'A' object has no attribute 'a'

Moving it to be an instance attribute, does what one expects:

>>> class A:
...     def __init__(self):
...         self.__dict__ = {'a': 1}
...         self.b = 2
... 
>>> a = A()
>>> print("a.b:", a.b)
a.b: 2
>>> print("a.__dict__", a.__dict__)
a.__dict__ {'a': 1, 'b': 2}
>>> print("a has a", hasattr(a, 'a'))
a has a True
>>> print("a has b", hasattr(a, 'b'))
a has b True
>>> print("dir(a)", dir(a))
dir(a) ['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'a', 'b']
>>> print("a.a", a.a)
a.a 1

I know one shouldn't have a __dict__ as a class attribute, but I think it would be nice to either have a good error message when that happens, or to make things more consistent, i.e. all([hasattr(a, x) for x in dir(a)]) should be true maybe?

We encountered this issue since certain pickle related logic was changed in 3.11: scikit-learn/scikit-learn#25188 (comment)

Your environment

CPython versions tested on: Python 3.11.0 | packaged by conda-forge | (main, Oct 25 2022, 06:24:40) [GCC 10.4.0] on linux
Operating system and architecture: archlinux, x86_64

The text was updated successfully, but these errors were encountered:

carljm · 2023-03-13T18:41:40Z

Thanks for the report! This is indeed confusing.

The reason for this behavior is that normally classes have a descriptor for the __class__ attribute which implements the behavior of finding and returning the actual instance dictionary (and allowing it to be set). By assigning a __dict__ attribute on the class, you overwrite this descriptor and thus lose all special behavior of the __dict__ attribute name. This means the real instance attribute dictionary is now hidden and inaccessible (barring extreme measures), and the __dict__ attribute is now a totally normal attribute which has no relationship to the real instance attribute dictionary, and has the normal instance-attribute-not-found-falls-back-to-class-attribute behavior. This then leads to further confusion because some parts of Python (e.g. dir) assume that they can use instance.__dict__ to get to the real instance attribute dictionary, so they are also now fooled as to what attributes the instance has.

I don't see a clear way to allow this override and "fix" the odd behavior without breaking Python's object model more deeply. We could raise an error on override of the __dict__ descriptor on types, but I suspect that someone out there has found a real use for this (probably not overriding it with a dictionary, but with a descriptor with more specialized behavior) and it would be a backwards-incompatible change to break it, that would require a deprecation process. Ultimately I'm not sure deprecating and disallowing this is worth it; I think it might be a case of "if you don't like the results, don't do that."

Will leave this open for now to see what others think.

AlexWaygood · 2023-03-13T19:27:14Z

I thought I remembered reading an issue about this before, but can't immediately find it

carljm · 2023-03-13T19:32:48Z

I thought I remembered reading an issue about this before, but can't immediately find it

I also searched and didn't find anything, but that gives me very little confidence that prior discussion doesn't exist :)

AlexWaygood · 2023-03-13T19:42:41Z

Anyway, whether or not we fix/document this, and whether or not we have discussed this before: it doesn't surprise me that this does something strange and unexpected. So "don't do that" is probably good advice here :)

sobolevn · 2023-03-14T08:27:41Z

I agree that we never said that this will work. And me personally would not want this to work.

Moreover, type.__dict__ is not even a dict:

>>> class My: ...
... 
>>> My.__dict__
mappingproxy({'__module__': '__main__', '__dict__': <attribute '__dict__' of 'My' objects>, '__weakref__': <attribute '__weakref__' of 'My' objects>, '__doc__': None})

It is an explicitly immutable type.

So, I think we can just recommend "Do not write this code, please".

adrinjalali added the type-bug An unexpected behavior, bug, or error label Mar 13, 2023

iritkatriel added the interpreter-core (Objects, Python, Grammar, and Parser dirs) label Nov 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unexpected behavior with `dict` as a class attribute #102648

Unexpected behavior with `dict` as a class attribute #102648

adrinjalali commented Mar 13, 2023

carljm commented Mar 13, 2023

AlexWaygood commented Mar 13, 2023

carljm commented Mar 13, 2023

AlexWaygood commented Mar 13, 2023

sobolevn commented Mar 14, 2023 •

edited

Loading

Unexpected behavior with __dict__ as a class attribute #102648

Unexpected behavior with __dict__ as a class attribute #102648

Comments

adrinjalali commented Mar 13, 2023

Bug report

Your environment

carljm commented Mar 13, 2023

AlexWaygood commented Mar 13, 2023

carljm commented Mar 13, 2023

AlexWaygood commented Mar 13, 2023

sobolevn commented Mar 14, 2023 • edited Loading

Unexpected behavior with `dict` as a class attribute #102648

Unexpected behavior with `dict` as a class attribute #102648

sobolevn commented Mar 14, 2023 •

edited

Loading