Descriptors PREMIUM

Descriptors are Python objects that can intercept attribute access on another object.

Properties

Python's property decorator is powered by the descriptor protocol.

When we use property like this:

import math

class Circle:
    def __init__(self, radius=1):
        self.radius = radius
    @property
    def area(self):
        return math.pi * self.radius**2

We're telling Python to call a function whenever a specific attribute is accessed.

Accessing the area will call that area function and return its value:

>>> circle = Circle(radius=1)
>>> circle.area
3.141592653589793

So if we change the radius of this circle object and we access its area attribute again, we'd see a different value:

>>> circle.radius = 5
>>> circle.area
78.53981633974483

The area isn't stored anywhere: it's computed each time we ask for the area attribute.

Properties are powered by the descriptor protocol, which relies on a __get__ method.

The basics of attribute lookups

When Python looks up an attribute on an object, you might think it looks on the object itself and then on the object's class and parent classes. And if a matching attribute is found, the object that attribute points to will be returned. Python does do that (as noted in How attribute lookups and assignments work), but it also does a little more.

Let's say we have a Circle class with a radius instance attribute, a pi class attribute, and an area property:

class Circle:
    pi = 3.14159
    def __init__(self, radius=1):
        self.radius = radius
    @property
    def area(self):
        return self.pi * self.radius**2

When we lookup radius, Python will first look in our object's __dict__ dictionary, which contains all data for our class instance:

>>> c = Circle(5)
>>> c.__dict__
{'radius': 5}
>>> c.radius
5

Because a matching attribute is found, Python stops and returns it to us.

But what about with the pi attribute? When we lookup the pi attribute, Python won't find a matching attribute in the __dict__ dictionary on our Circle instance:

>>> 'pi' in c.__dict__
False

So Python will then look in the __dict__ dictionary on the Circle class:

>>> type(c)
<class '__main__.Circle'>
>>> 'pi' in type(c).__dict__
True
>>> type(c).__dict__['pi']
3.14159

This is most of the story, as noted in How attribute lookups and assignments work... but it doesn't explain how properties work.

What happens when we lookup the area attribute, which is a property?

Class attribute check for __get__

When we lookup the area attribute on our Circle object, Python will again see that the Circle object doesn't have an area attribute... but the Circle class does!

>>> 'area' in c.__dict__
False
>>> 'area' in type(c).__dict__
True

But this time, Python won't just return the value that the area class attribute points to. If it did, we would see a property object:

>>> type(c).__dict__['area']
<property object at 0x7f4f20b60cc0>

When Python looks up an attribute on a class, it first checks whether that attribute's value is an object with a __get__ method:

>>> type(c).area.__get__
<method-wrapper '__get__' of property object at 0x7f4f20b60cc0>

If that object doesn't have a __get__ method (as was the case with our pi attribute) then that object is returned as-is:

>>> hasattr(type(c).pi, "__get__")
False
>>> type(c).pi
3.14159

If that object does have a __get__ method, Python will call that __get__ method, passing in the class instance for which it's performing the attribute lookup:

>>> Circle.area.__get__(c)
78.53975

So looking up area on a Circle instance is the same as looking up area on the Circle class and then calling the __get__ method on the returned object, passing in our class instance to __get__:

>>> c.area  # This
78.53975
>>> type(c).__dict__['area'].__get__(c)  # Results in this
78.53975

How attribute lookups mostly work

So, in terms of Python code, attribute lookups first check the __dict__ dictionary on the object, then they check the __dict__ dictionary on the object's class, checking whether there's a __get__ method to be called:

def lookup_attribute(instance, attr):
    if attr in instance.__dict__:
        return instance.__dict__[attr]
    elif attr in type(instance).__dict__:
        obj = type(instance).__dict__[attr]
        if hasattr(obj, '__get__'):
            return obj.__get__(instance)
        else:
            return obj
    else:
        raise AttributeError

Objects that have a __get__ attribute are called descriptors. Descriptors are able to intercept the attribute lookup on a class instance.

But the above code isn't the whole truth, as there's a bit more to attribute lookups. Python has two types of descriptors: data descriptor and non-data descriptors. Data descriptors make attribute lookups even more complex... as we'll see later. But first, let's make our own non-data descriptor.

Non-data descriptors

A descriptor is an object with a __get__ method. Here's a descriptor class:

class CallFunction:
    def __init__(self, function):
        self.function = function
    def __get__(self, obj, obj_type):
        return self.function()

Let's make a new class with a descriptor object attached to it:

from random import choice

def random_color():
    return choice(["purple", "blue", "green"])

class Point:
    color = CallFunction(random_color)
    def __init__(self, x, y, z):
        self.x = x
        self.y = y
        self.z = z

This Point class has a color attribute, which points to a descriptor (an instance of our CallFunction class).

When Python looks up an attribute on an object, if the attribute exists on that object's class it checks for the presence of a __get__ method on that object. So when we look up the attribute that we've attached our descriptor to, the descriptor object's __get__ method will be called:

>>> p = Point(1, 2, 3)
>>> p.color
'purple'
>>> p.color
'blue'

This type of descriptor is called a non-data descriptor.

Non-data descriptors are accessed after checking whether the class instance has a matching attribute name. So if we assign a value to the color attribute on our Point object, we'll see that this new value is returned whenever we access that attribute:

>>> p.color = 'black'
>>> p.color
'black'

Note that we're assigning that color attribute on our class instance (not the class itself). So a separate instance of this Point class would still show a random color attribute (as long as it doesn't have a color attribute on itself):

>>> p2 = Point(1, 2, 3)
>>> p2.color
'green'

Non-data descriptors are just one type of descriptor. The other descriptor type is a data descriptor.

Data descriptors

A data descriptor is a descriptor that has a __set__ method (in addition to the required __get__ method).

For example, this AlwaysThree class makes data descriptor objects:

class AlwaysThree:
    def __get__(self, obj, obj_type):
        return 3
    def __set__(self, obj, value):
        raise ValueError("Setting this attribute is not allowed")

When Python looks up an attribute on an object, it first checks the class of that object (not the object itself). If the attribute exists on the object's class and the attribute's value has a __get__ method and a __set__ method, the __get__ is called on that attribute's value.

Here's a Point class with a data descriptor attached to it:

class Point:
    dimensions = AlwaysThree()
    def __init__(self, x, y, z):
        self.x = x
        self.y = y
        self.z = z

If we make an instance of this Point class and then we access the dimensions attribute on the Point object, we'll see the value 3:

>>> p = Point()
>>> p.dimensions
3

That dimensions attribute points to a data descriptor. When a data descriptor is attached to a class as an attribute, it will control the access of that attribute on all instances of that class.

Unlike non-data descriptors, data descriptors also control assignments to that attribute. If we assign to the dimensions attribute, the __set__ method will be called on the data descriptor. Our AlwaysThree instance's __set__ method raises an exception:

>>> p.dimensions = 4
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 5, in __set__
ValueError: Setting this attribute is not allowed

Non-data descriptors control attribute access on class instances, but only if an attribute with the same name hasn't been assigned on that class instance. Data descriptors control both attribute access and attribute assignment on class instances.

Descriptors complicate attribute lookup order in Python

Thanks to data descriptors, attribute access isn't as simple as what I alluded to above. This isn't quite right:

def lookup_attribute(instance, attr):
    if attr in instance.__dict__:
        return instance.__dict__[attr]
    elif attr in type(instance).__dict__:
        obj = type(instance).__dict__[attr]
        if hasattr(obj, '__get__'):
            return obj.__get__(instance)
        else:
            return obj
    else:
        raise AttributeError

When Python looks up an attribute on an object, it actually checks the class for a data descriptor before it checks the object.

Let's redefine our Point class to have a data descriptor, a non-data descriptor, and a regular class attribute:

from random import choice

def random_color():
    return choice(["purple", "blue", "green"])

class CallFunction:
    def __init__(self, function):
        self.function = function
    def __get__(self, obj, obj_type):
        return self.function()

class AlwaysThree:
    def __get__(self, obj, obj_type):
        return 3
    def __set__(self, obj, value):
        raise ValueError("Setting this attribute is not allowed")

class Point:
    color = CallFunction(random_color)  # non-data descriptor
    dimensions = AlwaysThree()  # data descriptor
    origin = (0, 0, 0)
    def __init__(self, x, y, z):
        self.x = x
        self.y = y
        self.z = z

Now let's make one of these new Point objects and manually add a dimensions key to that object's __dict__ dictionary:

>>> p = Point(1, 2, 3)
>>> p.__dict__['dimensions'] = 4

If we lookup the dimensions attribute on our Point object, we'll see that it uses the data descriptor attached to the dimensions class attribute:

>>> p.dimensions
3

Python didn't even look at our Point object's attributes! Instead, it went to the class first to look for data a descriptor. Because it found a dimensions attribute that has a __get__ method and a __set__ method, it called the __get__ method on that object.

If we do the same thing with our non-data descriptor, we'll see that the attribute that lives on our Point object will be used instead:

>>> p.color
'green'
>>> p.__dict__['color'] = 'black'
>>> p.color
'black'

So a more accurate version of that lookup_attribute function above might look like this:

def lookup_attribute(instance, attr):
    cls = type(instance)
    # Check for a data descriptor on the object's class
    if attr in cls.__dict__ and hasattr(cls.__dict__[attr], '__set__'):
        return cls.__dict__[attr].__get__(instance)
    # Check for an attribute on the object
    elif attr in instance.__dict__:
        return instance.__dict__[attr]
    # Check for an attribute on the class
    elif attr in cls.__dict__:
        obj = cls.__dict__[attr]
        # Check whether the class attribute is a non-data descriptor
        if hasattr(obj, '__get__'):
            return obj.__get__(instance)
        else:
            return obj
    else:
        raise AttributeError

Though this also isn't the whole truth, as we're assuming our object uses a __dict__ dictionary (it may use __slots__) and we're not checking parent classes at all. For an even more accurate version (though one which still ignores __slots__), see this Python snippet or the blog post that inspired it.

How the property decorator works

Before we wrap up, we should talk about Python's property decorator. Python's property decorator is a data descriptor.

You can think of the propery class as being nearly equivalent to this:

class property:

    def __init__(self, fget=None, fset=None, fdel=None):
        self.fget = fget
        self.fset = fset
        self.fdel = fdel
        self._name = ""

    def __set_name__(self, owner, name):
        self._name = name

    def __get__(self, obj, obj_type=None):
        if obj is None:
            return self
        if self.fget is None:
            raise AttributeError(f"property {self._name!r} has no getter")
        return self.fget(obj)

    def __set__(self, obj, value):
        if self.fset is None:
            raise AttributeError(f"property {self._name!r} has no setter")
        self.fset(obj, value)

    def __delete__(self, obj):
        if self.fdel is None:
            raise AttributeError(f"property {self._name!r} has no deleter")
        self.fdel(obj)

    def setter(self, fset):
        self.fset = fset
        return self

    def deleter(self, fdel):
        self.fdel = fdel
        return self

The above property class supports getting and setting, just as the built-in property class does. In addition to __get__ and __set__, data descriptors also support a __delete__ method, which controls what happens when attempting to delete the attribute with Python's del statement. Properties allow customizing this as well, thanks to their (rarely seen) deleter method.

So whenever you use Python's property decorator, you're using a descriptor.

Storing names

You may have noticed the __set_name__ method in the property copycat descriptor we made above.

    def __set_name__(self, owner, name):
        self._name = name

When an attribute is attached to a class, if that attribute has a __set_name__ method, Python will call that method with the class and the attribute name. So if we make a class with a __set_name__ method:

class AttributeUseWatcher:
    def __set_name__(self, owner, name):
        print(f"Attached to {owner.__name__} class's {name} attribute.")

When we create a new class with an instance of that class used as an attribute on it, we'll see that __set_name__ method will be called:

>>> class Example:
...     my_attribute = AttributeUseWatcher()
...
Attached to Example class's my_attribute attribute.

That __set_name__ method is only called for attributes that are attached to a class during the class's creation. So attaching an object after the class has been made will not call __set_name__:

>>> Example.another_attribute = AttributeUseWatcher()
>>>

Prefer properties over custom descriptors

The most common descriptors in Python are properties. In fact, if you're trying to customize the behavior of a specific attribute lookup, you should almost always use a property instead of creating a custom descriptor.

Creating a custom descriptor object typically only makes sense if the behavior you're seeking should be reusable between multiple attributes, whether on the same class or different classes. If you're considering copy-pasting the same property between multiple classes, you may want to make a descriptor instead.

Attribute lookups use descriptors

Descriptors are objects that can customize what happens when we lookup the class attribute they're attached to.

A data descriptor is an object that has a __get__ method and a __set__ method and/or a __delete__ method. A non-data descriptor is an object that has a __get__ method but does not have a __set__ method or a __delete__ method.

When Python looks up an attribute on an object, it does this:

1. Check for a data descriptor on the object's class (and parent classes) with the correct attribute name
2. Check for an instance attribute (an attribute on the object itself) with the correct attribute name
3. Check for a non-data descriptor on the object's class (and parent classes) with the correct attribute name
4. Check for a class attribute on the object's class (and parent classes) with the correct attribute name

Properties are usually preferable over creating custom descriptor objects, but descriptor classes can occasionally come in handy for creating reusable attribute lookup customizations.