FIX fix pickling for empty object with Python 3.11+ #25188

BenjaminBossan · 2022-12-13T15:33:52Z

As discussed with @adrinjalali

Since Python 3.11, objects have a __getstate__ method by default:

Therefore, the exception in BaseEstimator.__getstate__ will no longer be raised, thus not falling back on using the object's __dict__:

scikit-learn/sklearn/base.py

Lines 274 to 280 in dc580a8

    
           try: 
        
               state = super().__getstate__() 
        
           except AttributeError: 
        
               state = self.__dict__.copy() 
        
           if type(self).__module__.startswith("sklearn."): 
        
               return dict(state.items(), _sklearn_version=__version__)

If the instance dict of the object is empty, the return value will, however, be None. Therefore, the line below calling state.items() results in an error.

In this bugfix, it is checked if the state is None and if it is, the object's __dict__ is used (which should always be empty).

Not addressed in this PR is how to deal with slots (see also discussion in #10079). When there are __slots__, __getstate__ will actually return a tuple, as documented here.

The user would thus still get an indiscriptive error message.

Since Python 3.11, objects have a __getstate__ method by default: python/cpython#70766 Therefore, the exception in BaseEstimator.__getstate__ will no longer be raised, thus not falling back on using the object's __dict__: https://github.com/scikit-learn/scikit-learn/blob/dc580a8ef5ee2a8aea80498388690e2213118efd/sklearn/base.py#L274-L280 If the instance dict of the object is empty, the return value will, however, be None. Therefore, the line below calling state.items() results in an error. In this bugfix, it is checked if the state is None and if it is, the object's __dict__ is used (which should always be empty). Not addressed in this PR is how to deal with slots (see also discussion in scikit-learn#10079). When there are __slots__, __getstate__ will actually return a tuple, as documented here: https://docs.python.org/3/library/pickle.html#object.__getstate__ The user would thus still get an indiscriptive error message.

adrinjalali

It's interesting that now if a class has __slots__, it also has to implement __getstate__.

https://github.com/python/cpython/pull/2821/files#diff-3fd2e552779d3d96e7c1f2d80f4b8fd88752ef5437f42201037ed7c76bba2779R92

I think it makes sense to handle __slots__, now that it's influencing what's returned by __getstate__. But I'm not sure if it needs to be in this PR or a different one.

I recon this is somewhat a major change to warrant a @scikit-learn/core-devs ping.

Also needs a whatsnew entry.

adrinjalali · 2022-12-13T16:08:45Z

sklearn/base.py

+            # TODO: Remove once Python < 3.11 is dropped, as there will never be
+            # an AttributeError


this would still be raised for C extension types which don't inherit from PyObject though, right? One can have a C extension type w/o __reduce__ for instance. It doesn't make much sense, but one can do that.

Yes, right, I'll remove the comment.

sklearn/tests/test_base.py

adrinjalali · 2022-12-13T16:43:36Z

For an end to end test, could you please add a test where pickle.loads(pickle.dumps(obj)) fails on main but passes here?

Co-authored-by: Adrin Jalali <[email protected]>

BenjaminBossan · 2022-12-13T16:49:55Z

For an end to end test, could you please add a test where pickle.loads(pickle.dumps(obj)) fails on main but passes here?

Yeah, I had that and removed it again :) Added it back.

adrinjalali

I'm happy with this PR, especially since we're not changing behavior from what we already have in main; but I think there are cases where we might be having a different behavior than python's default __getstate__, and it'd be nice to have them less diverged from one another.

Note to reviewers, the circleci job doesn't work for @BenjaminBossan 's PRs on this repo somehow.

adrinjalali · 2022-12-14T10:34:29Z

sklearn/base.py

@@ -273,6 +273,8 @@ def __repr__(self, N_CHAR_MAX=700):
    def __getstate__(self):
        try:
            state = super().__getstate__()
+            if state is None:
+                state = self.__dict__.copy()


python seems to be making a distinction between a __dict__ attached to the instance, and a __dict__ attached to the class. I'm not sure if we should do the same:

>>> class A: ... __dict__ = {"a": 10} ... >>> a = A() >>> a.__dict__ {'a': 10} >>> a.__getstate__() >>> a.__dict__.update({"b": 3}) >>> a.__dict__ {'a': 10, 'b': 3} >>> a.__getstate__() >>> class B: ... ... ... >>> b = B() >>> b.b = 10 >>> b.__dict__ {'b': 10} >>> b.__getstate__() {'b': 10}

Having a __dict__ as class attribute results in some (for me) quite surprising behavior. Not sure if that should be supported and if yes, how to differentiate that here.

class F: __dict__ = {"x": 0} def __init__(self): self.y = 1 f = F() print(f"{f.__getstate__()=}") # {'y': 1} print(f"{f.__dict__=}") # {'x': 0} print(f"{'x' in dir(f)=}") # True print(f"{'y' in dir(f)=}") # False f.y # works f.x # AttributeError: 'F' object has no attribute 'x'

python seems to be making a distinction between a __dict__ attached to the instance, and a __dict__ attached to the class.

Class attributes and instance attributes are different, which is why you see what you are seeing. While this is maybe not a super useful answer, as it is just stating a fact about how Python works, I don't understand what you are trying to do that leads you down the path of poking around at this level. Can you give an example or use-case? Would help with giving a more helpful answer.

adrinjalali · 2022-12-14T11:25:44Z

Reading more about it here (https://wiki.python.org/moin/UsingSlots) it seems that __slots__ would prevent __dict__ to be populated. For instance, this would raise:

class C:
	__slots__ = ('a',)
	def __init__(self):
		setattr(self, 'b', 10)
C()
AttributeError: 'C' object has no attribute 'b'

Which also means it's going to be really tricky for a user to have a subclass with __slots__ and also handle the descriptors we've recently introduced in BaseEstimator. For this reason, I think we can simply ignore handling __slots__.

EDIT: __slots__ only affects instance attributes, so descriptors would still work since they're class attributes.

betatim · 2022-12-14T14:10:49Z

I was trying to find a case where you run into this problem. But I couldn't quickly find an estimator that is broken without this change. Do you know one or maybe elaborate when you hit this bug?

BenjaminBossan · 2022-12-14T14:20:54Z

Do you know one or maybe elaborate when you hit this bug?

I stumbled upon this first with KernelCenterer, there may be others. Here is code to reproduce it:

>>> from sklearn.preprocessing import KernelCenterer
>>> import pickle
>>> pickle.dumps(KernelCenterer())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../sklearn/base.py", line 280, in __getstate__
    return dict(state.items(), _sklearn_version=__version__)
                ^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'items'

betatim · 2022-12-14T14:23:31Z

sklearn/base.py

@@ -273,6 +273,8 @@ def __repr__(self, N_CHAR_MAX=700):
    def __getstate__(self):
        try:
            state = super().__getstate__()
+            if state is None:
+                state = self.__dict__.copy()


One thing that puzzles me here is that the docs say that __getstate__() will return None if there is no instance __dict__. So how can we end up going down this if branch and then try to access self.__dict__? I've thought about this for a few minutes now but I can't work out when this would work, I'd expect that if we go down this path, then there is no self.__dict__, and so we can't access it here. So puzzled.

Yes, I have been thoroughly confused by the documentation as well. What I think it doesn't mention, which is relevant here, is that if the object does not define __getstate__ and if its instance dict is empty (but exists), it will also return None. This is the case we cover here.

>>> sys.version 3.11.0 >>> class A: ... pass >>> print(A().__getstate__()) None >>> A().__dict__ {}

Perhaps that's what they mean by:

For a class that has no instance __dict__ and no __slots__, the default state is None.

but I'm not sure.

The other cases are indeed not covered, so this code can still break in very specific circumstances.

I'm glad I'm not the only one puzzled by this. For me "has no instance __dict__ sounds like "it doesn't exist" not "it exists but is empty". So yeah, confused.com

Could state = self.__dict__.copy() ever be anything than an empty dict? And if not, we could change it to state = {} no?

I'm wondering about C extension types which don't necessarily have to comply with python object conventions.

This part of our code seems to be quite buggy if we consider all sorts of things people could do with __dict__ and __slots__ both on the instance and on the class (I know they shouldn't lol). But I'm okay with not supporting all those odd usecases.

Could state = self.dict.copy() ever be anything than an empty dict? And if not, we could change it to state = {} no?

I wondered about this too. Probably, it would always be an empty dict. If there is ever a situation (maybe Python 3.12?) where it's not an empty dict, would it be better to return that non-empty dict or {}? I thought the former, so I chose this solution. Not sure what other considerations to make here, runtime performance shouldn't matter much.

But I'm okay with not supporting all those odd usecases.

At least as a first step, we could focus on making the existing sklearn estimators run again. I used the skops persistence tests, which should cover __getstate__ of all estimators, and in addition to KernelCenterer, I also found LabelEncoder, which is a big one. Both of these don't have any C extensions, right?

There could be more estimators that I'm missing because they're masked by other errors in the test suite that occur due to Python 3.11.

adrinjalali · 2022-12-14T14:54:16Z

Do you know one or maybe elaborate when you hit this bug?

@betatim this is in the context of skops's persistence model, which makes use of __getstate__/__setstate__.

adrinjalali · 2022-12-15T13:27:07Z

@BenjaminBossan might make sense to run our skops test suit against this branch to see if it all works. WDYT?

BenjaminBossan · 2022-12-15T13:34:03Z

might make sense to run our skops test suit against this branch to see if it all works. WDYT?

Yes, in theory that would be a good test. However, we have a few other errors in the same tests when using Python 3.11, which could mask errors stemming from this issue. So it would take a little bit of work on the skops side to clean up the tests (but we will have to do it eventually anyway).

Maybe it would also be good to check that pickle works for all estimators in sklearn test_common.py (though not necessarily in this PR).

lesteve · 2022-12-15T14:00:08Z

Maybe it would also be good to check that pickle works for all estimators in sklearn test_common.py (though not necessarily in this PR).

There is already such a test, I would have been surprised of the contrary, see

scikit-learn/sklearn/utils/estimator_checks.py

Line 1853 in 205f3b7

def check_estimators_pickle(name, estimator_orig):

Maybe some estimators are not tested for some reason?

BenjaminBossan · 2022-12-15T14:16:03Z

There is already such a test, I would have been surprised of the contrary, see

Ah, I was not looking there

Maybe some estimators are not tested for some reason?

I think the reason why this passes is because the estimators are always fitted there. If fitted, there is always an instance dict for all tested estimators, therefore the edge case is not triggered and the tests pass.

The question is: Does sklearn want unfitted estimators to also be pickleable? And should we rely on the fact that all estimators, after fitting, have an instance dict? Theoretically, there could be estimators that don't need instance variables.

lesteve · 2022-12-15T14:20:30Z

Maybe some estimators are not tested for some reason?

Actually, I checked and KernelCenterer is tested. The thing is that we try to pickle the estimator after calling .fit. I am guessing after .fit we are not in the problematic situation in Python 3.11, maybe because we set some attributes?

The following snippet works fine:

from sklearn.preprocessing import KernelCenterer
import pickle

est = KernelCenterer().fit([[1.]])

pickle.dumps(est)

I edited the test to pickle before fit, and KernelCenterer is the only one that fails. LabelEncoder is not tested in this test for some reason (I don't know why).

adrinjalali · 2022-12-15T14:22:19Z

We certainly want unfitted estimators to be pickle-able, which I think might be used in some distributed settings. I don't see any reason why we'd intentionally not support pickling unfitted estimators.

glemaitre

I think that we also need an entry in 1.2.1 since this is a bug fix that solves an issue if people try to pickle in 1.2.0.

Otherwise, I only have some formatting changes. LGTM.

sklearn/base.py

glemaitre · 2022-12-15T14:20:43Z

sklearn/tests/test_base.py

+def test_parent_object_empty_instance_dict():
+    # Since Python 3.11, Python objects have a __getstate__ method by default
+    # that returns None if the instance dict is empty. See #25188.
+    class Empty:
+        pass
+
+    class Estimator(Empty, BaseEstimator):
+        pass
+
+    state = Estimator().__getstate__()
+    expected = {"_sklearn_version": sklearn.__version__}
+    assert state == expected
+
+
+def test_base_estimator_empty_instance_dict():
+    # Since Python 3.11, Python objects have a __getstate__ method by default
+    # that returns None if the instance dict is empty. See #25188.
+
+    # this should not raise
+    state = BaseEstimator().__getstate__()
+    expected = {"_sklearn_version": sklearn.__version__}
+    assert state == expected


Since we are testing the same behaviour here, I would prefer to parameterise the test and provide the object on which to __getstate__.

Also could you put the comment inside a docstring of the test

def test_base_estimator_empty_instance(): """Check that `__getstate__` returns an empty `dict` with an empty instance. Python 3.11+ changed behaviour by returning `None` instead of raising an `AttributeError`. Non-regression test for gh-25188. """ ...

I am wondering if we could make the round-trip pickling-unpickling directly in this test instead of having a new test.

betatim · 2022-12-15T14:23:21Z

This is a summary of what I've understood:

For Python before v3.11, __getstate__ could be missing. In which case the output of self.__dict__.copy() would be used.

For Python v3.11, __getstate__ always exists, but sometimes returns None. In that case we continue to use self.__dict__.copy(), which seems to always be an empty dict. The difference to the old state is that we expect self.__dict__.copy() to always return an empty dict, where as before it could contain things. Doesn't seem to be a problem.

Next __slots__. In Python before v3.11 instances of a class like the following:

class A:
  __slots__ = ["a"]
  def __init__(self, a):
    self.a = a

does not have a __getstate__ and also does not have a __dict__. I think this means the current code would break if it ever encountered an estimator like this.

In Python 3.11, the instances will have a __getstate__ (that returns (None, {'a': 42})). They still don't have a __dict__. This means if we adjust the code in this PR to handle the different return value of __getstate__ we'd not get exceptions. Or at least not get exceptions for things that used to create exceptions.

For completeness: Instances of this class

class A:
  __slots__ = ["a"]
  def __init__(self, a):
    pass

will not have a __dict__ and __getstate__ will return None (in 3.11).

glemaitre · 2022-12-15T14:24:00Z

The thing is that we try to pickle the estimator after calling .fit

Just a guess... what about stateless models that does not store anything during fit.

glemaitre · 2022-12-15T14:32:51Z

We agree that regardless of the Python version, the __getstate__ would not work with __slots__ anyway. We don't introduce a regression since it would not work before as well (i.e. no __dict__ and thus failing copy).

betatim · 2022-12-15T14:40:11Z

We agree that regardless of the Python version, the __getstate__ would not work with __slots__ anyway. We don't introduce a regression since it would not work before as well (i.e. no __dict__ and thus failing copy).

Just to be super explicit, in Python 3.11 the default __getstate__ will succeed for instances that use __slots__. In older versions the scikit-learn __getstate__ would raise an exception for classes with __slots__ and without a __getstate__.

This means, we should handle the case where __getstate__ returns something in Python 3.11 for classes that use __slots__. It will return (None, {'x': 42}) in my example above, so if we don't adjust the code in this PR users will get some super weird exception.

adrinjalali · 2022-12-15T14:47:42Z

I don't mind us handling __slots__ here. But we never really supported __slots__, and users were encountering silent bugs if the code were not raising an exception, and now we would raise, I'm okay with that.

I'm not sure if we should put code for something which we're really not supporting and there's no way we'll go down the road of supporting it.

glemaitre · 2022-12-15T14:55:28Z

This means, we should handle the case where getstate returns something in Python 3.11 for classes that use slots. It will return (None, {'x': 42}) in my example above, so if we don't adjust the code in this PR users will get some super weird exception.

Probably something like:

AttributeError: 'tuple' object has no attribute 'items'

If we want to handle it, my first guess would be something like:

        err_msg_slots = (
            "Pickling an estimator is not supported for classes using __slots__ "
            "instead of __dict__."
        )
        try:
            try:
                state = super().__getstate__()
                if state is None:
                    state = self.__dict__.copy()
            except AttributeError:
                state = self.__dict__.copy()
        except AttributeError as exc:
            # Fails if `__slots__` are used in an empty instance
            raise ValueError(err_msg_slots) from exc

        if not isinstance(state, Mapping):
            # Fails if `__slots__` are used with a non-empty instance
            raise ValueError(err_msg_slots)

and I hate nested try ... except .... A bit less nested:

        try:
            state = getattr(super, "__getstate__", self.__dict__.copy())
        except AttributeError as exc:
            # Fails if `__slots__` are used in an empty instance
            raise ValueError(err_msg_slots) from exc

        if state is None and hasattr(self, "__dict__"):
            state = self.__dict__.copy()
        else:
            raise ValueError(err_msg_slots)

adrinjalali · 2022-12-15T15:10:42Z

Python's pickle code checks if the output of __getstate__ is a tuple, and if yes, it assumes __slots__ are available. We can do the same here and raise and say we don't support slots.

Co-authored-by: Guillaume Lemaitre <[email protected]>

BenjaminBossan · 2022-12-15T15:33:41Z

Python's pickle code checks if the output of __getstate__ is a tuple, and if yes, it assumes __slots__ are available. We can do the same here and raise and say we don't support slots.

I can add a check for slots if we agree on that. What would be an appropriate error to raise here?

adrinjalali · 2022-12-15T15:40:05Z

We could simply do:

if getattr(self, "__slots__", None):
	TypeError("You cannot use `__slots__` in objects inheriting from `sklearn.base.BaseEstimator`")

similar to: https://github.com/python/cpython/pull/2821/files#diff-3fd2e552779d3d96e7c1f2d80f4b8fd88752ef5437f42201037ed7c76bba2779R92

- Add entry to changelog - Check if slots are used, raise TypeError - Refactor tests to use parametrize, test __getstate__ and pickle in same test

BenjaminBossan · 2022-12-15T15:47:28Z

I think I made all changes as requested by reviewers:

Add entry to change log
Check if slots are used, raise TypeError
Refactor tests to use parametrize, test __getstate__ and pickle in same test

adrinjalali · 2022-12-15T15:48:36Z

sklearn/base.py

@@ -271,9 +271,20 @@ def __repr__(self, N_CHAR_MAX=700):
        return repr_

    def __getstate__(self):
+        if hasattr(self, "__slots__"):


There's a difference between getattr and hasattr here. We should do getattr since if the attribute exists but it's none, we shouldn't fail.

if the attribute exists but it's none, we shouldn't fail.

Isn't that what's happening with the existing code?

In [1]: class A: ...: def __init__(self): ...: self.x = None ...: In [2]: hasattr(A(), "x") Out[2]: True

Also, it seems that you can't do __slots__ = None in the first place.

@adrinjalali I think that __slots__ cannot be None:

Cell In[33], line 1 ----> 1 class A: 2 __slots__ = None TypeError: 'NoneType' object is not iterable

Edit: arfff, GitHub latency :)

Sure, but if the code in cpython does getattr, I rather stay on that safer side. I don't know why they did that, and I'm not sure if I wanna know why lol

I think there can be a C extension type which exposes a __slots__=None, and that's why getattr(obj, '__slots__', None) is more correct than getattr(obj, '__slots__').

But what about __slots__ = [], that should also trigger the error. We would have to additionally check for empty list and empty tuple.

I wouldn't try to be more correct than cpython, and I'm not sure what __slot__=[] would mean. I can understand __slots__=None would be equivalent to it not existing.

On having to understand the subtle differences between getattr and hasattr: I'm with Adrin on the "there must be a reason the CPython devs used that, even if we don't fully understand why".

Okay, I changed to if getattr(self, "__slots__", None)

glemaitre

LGTM

doc/whats_new/v1.2.rst

sklearn/base.py

sklearn/tests/test_base.py

Co-authored-by: Guillaume Lemaitre <[email protected]>

Co-authored-by: Adrin Jalali <[email protected]> Co-authored-by: Guillaume Lemaitre <[email protected]> Python 3.11 introduces `__getstate__` on the `object` level, which breaks our existing `__getstate__` code for objects w/o any attributes. This fixes the issue.

adrinjalali reviewed Dec 13, 2022

View reviewed changes

BenjaminBossan and others added 4 commits December 13, 2022 17:43

Reviewer comment: better comment

db90a42

Co-authored-by: Adrin Jalali <[email protected]>

Reviewer comment: better comment

d175c86

Co-authored-by: Adrin Jalali <[email protected]>

Remove incorrect comment

7b812db

Add test for bug that uses pickle

fd4a2bb

adrinjalali added the No Changelog Needed label Dec 14, 2022

adrinjalali approved these changes Dec 14, 2022

View reviewed changes

adrinjalali added the Waiting for Second Reviewer First reviewer is done, need a second one! label Dec 14, 2022

betatim reviewed Dec 14, 2022

View reviewed changes

adrinjalali added this to the 1.2.1 milestone Dec 14, 2022

glemaitre changed the title ~~Bugfix: BaseEstimator __getstate__ in Python 3.11~~ FIX update BaseEstimator __getstate__ in Python 3.11 Dec 15, 2022

glemaitre self-requested a review December 15, 2022 13:39

glemaitre changed the title ~~FIX update BaseEstimator __getstate__ in Python 3.11~~ FIX fix pickling for empty object with Python 3.11+ Dec 15, 2022

glemaitre reviewed Dec 15, 2022

View reviewed changes

betatim removed the No Changelog Needed label Dec 15, 2022

BenjaminBossan and others added 2 commits December 15, 2022 16:30

Reviewer: add comment

18b15cf

Co-authored-by: Guillaume Lemaitre <[email protected]>

Reviewer: add comment

a37cf9d

Co-authored-by: Guillaume Lemaitre <[email protected]>

Address reviewer comments

ecc0ef5

- Add entry to changelog - Check if slots are used, raise TypeError - Refactor tests to use parametrize, test __getstate__ and pickle in same test

adrinjalali reviewed Dec 15, 2022

View reviewed changes

Use __getattr__ instead of __hasattr__

bc91dd5

glemaitre approved these changes Dec 16, 2022

View reviewed changes

doc/whats_new/v1.2.rst Outdated Show resolved Hide resolved

sklearn/base.py Outdated Show resolved Hide resolved

sklearn/tests/test_base.py Outdated Show resolved Hide resolved

sklearn/tests/test_base.py Outdated Show resolved Hide resolved

BenjaminBossan and others added 5 commits December 16, 2022 11:40

Reviewer: markup fix

d9c16e5

Co-authored-by: Guillaume Lemaitre <[email protected]>

Reviewer: phrasing error message

2896b01

Co-authored-by: Guillaume Lemaitre <[email protected]>

Reviewer: remove empty line

cafa294

Co-authored-by: Guillaume Lemaitre <[email protected]>

Reviewer: markup fix

b161c67

Co-authored-by: Guillaume Lemaitre <[email protected]>

empty commit to trigger circle ci from my account

cfa9fa8

adrinjalali enabled auto-merge (squash) December 16, 2022 10:46

betatim approved these changes Dec 16, 2022

View reviewed changes

adrinjalali merged commit 9017c70 into scikit-learn:main Dec 16, 2022

BenjaminBossan deleted the bugfix-getstate-python-3.11 branch December 16, 2022 12:00

adrinjalali mentioned this pull request Mar 13, 2023

Unexpected behavior with __dict__ as a class attribute python/cpython#102648

Open

	try:
	state = super().__getstate__()
	except AttributeError:
	state = self.__dict__.copy()

	if type(self).__module__.startswith("sklearn."):
	return dict(state.items(), _sklearn_version=__version__)

		# TODO: Remove once Python < 3.11 is dropped, as there will never be
		# an AttributeError

FIX fix pickling for empty object with Python 3.11+ #25188

FIX fix pickling for empty object with Python 3.11+ #25188

Conversation

BenjaminBossan commented Dec 13, 2022

adrinjalali left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adrinjalali commented Dec 13, 2022

BenjaminBossan commented Dec 13, 2022

adrinjalali left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adrinjalali commented Dec 14, 2022 • edited Loading

betatim commented Dec 14, 2022

BenjaminBossan commented Dec 14, 2022 • edited Loading

betatim Dec 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adrinjalali commented Dec 14, 2022

adrinjalali commented Dec 15, 2022

BenjaminBossan commented Dec 15, 2022

lesteve commented Dec 15, 2022

BenjaminBossan commented Dec 15, 2022

lesteve commented Dec 15, 2022 • edited Loading

adrinjalali commented Dec 15, 2022

glemaitre left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

betatim commented Dec 15, 2022

glemaitre commented Dec 15, 2022 • edited Loading

glemaitre commented Dec 15, 2022

betatim commented Dec 15, 2022

adrinjalali commented Dec 15, 2022

glemaitre commented Dec 15, 2022 • edited Loading

adrinjalali commented Dec 15, 2022

BenjaminBossan commented Dec 15, 2022

adrinjalali commented Dec 15, 2022

BenjaminBossan commented Dec 15, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glemaitre Dec 15, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glemaitre left a comment

Choose a reason for hiding this comment

adrinjalali commented Dec 14, 2022 •

edited

Loading

BenjaminBossan commented Dec 14, 2022 •

edited

Loading

betatim Dec 14, 2022 •

edited

Loading

lesteve commented Dec 15, 2022 •

edited

Loading

glemaitre commented Dec 15, 2022 •

edited

Loading

glemaitre commented Dec 15, 2022 •

edited

Loading

glemaitre Dec 15, 2022 •

edited

Loading