[MRG] ChainedImputer -> IterativeImputer, and documentation update #11350

sergeyf · 2018-06-25T02:49:26Z

Addresses two points as discussed in #11259:
(a) Removes the "average last n_imputations behavior". Now there just one parameter n_iter instead of the two: n_burn_in and n_imputations.
(b) New flag: sample_after_predict, which is False by default. If true it will sample from the predictive posterior after predicting during each round-robin iteration. Turning it on will make ChainedImputer run a single

@jnothman @glemaitre: I think ChainedImputer needs a new check: if sample_after_predict=True, then the predictor needs to have return_std as a parameter of its predict method. What's best practice for checking something like that?

Also, we may want a new example of how to use ChainedImputer for the purpose of a MICE-type analysis, and how to use ChainedImputer with a RandomForest as the predictor instead of BayesianRidge to demonstrate missForest functionality.

I think @RianneSchouten would be a good candidate for the MICE example as a contribution to this PR or a new one, and I can stick ChainedImputer + RF into plot_missing_values.py as another bar on the plot. Let me know how that sounds.

jnothman · 2018-06-25T03:56:57Z

CIs failing, FYI

sergeyf · 2018-06-25T05:43:11Z

Fixed, thank you!

RianneSchouten · 2018-06-25T05:51:24Z

I will work on the example next wednesday and make a PR then. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virusvrij. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> 2018-06-25 7:44 GMT+02:00 Sergey Feldman <[email protected]>:

…

Fixed, thank you! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11350 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AVQqew6SZKjfI7oEWczNqwS6XHmkYw5zks5uAHihgaJpZM4U1b8P> .

jnothman · 2018-06-25T05:37:52Z

sklearn/impute.py

-        Number of initial imputation rounds to perform the results of which
-        will not be returned.
+    n_iter : int, optional (default=10)
+        Number of imputation rounds to perform before returning the final


This should probably clarify that a round indicates a single imputation of each feature

Remove final here.

jnothman · 2018-06-25T05:40:26Z

sklearn/impute.py

+
+    sample_after_predict : boolean, default=False
+        Whether to sample from the predictive posterior of the fitted
+        predictor for each Imputation. Set to ``True`` if using


Assuming gaussian posterior. Predictor requires return_std support in predict.

jnothman · 2018-06-25T06:16:17Z

sklearn/impute.py

+        It must support ``return_std`` in its ``predict`` method if
+        ``sample_after_predict`` option is set to ``True`` below.
+
+    sample_after_predict : boolean, default=False


I think this name is unnecessarily verbose.

Brainstorming.

fill_with='expectation' vs 'sample'

random_impute bool

use_std

sampled

random_draw

randomized (but there are other random components)

How about predict_posterior bool?

sergeyf · 2018-06-25T14:44:46Z

@stefvanbuuren This PR has a lot of the documentation changes you suggested. Please take a look at what's here and let me know if we're addressing your concerns. What's left to do is to make an example that actually demonstrates how to use ChainedImputer as MICE.

sergeyf · 2018-06-25T15:03:27Z

@jnothman I'd like to raise an error if a predictor without return_std is passed in when predict_posterior=True. The only way I've found to do this is the inspect module as follows:

predictor = BayesianRidge()
'return_std' in list(inspect.signature(predictor.predict).parameters.keys()) # True

predictor = RandomForestRegressor()
'return_std' in list(inspect.signature(predictor.predict).parameters.keys()) # False

OK to rely on the inspect module in such a fashion in the middle of impute.py?

jorisvandenbossche

@sergeyf Thanks a lot for keeping up making changes based on the discussion!
Added some quick comments on the docs.

jorisvandenbossche · 2018-06-25T15:03:56Z

doc/modules/impute.rst

-Then, the regressor is used to predict the unknown values of `y`. This is repeated
-for each feature in a chained fashion, and then is done for a number of imputation
-rounds. Here is an example snippet::
+A more sophisticated approach is to use the :class:`ChainedImputer` class, models


"model" -> "which models" ?

Whoops, thanks.

jorisvandenbossche · 2018-06-25T15:05:31Z

doc/modules/impute.rst

+estimate for imputation. It does so in an iterated round-robin fashion: at each step,
+a feature column is designated as output `y` and the other feature columns are treated
+as inputs `X`. A regressor is fit on `(X, y)` for known `y`. Then, the regressor is
+used to predict the unknown values of `y`. This is repeated for each feature in a


"unknown" -> "missing"? (only a suggestion, the unknown is fine as well, but the missing might be more clear since it about imputing the missing values)

Changed, thanks.

jorisvandenbossche · 2018-06-25T15:08:10Z

sklearn/impute.py

-    of the features are Gaussian.
+    Basic implementation of chained mutual regressions to find replacement
+    values in multivariate missing data. This version assumes all features
+    are Gaussian.


Is it needed to mention this restriction of Gaussian? As I assume it depends on the predictor one uses.

So far yes. The posterior sampling is done via a call to predict that also returns the standard deviation and then we sample from a Gaussian posterior. It's a limitation at the moment.

jorisvandenbossche · 2018-06-25T15:09:47Z

doc/modules/impute.rst

+chained fashion, and then is done for a number of imputation rounds. The results
+of the final imputation round are returned. Our implementation was inspired by the
+R MICE package (Multivariate Imputation by Chained Equations), but differs from
+it in setting single imputation to default instead of multiple imputation. This


I find the "setting single imputation to default" still confusing.
As AFAIK, the ChainedImputer itself can only do single imputation, that's not a matter of its default settings. You can of course then use it multiple times (setting the appropriate arguments for this case) to do multiple imputation.

Yes, you're right, but by now users have come to expect that "chained equations" refers to multiple imputation, so the combination "chained + single" would confuse quite a few.

stefvanbuuren · 2018-06-25T15:47:25Z

@sergeyf I think it is clear now that your method does single imputation by default.

Perhaps for balance also mention that what you describe as "the most common use case" differs from what is generally recommended by the statistical community (you can refer to Little & Rubin (2002, Ch 4) and Rubin (1987, Ch 1)), and only then say it's an open problem.

sergeyf · 2018-06-25T16:09:45Z

@stefvanbuuren this is the current full paragraph, which starts with the point about it being different in the statistics community:

In the statistics community, it is common practice to perform multiple imputations,
generating, for example, 10 separate imputations for a single feature matrix.
Each of these 10 imputations is then put through the subsequent analysis pipeline
(e.g. feature engineering, clustering, regression, classification). The 10 final
analysis results (e.g. held-out validation error) allow the data scientist to
obtain understanding of the uncertainty inherent in the missing values. The above
practice is called multiple imputation. As implemented, the :class:`ChainedImputer`
class generates a single imputation for each missing value because this
is a common use case for machine learning applications. However, it can also
be used for multiple imputations by applying it repeatedly to the same dataset with
different random seeds.

I'll add a parenthetical reference to it.

jnothman · 2018-06-25T23:20:01Z

I don't really like predict_posterior as the difference between the settings is not about whether prediction is from the posterior, but about whether we use the expectation or a random draw from that posterior

I don't think it is essential to raise an explicit error for the case that the estimator lacks return_std: the default TypeError will be informative enough when it tries to predict, and users who want to use the sampling variant will likely have read it's documentation which mentions return_std

sergeyf · 2018-06-25T23:22:44Z

OK. thanks. I'll change the name to posterior_draw?

The only thing decision left is whether to change the name to IterativeImputer.

jnothman · 2018-06-25T23:34:22Z

I'm +1 for IterativeImputer. But I was +1 for ChainedImputer too :) But I suppose then it was more urgent to make it anything but MICE :P

…

On 26 June 2018 at 09:22, Sergey Feldman ***@***.***> wrote: OK. thanks. I'll change the name to posterior_draw? The only thing decision left is whether to change the name to IterativeImputer. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11350 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz6wyVh0wTjWhKlGBhwQYw3g7bRw8Bks5uAXDHgaJpZM4U1b8P> .

sergeyf · 2018-06-26T01:30:45Z

@glemaitre @jnothman I changed the default to be RidgeCV in case sample_posterior=False. One thing that I saw is that some of the tests started failing. It looks like the default alphas in RidgeCV are only [0.1, 1, 10]. I'm worried that's too much regularization. So, I also added a small value in case no regularization is needed:

self._predictor = RidgeCV(alphas=np.array([1e-5, 0.1,  1, 10]))

Let me know if there are any other gotchas I should be aware of in instantiating a default RidgeCV.

Thanks!

jnothman

Please check that sample_posterior=False is deterministic if n_nearest_features is None.

I've also realised that n_nearest_features documentation fails to mention that the features are not necessarily the nearest, but are drawn with probability proportional to correlation.

(And now that we're supporting non-linear regresssors, I wonder if we should be using a rank correlation in n_nearest_features, or max(spearman_rho, pearson_rho) or something nasty like that. This n_nearest_features is useful but might get in the way of good results with non-linear prediction.)

Thanks for all this great work. I think when we're finished here, we'll have a very powerful and useful tool.

jnothman · 2018-06-26T02:38:32Z

doc/modules/impute.rst

+As implemented, the :class:`IterativeImputer` class generates a single imputation
+for each missing value because this is the most common use case for machine learning
+applications. However, it can also be used for multiple imputations by applying it
+repeatedly to the same dataset with different random seeds.


Unless you're using a posterior sample or perhaps a highly randomised regressor, I don't think there is quite enough randomness in IterativeImputer for it to be an appropriate method of multiple imputation.

Right, I should change this to say set sample_posterior=True.

jnothman · 2018-06-26T02:41:24Z

doc/modules/impute.rst

+See Chapter 4 of "Statistical Analysis with Missing Data" by Little and Rubin for
+more discussion on multiple vs. single imputations.
+
+It is still an open problem as to how useful single vs. multiple imputation is in


"when the user is not interested in measuring uncertainty due to missing values"

Good addendum, thanks.

jnothman · 2018-06-26T02:45:18Z

sklearn/impute.py

-    n_imputations : int, optional (default=100)
-        Number of chained imputation rounds to perform, the results of which
-        will be used in the final average.
+    n_iter : int, optional (default=10)


There was a suggestion to call this max_iter and consider early stopping in a later pr

I'd prefer to leave it as-is until we actually support early stopping. The word max would be confusing as it is now.

If we agree that in the future we would like to do the early stopping, I would already rename it now. It's true that this will be a bit confusing, but having to rename it after it is released also doesn't seem a good option.

This will be one point to address in future questions about handling non-MICE cases

jnothman · 2018-06-26T02:56:00Z

sklearn/impute.py

@@ -498,18 +505,24 @@ class ChainedImputer(BaseEstimator, TransformerMixin):

    Attributes
    ----------
-    initial_imputer_ : object of class :class:`sklearn.preprocessing.Imputer`'
-        The imputer used to initialize the missing values.
+    initial_imputer_ : object of type :class:`sklearn.impute.SimpleImputer`


Good catch! I was thinking we should make it possible to set initial_strategy to an imputer, to allow something non-parametric, like the SamplingImputer which may or may not appear soon.

Another thing I'd like to defer to future Sergey =)

jnothman · 2018-06-26T02:57:07Z

sklearn/impute.py

+                self._predictor = BayesianRidge()
+            else:
+                from .linear_model import RidgeCV
+                # including a very small alpha to approximate OLS


I do wonder if we should just be using OLS. But for near-parity with the sample_posterior case, this makes sense.

jnothman · 2018-06-26T03:02:01Z

sklearn/impute.py

-        # then there is no need to do burn in and the result should be
-        # just the initial imputation (before clipping)
-        if self.n_imputations < 1:
+        # edge case: in case the user specifies 0 for n_burn_in,


jnothman · 2018-06-26T03:03:02Z

sklearn/impute.py

            return X_filled

-        X_filled = np.clip(X_filled, self._min_value, self._max_value)
+        # clip only the initial filledin values


I don't really get why we would expect the initial imputation to require clipping.

Huh, that's a good point. What I should probably do is this:

if self.n_iter < 1: X_filled[mask_missing_values] = np.clip(X_filled[mask_missing_values], self._min_value, self._max_value) return X_filled

Or actually just not clip before imputing.

jnothman · 2018-06-26T03:03:42Z

sklearn/impute.py

                if self.verbose > 1:
-                    print('[ChainedImputer] Ending imputation round '
+                    print('[IterativeImputer] Ending imputation round '


This isn't covered in tests, FWIW. It should be covered to ensure there's no AttributeError or whatever

jnothman · 2018-06-26T03:05:07Z

sklearn/tests/test_impute.py

@@ -486,44 +486,25 @@ def test_imputation_copy():
    # made, even if copy=False.


-def test_chained_imputer_rank_one():


This should still be valid if we run multiple imputations and take the mean, shouldn't it? Let's try leave the tests in.

It still is: I changed the name to test_iterative_imputer_rank_one and moved it to be with the other functional tests.

jnothman · 2018-06-26T03:06:07Z

sklearn/tests/test_impute.py

@@ -614,17 +592,17 @@ def test_chained_imputer_missing_at_transform(strategy):
                  initial_imputer.transform(X_test)[:, 0])


-def test_chained_imputer_transform_stochasticity():


This should be run with and without sample_posterior=True

jnothman · 2018-09-03T03:25:01Z

A couple of minor merge conflicts will need resolution before we continue here. The what's new entry for ChainedImputer is now in v0.21.rst.

sergeyf · 2018-09-03T20:37:31Z

@jnothman This was a strange merge conflict. It looked like both of the conflicting code snippets were something to add, so I did that. Please take a look and see if you agree. What are next steps here?

Impute not part of 0.20!

jnothman · 2018-09-04T00:07:26Z

I think you did an unfinished rename of a variable:

flake8 failures

examples/plot_missing_values.py:81:5: F841 local variable 'chained_impute_scores' is assigned to but never used
    chained_impute_scores = cross_val_score(estimator, X_missing, y_missing,
    ^
examples/plot_missing_values.py:87:14: F821 undefined name 'iterative_impute_scores'
            (iterative_impute_scores.mean(), iterative_impute_scores.std()))
             ^
examples/plot_missing_values.py:87:46: F821 undefined name 'iterative_impute_scores'
            (iterative_impute_scores.mean(), iterative_impute_scores.std()))
                                             ^

Error in examples:


Unexpected failing examples:
/home/circleci/project/examples/plot_missing_values.py failed leaving traceback:
Traceback (most recent call last):
  File "/home/circleci/project/examples/plot_missing_values.py", line 90, in <module>
    results_diabetes = np.array(get_results(load_diabetes()))
  File "/home/circleci/project/examples/plot_missing_values.py", line 87, in get_results
    (iterative_impute_scores.mean(), iterative_impute_scores.std()))
NameError: name 'iterative_impute_scores' is not defined

sergeyf · 2018-09-04T09:31:08Z

Thanks @jnothman. This is kind of weird: I remember all tests passing just fine before all the unmerging started happening.

sergeyf · 2018-09-04T19:00:09Z

@glemaitre it still says "requested changes" from you. Do you remember what these were?

glemaitre · 2018-09-04T19:22:38Z

I would need to review once more since that a lot of changes have been done. For sure my previous comments have been addressed in some way.

sergeyf · 2018-09-04T19:53:12Z

Great, thanks.

…

On Tue, Sep 4, 2018, 7:23 PM Guillaume Lemaitre ***@***.***> wrote: I would need to review once more since that a lot of changes have been done. For sure my previous comments have been addressed in some way. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11350 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABya7BBNbcaQRCLXAqyWDsaNJev4o45qks5uXtMtgaJpZM4U1b8P> .

sergeyf · 2018-09-12T08:25:32Z

@jnothman any eta for further reviews? I'd like to get this merged and copy the code into fancyimpute while this is going through the approval process here.

jnothman

Minor comments. I'd like to merge this soon, but if you could please pull together a list of remaining questions/issues/goals, @sergeyf, that would be great

jnothman · 2018-09-13T12:05:27Z

sklearn/impute.py

-    n_imputations : int, optional (default=100)
-        Number of chained imputation rounds to perform, the results of which
-        will be used in the final average.
+    n_iter : int, optional (default=10)


This will be one point to address in future questions about handling non-MICE cases

jnothman · 2018-09-13T12:06:52Z

sklearn/impute.py

+        If ``sample_posterior`` is True, the predictor must support
+        ``return_std`` in its ``predict`` method. Also, if
+        ``sample_posterior=True`` the default predictor will be
+        ``BayesianRidge()`` and ``RidgeCV`` otherwise.


Use () in both cases or neither

I think use a :class: reference in both

jnothman · 2018-09-13T12:09:48Z

sklearn/impute.py

+        Whether to sample from the (Gaussian) predictive posterior of the
+        fitted predictor for each imputation. Predictor must support
+        ``return_std`` in its ``predict`` method if set to ``True``. Set to
+        ``True`` if using ``IterativeImputer`` for multiple imputations.


Although a small random forest with changing seeds might also be a good way to do multiple imputation.

Just to clarify: this statement is in response to me specify "Gaussian"? If so, yes, but we need a return_std for the random forest and I'm not sure what the right way to do this is yet. I think a good way to sample the posterior of the random forest is to pick a prediction of ONE of the n_trees uniformly at random. What we actually care about is sampling from the posterior, not the return_std.

I'm not sure how to pick one of the trees uniformly at random in a sensible way. But the API could be extended to support different ways of sampling the posterior via a non-binary switch here, i.e. sample_posterior='std' or something

Ok I can investigate if we agree it actually makes sense. I am not even sure how to tell if it does.

jnothman · 2018-09-13T12:12:01Z

sklearn/impute.py

-    additional fitting. We do this by storing each feature's predictor during
-    the round-robin ``fit`` phase, and predicting without refitting (in order)
-    during the ``transform`` phase.
+    This implementation was inspired by the R MICE package (Multivariate


I think this can be dropped here, and stay in the user guide

jnothman · 2018-09-13T12:13:24Z

sklearn/impute.py

+    Imputation by Chained Equations), but differs from it by returning a single
+    imputation instead of multiple imputations. However, multiple imputation
+    can be achieved with multiple instances of the imputer with different
+    random seeds run in parallel.


This would need the regressor to use different random seeds, which we don't easily support atm. Something to think about

I don't think so. As long as we can sample from the posterior predictive distribution of the regressor within IterativeImputer (with different random seeds), it's OK not to have different random seeds inside the regressor. Many regressors are deterministic, right?

jnothman · 2018-09-13T12:14:01Z

sklearn/impute.py

+    can be achieved with multiple instances of the imputer with different
+    random seeds run in parallel.
+
+    To support imputation in inductive mode we store each feature's predictor


I feel like, again, this might be User Guide material. But ambivalent

I'd like to keep as much verbosity as possible, so I'll take your ambivalence as acquiescence to keep it here =)

sergeyf · 2018-09-14T07:36:35Z

@jnothman thanks for the review.

In terms of outstanding issues, I'm not that sure actually.

I think there was a concern that n_iter should be called max_iter if we ever use early stopping.

We also still need a solid multiple imputation example.

Gael asked for "examples that convey the compromises and help users (and library developers) to make the right choices." I think he meant "what are sane defaults?"

We probably need to do a survey of the packages in R, figure out which ones have been battle tested and are well-respected, and see if IterativeImputer can do what they do as it currently exists.

Am I missing something obvious?

sergeyf · 2018-09-14T07:56:44Z

@jnothman lots of weird test errors: SyntaxError: invalid escape sequence \_ I am almost sure this isn't my fault!

jnothman · 2018-09-15T11:34:06Z

I've merged in master where that CI issue was fixed.

jnothman · 2018-09-15T11:35:18Z

Yes, I think you're right to make further changes driven by examples: we want an example that illustrates something along the lines of missforest functionality, and another along the lines of MICE, and use that to identify appropriate. default behaviour.

jnothman · 2018-09-15T11:36:36Z

sklearn/impute.py

@@ -462,7 +462,8 @@ class IterativeImputer(BaseEstimator, TransformerMixin):
        If ``sample_posterior`` is True, the predictor must support
        ``return_std`` in its ``predict`` method. Also, if
        ``sample_posterior=True`` the default predictor will be
-        ``BayesianRidge()`` and ``RidgeCV`` otherwise.
+        ``:class:sklearn.linear_model:BayesianRidge()`` and


This is not the right syntax. Git grep :class: for examples

sergeyf · 2018-09-15T17:00:11Z

@jnothman I'm concerned that a few examples won't tell us about sensible defaults. What we actually need is a large-enough set of experiments where we sweep meaningful parameters to get a sense of reasonable defaults. What's big enough? How has sklearn previous decided on defaults if it's unclear?

jnothman · 2018-09-16T01:05:36Z

Here we know less about datasets than about existing implementations. But we could use Rianne's amputation to help apply this to assorted classification or regression tasks. But there are a lot of variables in the problem. I actually think defaults are less important here than making sure it's easy to get commonly used functionality, like what you would expect of missforest. One other thing that may be of interest is the ability to impute categorical variables. I.e. have a categorical_features mask and accept a classifier as well as regressor for imputation. But that can certainly come with time.

sergeyf · 2018-09-16T14:43:11Z

Ok makes sense. All tests pass now. I'll wait for this to merge and then build a missForest example on top of what we currently have. Let me know if there's anything else for this PR.

jnothman · 2018-09-16T23:15:37Z

Thanks!

sergeyf added 2 commits June 24, 2018 10:28

first commit

0e415aa

complete functional overhaul

5445012

sergeyf mentioned this pull request Jun 25, 2018

MICE/Multiple Imputation branch #11259

Closed

4 tasks

fixing some documentation

3a2d510

fixing broken test

bf58d92

jnothman previously approved these changes Jun 25, 2018

View reviewed changes

addressing review comments + stef's suggestions for documentation

45f228c

sergeyf closed this Jun 25, 2018

sergeyf reopened this Jun 25, 2018

jorisvandenbossche reviewed Jun 25, 2018

View reviewed changes

fixing docs

b231b05

more update to impute.rst

ecfac5c

sergeyf changed the title ~~[WIP] ChainedImputer update~~ [MRG+1] ChainedImputer update Jun 25, 2018

sergeyf added 3 commits June 25, 2018 18:23

name change again, new default ridgeCV

5b2b181

small bug

cbbc8b0

output changed in impute.rst because of RidgeCV

0c9f64f

jnothman reviewed Jun 26, 2018

View reviewed changes

jnothman changed the title ~~[MRG+1] ChainedImputer update~~ [MRG] ChainedImputer update Jun 26, 2018

Update impute.rst a smidge.

c0c6a4c

jnothman mentioned this pull request Sep 3, 2018

[MRG] Merge IterativeImputer into master branch #11977

Merged

8 tasks

Merge branch 'iterativeimputer' into chained_imputer_update

b772a5a

sergeyf added 3 commits September 3, 2018 20:43

Undoing changes to 0.20 rst

0ba454c

Impute not part of 0.20!

Updating 0.21 changes to IterativeImputer

3744445

Update v0.20 rst to not have any changes

5d6865d

Fix plot_missing_values.py

24bb77b

glemaitre approved these changes Sep 4, 2018

View reviewed changes

jnothman approved these changes Sep 14, 2018

View reviewed changes

Addressing review comments

12a3456

sergeyf closed this Sep 14, 2018

sergeyf reopened this Sep 14, 2018

jnothman reviewed Sep 15, 2018

View reviewed changes

Fixed :class: syntax

222b269

jnothman merged commit a4f2a89 into scikit-learn:iterativeimputer Sep 16, 2018

		@@ -486,44 +486,25 @@ def test_imputation_copy():
		# made, even if copy=False.


		def test_chained_imputer_rank_one():

		@@ -614,17 +592,17 @@ def test_chained_imputer_missing_at_transform(strategy):
		initial_imputer.transform(X_test)[:, 0])


		def test_chained_imputer_transform_stochasticity():

Uh oh!

[MRG] ChainedImputer -> IterativeImputer, and documentation update #11350

[MRG] ChainedImputer -> IterativeImputer, and documentation update #11350

Uh oh!

Conversation

sergeyf commented Jun 25, 2018

Uh oh!

jnothman commented Jun 25, 2018

Uh oh!

sergeyf commented Jun 25, 2018

Uh oh!

RianneSchouten commented Jun 25, 2018 via email

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sergeyf commented Jun 25, 2018

Uh oh!

sergeyf commented Jun 25, 2018

Uh oh!

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stefvanbuuren Jun 25, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stefvanbuuren commented Jun 25, 2018

Uh oh!

sergeyf commented Jun 25, 2018

Uh oh!

jnothman commented Jun 25, 2018

Uh oh!

sergeyf commented Jun 25, 2018

Uh oh!

jnothman commented Jun 25, 2018 via email

Uh oh!

sergeyf commented Jun 26, 2018

Uh oh!

jnothman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

stefvanbuuren Jun 25, 2018 •

edited

Loading