[MRG+2] Moves Imputation out of Preprocessing #10483

thechargedneutron · 2018-01-16T20:27:35Z

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Changes done as per suggestions

Tasks

copy sklearn/preprocessing/imputation.py to sklearn/impute.py, as well as the corresponding tests.
deprecate Imputer in sklearn/preprocessing/imputation.py to be removed in v0.22.
create a sklearn.impute section in doc/modules/classes.rst
update the deprecated section at the bottom of doc/modules/classes.rst
update sklearn/__init__.py's __all__
move imputation documentation from doc/modules/preprocessing.rst to doc/modules/impute.rst
we might also want to rename Imputer in the new module to SimpleImputer, DummyImputer or something
update any references to Imputer (in sklearn/, doc/ or examples/) to refer to the new location.

jnothman · 2018-01-16T22:29:08Z

thanks! could we adopt SimpleImputer or NaiveImputer or ConstantImputer (choose one) for now and we can change it with sed later?

jnothman

You need to add impute.rst to the user guide table of contents

jnothman · 2018-01-16T23:52:47Z

doc/glossary.rst

@@ -480,7 +480,7 @@ General Concepts

    missing values
        Most Scikit-learn estimators do not work with missing values. When they
-        do (e.g. in :class:`preprocessing.Imputer`), NaN is the preferred
+        do (e.g. in :class:`Imputer`), NaN is the preferred


This still needs to be impute.Imputer

jnothman · 2018-01-16T23:53:28Z

sklearn/preprocessing/imputation.py

@@ -60,6 +61,8 @@ def _most_frequent(array, extra_value, n_repeat):
            return extra_value


+@deprecated("Imputer was deprecated in version 0.20 and will be "
+            "removed in 0.22. Import Imputer from sklearn instead.")


sklearn.impute

jnothman · 2018-01-16T23:56:25Z

doc/modules/classes.rst

@@ -609,6 +609,25 @@ Kernels:
   isotonic.check_increasing
   isotonic.isotonic_regression

+.. _impute_ref:


Is this referenced anywhere?

I am not familiar with reST, I am not sure why there's _ref here. This was with other modules here so I used this :( .

you need this only if it is referenced somewhere. But I guess if it is not referenced now, it can be in the futur so it can stay here

jnothman · 2018-01-17T00:00:46Z

doc/modules/impute.rst

@@ -0,0 +1,54 @@
+


You need to remove this content from the preprocessing docs. You're welcome to leave a "see also" reference to these docs in preprocessing if you think that will help users.

…into imputer

jnothman · 2018-01-17T12:49:17Z

sklearn/preprocessing/imputation.py

@@ -62,7 +62,8 @@ def _most_frequent(array, extra_value, n_repeat):


 @deprecated("Imputer was deprecated in version 0.20 and will be "
-            "removed in 0.22. Import Imputer from sklearn instead.")
+            "removed in 0.22. Import import.Imputer from sklearn"


Should be impute.Imputer

jnothman · 2018-01-17T12:51:03Z

Will you do the SimpleImputer rename?

thechargedneutron · 2018-01-17T12:52:56Z

Will you do the SimpleImputer rename?

Yeah, Sure.

…into imputer

jnothman

Thanks!

please use @ignore_warnings on all tests in sklearn.preprocessing.tests.test_imputation.

Please don't change test_cross_validation.py and test_grid_search.py. They're about to be removed and you only make merge conflicts.

jnothman · 2018-01-17T21:13:29Z

sklearn/preprocessing/imputation.py

@@ -60,6 +61,9 @@ def _most_frequent(array, extra_value, n_repeat):
            return extra_value


+@deprecated("Imputer was deprecated in version 0.20 and will be "
+            "removed in 0.22. Import impute.Imputer from sklearn"


now SimpleImputer

jnothman

LGTM!

jnothman · 2018-01-17T21:29:23Z

sklearn/impute.py

@@ -0,0 +1,375 @@
+# Authors: Nicolas Tresegnie <[email protected]>


Please add a short docstring: "Transformers for missing value imputation"

jnothman

You didn't import ignore_warnings

Thanks for the efficient work

thechargedneutron · 2018-01-17T21:44:30Z

@jnothman Thanks for the quick review. Just out of curiosity, have you moved to a new time zone? ;) It looks so from your review time.

jnothman · 2018-01-17T21:54:44Z

No, I haven't. I'm just sneaking in an early morning review before disappearing for a few days. Hopeful we can merge this and get on with business.

Please add an API changes entry to what's new

thechargedneutron · 2018-01-17T22:27:35Z

_____________________________ [doctest] impute.rst _____________________________
 also allows for different missing values encodings.
 
 The following snippet demonstrates how to replace missing values,
 encoded as ``np.nan``, using the mean value of the columns (axis 0)
 that contain the missing values::
 
     >>> import numpy as np
     >>> from sklearn.impute import SimpleImputer
     >>> imp = SimpleImputer(missing_values='NaN', strategy='mean', axis=0)
     >>> imp.fit([[1, 2], [np.nan, 3], [7, 6]])
Expected:
    SimpleImputer(axis=0, copy=True, missing_values='NaN', strategy='mean', verbose=0)
Got:
    SimpleImputer(axis=0, copy=True, missing_values='NaN', strategy='mean',
           verbose=0)

Why this doctest failed? The two results are same. Also, there's a failure in pep8 which says l is ambigious variable. I have not made any such changes.

jnothman · 2018-01-17T22:41:43Z

Use the normalize whitespace flag for doctest

jnothman · 2018-01-17T22:44:19Z

I think the ambiguous variable name just means some code was unnecessarily repeated in the test. You can fix it

jnothman · 2018-02-13T11:19:31Z

of course, silly me. we wasted a while lot of effort on that parameter deprecation. let's just drop it from the new version

qinhanmin2014 · 2018-02-14T03:12:32Z

I'm OK with a deprecation warning in the old version and directly dropping axis parameter in the new version.

jnothman · 2018-02-14T03:17:15Z

We don't exactly need a deprecation warning in the old version if the entire module is deprecated...?

…

On 14 February 2018 at 14:12, Hanmin Qin ***@***.***> wrote: I'm OK with a deprecation warning in the old version and directly dropping axis parameter in the new version. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10483 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz61yXPouOaGvUPBOIZRUnbJ3M48RZks5tUk8kgaJpZM4RgYMJ> .

qinhanmin2014 · 2018-02-14T03:25:20Z

We don't exactly need a deprecation warning in the old version if the entire module is deprecated...?

Not sure, I'll revert the deprecation if you have made the decision, but I think users will expect the new version to behave the same as the old version. It seems strange to remove a parameter without any warning when moving the class, so I would prefer a deprecation warning at least in the old version (maybe also in the new version from my side).

jnothman · 2018-02-14T03:30:41Z

Why not just having what's new and warning say: "sklearn.preprocessing.Imputer has been renamed to sklearn.impute.SimpleImputer, and its axis parameter is no longer available. The deprecated Imputer will be removed in version 0.22." and if the user uses Imputer(axis=1), can add "Imputation on axis=1 can be implemented with FunctionTransformer."

…

On 14 February 2018 at 14:25, Hanmin Qin ***@***.***> wrote: We don't exactly need a deprecation warning in the old version if the entire module is deprecated...? Not sure, I'll revert the deprecation if you have made the decision, but I think users will expect the new version to behave the same as the old version. It seems strange to remove a parameter without any warning when moving the class, so I would prefer a deprecation warning at least in the old version (maybe also in the new version from my side). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10483 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz6102tUyZvG62iD-yclUTsrAQ8NNZks5tUlIhgaJpZM4RgYMJ> .

qinhanmin2014 · 2018-02-14T08:49:41Z

Why not just having what's new and warning say: "sklearn.preprocessing.Imputer has been renamed to sklearn.impute.SimpleImputer, and its axis parameter is no longer available. The deprecated Imputer will be removed in version 0.22."

I think it's a better way. @jnothman revert #10558?

jnothman · 2018-02-14T09:35:02Z

Yes, let's go ahead when this is ready to merge.

…

On 14 February 2018 at 19:49, Hanmin Qin ***@***.***> wrote: Why not just having what's new and warning say: "sklearn.preprocessing.Imputer has been renamed to sklearn.impute.SimpleImputer, and its axis parameter is no longer available. The deprecated Imputer will be removed in version 0.22." I think it's a better way. @jnothman <https://github.com/jnothman> revert #10558 <#10558>? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10483 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz62pt7aiOG8hfVm_XmnP3kZRIhRWQks5tUp4qgaJpZM4RgYMJ> .

qinhanmin2014 · 2018-02-14T10:18:06Z

@jnothman I've created #10635 to revert #10558.
Finishing all the work here is fine but I'm also OK to merge without the deprecation of axis parameter (I think we can leave this to another PR).

jnothman · 2018-02-14T10:41:58Z

Okay. Let's do it that way.

jnothman · 2018-02-14T10:43:33Z

Thanks @thechargedneutron

jnothman · 2018-02-14T10:44:19Z

Do you want to remove axis and update the what's new, @qinhanmin2014, or should we open an issue?

qinhanmin2014 · 2018-02-14T10:59:00Z

@jnothman Please open a new issue. I'll try to take care of it after couple of days if no one takes it (it's Chinese new year :) )

jnothman · 2018-02-14T11:20:43Z

of course

sklearn-lgtm · 2018-02-14T11:35:08Z

This pull request introduces 1 alert - view on lgtm.com

new alerts:

1 for Explicit export is not defined

Comment posted by lgtm.com

qinhanmin2014

LGTM. I've resolved the conflicts and checked the PR. merge? @jnothman @glemaitre

jnothman · 2018-02-14T19:47:27Z

Ah, I thought I'd already merged this. I must have been blocked by a merge conflict and not seen.

jnothman · 2018-02-14T19:48:41Z

Thanks a lot @qinhanmin2014 for pointing out our folly

jnothman · 2018-02-24T19:36:14Z

We failed to remove references to Imputer from sklearn/modules/preprocessing.rst :( Needs to be done.

thechargedneutron · 2018-02-24T19:44:09Z

I'll do this. :) Are you referring to doc/modules/preprocessing.rst ?
Also, the whole of Imputation section is to be removed, right?

jnothman · 2018-02-25T04:14:30Z

Yes

thechargedneutron added 2 commits January 17, 2018 00:56

files copied to the appropriate location

73e58e6

all references to preprocessing.impute changed to impute

f914ab0

thechargedneutron changed the title ~~[WIP] Moves Imputation out of Preprocessing~~ [MRG] Moves Imputation out of Preprocessing Jan 16, 2018

jnothman reviewed Jan 17, 2018

View reviewed changes

thechargedneutron added 2 commits January 17, 2018 18:10

added suggested changes

6e4e512

Merge branch 'master' of https://github.com/scikit-learn/scikit-learn …

7672c2f

…into imputer

jnothman reviewed Jan 17, 2018

View reviewed changes

typo resolved

d8680b5

thechargedneutron added 2 commits January 18, 2018 02:30

Imputer changed to SimpleImputer

ea1f3c3

Merge branch 'master' of https://github.com/scikit-learn/scikit-learn …

2639675

…into imputer

jnothman requested changes Jan 17, 2018

View reviewed changes

thechargedneutron added 2 commits January 18, 2018 02:50

ignore_warnings added

86f8092

changes reverted in two files

43a64a0

jnothman approved these changes Jan 17, 2018

View reviewed changes

jnothman changed the title ~~[MRG] Moves Imputation out of Preprocessing~~ [MRG+1] Moves Imputation out of Preprocessing Jan 17, 2018

jnothman reviewed Jan 17, 2018

View reviewed changes

thechargedneutron added 2 commits January 18, 2018 03:11

short doc added

75a4d03

import ignore_warning added

a6d1f72

thechargedneutron added 2 commits January 18, 2018 03:29

what's new added

550c41d

pep8 corrected

ba87c8e

thechargedneutron added 2 commits January 18, 2018 04:22

other failures checked

5c7c1f6

indentation corrected

cc6bdec

qinhanmin2014 mentioned this pull request Feb 14, 2018

[MRG] revert #10558 Deprecate axis parameter in imputer #10635

Merged

DOC Module docstring

6798e68

This was referenced Feb 14, 2018

[MRG+1] MissingIndicator transformer #8075

Merged

[MRG] Added k-Nearest Neighbor imputation for missing data #9212

Closed

Merge branch 'master' into imputer

393c16c

jnothman mentioned this pull request Feb 14, 2018

Remove SimpleImputer's axis parameter #10636

Closed

resolve conflict

2210504

qinhanmin2014 approved these changes Feb 14, 2018

View reviewed changes

qinhanmin2014 changed the title ~~[MRG+1] Moves Imputation out of Preprocessing~~ [MRG+2] Moves Imputation out of Preprocessing Feb 14, 2018

jnothman merged commit 71b29ac into scikit-learn:master Feb 14, 2018

jnothman mentioned this pull request Feb 24, 2018

Remove Imputer references in doc/modules/preprocessing.rst #10689

Closed

		@@ -0,0 +1,375 @@
		# Authors: Nicolas Tresegnie <[email protected]>

Uh oh!

[MRG+2] Moves Imputation out of Preprocessing #10483

[MRG+2] Moves Imputation out of Preprocessing #10483

Uh oh!

Conversation

thechargedneutron commented Jan 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Tasks

Uh oh!

jnothman commented Jan 16, 2018 via email

Uh oh!

jnothman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jnothman commented Jan 17, 2018

Uh oh!

thechargedneutron commented Jan 17, 2018

Uh oh!

jnothman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jnothman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jnothman left a comment

Choose a reason for hiding this comment

Uh oh!

thechargedneutron commented Jan 17, 2018

Uh oh!

jnothman commented Jan 17, 2018

Uh oh!

thechargedneutron commented Jan 17, 2018

Uh oh!

jnothman commented Jan 17, 2018

Uh oh!

jnothman commented Jan 17, 2018

Uh oh!

jnothman commented Feb 13, 2018 via email

Uh oh!

qinhanmin2014 commented Feb 14, 2018

Uh oh!

jnothman commented Feb 14, 2018 via email

Uh oh!

qinhanmin2014 commented Feb 14, 2018

Uh oh!

jnothman commented Feb 14, 2018 via email

Uh oh!

qinhanmin2014 commented Feb 14, 2018

Uh oh!

jnothman commented Feb 14, 2018 via email

Uh oh!

qinhanmin2014 commented Feb 14, 2018

Uh oh!

jnothman commented Feb 14, 2018

Uh oh!

jnothman commented Feb 14, 2018

Uh oh!

jnothman commented Feb 14, 2018

Uh oh!

qinhanmin2014 commented Feb 14, 2018

thechargedneutron commented Jan 16, 2018 •

edited

Loading