[MRG] Remove SimpleImputer axis parameter (#10636) #10652

giba0 · 2018-02-17T18:33:05Z

Reference Issues/PRs

What does this implement/fix? Explain your changes.

In issue #10636 was requested to remove the axis parameter entries, both in the code and in the document. Also included information about the removal of the axis parameter in the documentation and in what is news.

Any other comments?

No

…into Remove-SimpleImputer-axis-parameter-scikit-learn#10636

sklearn-lgtm · 2018-02-17T19:07:42Z

This pull request introduces 2 alerts - view on lgtm.com

new alerts:

2 for Wrong number of arguments in a call

Comment posted by lgtm.com

jnothman

Thank

jnothman · 2018-02-17T23:04:56Z

doc/whats_new/v0.20.rst

@@ -376,6 +376,13 @@ Imputer
 - Deprecate :class:`preprocessing.Imputer` and move the corresponding module to
  :class:`impute.SimpleImputer`. :issue:`9726` by :user:`Kumar Ashutosh
  <thechargedneutron>`.
+- The `sklearn.preprocessing.Imputer` has been renamed to 


There should already be an entry here about the renaming

jnothman · 2018-02-17T23:08:11Z

sklearn/impute.py

@@ -98,28 +92,23 @@ class SimpleImputer(BaseEstimator, TransformerMixin):
        a new copy will always be made, even if `copy=False`:

        - If X is not an array of floating values;
-        - If X is sparse and `missing_values=0`;
-        - If `axis=0` and X is encoded as a CSR matrix;


This is still relevant. We just assume axis=0 always

jnothman · 2018-02-17T23:08:43Z

sklearn/impute.py


    Notes
    -----
-    - When ``axis=0``, columns which only contained missing values at `fit`


This still applies

jnothman · 2018-02-17T23:17:29Z

sklearn/impute.py

-    - When ``axis=1``, an exception is raised if there are rows for which it is
-      not possible to fill in the missing values (e.g., because they only
-      contain missing values).
+    - The sklearn.preprocessing.Imputer has been renamed to 


I'm not sure this is helpful. People reading here have already found SimpleImputer. It might be helpful to just say that to do imputation across rows, they can use...

jnothman · 2018-02-17T23:18:20Z

sklearn/preprocessing/imputation.py

@@ -63,7 +63,8 @@ def _most_frequent(array, extra_value, n_repeat):

 @deprecated("Imputer was deprecated in version 0.20 and will be "
            "removed in 0.22. Import impute.SimpleImputer from "
-            "sklearn instead.")
+            "sklearn instead."


Missing a space at the end

jnothman · 2018-02-17T23:21:14Z

sklearn/impute.py


                # Count the elements != 0
                mask_non_zeros = sparse.csc_matrix(
                    (mask_valids.astype(np.float64),
                     X.indices,
                     X.indptr), copy=False)
-                s = mask_non_zeros.sum(axis=0)
+                s = mask_non_zeros.sum()


jnothman · 2018-02-17T23:21:33Z

sklearn/impute.py

                n_non_missing = np.add(n_non_missing, s)

            else:
-                sums = X.sum(axis=axis)


jnothman · 2018-02-17T23:21:45Z

sklearn/impute.py

@@ -255,7 +218,7 @@ def _dense_fit(self, X, strategy, missing_values, axis):

        # Mean
        if strategy == "mean":
-            mean_masked = np.ma.mean(masked_X, axis=axis)
+            mean_masked = np.ma.mean(masked_X)


jnothman · 2018-02-17T23:22:00Z

sklearn/impute.py

@@ -270,7 +233,7 @@ def _dense_fit(self, X, strategy, missing_values, axis):
                # recent versions of numpy, which we want to mimic
                masked_X.mask = np.logical_or(masked_X.mask,
                                              np.isnan(X))
-            median_masked = np.ma.median(masked_X, axis=axis)
+            median_masked = np.ma.median(masked_X)


jnothman · 2018-02-17T23:24:16Z

sklearn/impute.py

@@ -363,13 +320,10 @@ def transform(self, X):
                X = X.toarray()

            mask = _get_mask(X, self.missing_values)
-            n_missing = np.sum(mask, axis=self.axis)


This reverts commit 14f2f6d.

This reverts commit a3155c7.

giba0 · 2018-02-18T20:43:22Z

@jnothman Tks!

sklearn-lgtm · 2018-02-18T21:14:18Z

This pull request introduces 2 alerts - view on lgtm.com

new alerts:

2 for Wrong number of arguments in a call

Comment posted by lgtm.com

jnothman

Otherwise this looks good!

jnothman · 2018-02-18T21:59:50Z

sklearn/impute.py

@@ -142,8 +135,7 @@ def fit(self, X, y=None):
            raise ValueError("Can only use these strategies: {0} "
                             " got strategy={1}".format(allowed_strategies,
                                                        self.strategy))
-
-        if self.axis not in [0, 1]:
+        if self.axis:


This should be removed. The attribute no longer exists and the error is no longer relevant

giba0 · 2018-02-19T21:07:13Z

=) tks!

sklearn-lgtm · 2018-02-19T21:40:33Z

This pull request introduces 2 alerts - view on lgtm.com

new alerts:

2 for Wrong number of arguments in a call

Comment posted by lgtm.com

jnothman · 2018-02-19T22:06:04Z

Sorry for my poor reviewing. Tests are not passing. You've forgotten to remove axis parameter from _sparse_fit and _dense_fit

jnothman · 2018-02-20T00:40:09Z

sklearn/impute.py

+            self.statistics_ = self._sparse_fit(X,
+                                                self.strategy,
+                                                self.missing_values,
+                                                self.axis)


!!!!!

Please search the file for any remaining mentions of axis

jnothman · 2018-02-20T01:01:26Z

You have another flake8 error, and it looks like tests are still failing. Please check the travis logs.

jnothman · 2018-02-20T02:43:22Z

Tests still failing. Let us know if you need help

giba0 · 2018-02-24T18:39:05Z

I'm sorry, I'm still a newbie here! I believe it will be all right now. Thank you for your help.

giba0 · 2018-02-24T21:51:27Z

@jnothman I need your help! How can I test my modification locally in travis and appveyor? Because I use the test file and all test passed! Can you help me? For example, travis.ci is triggering an error in the Base.py file, which I have not even worked with.

glemaitre · 2018-02-24T23:47:40Z

The difference between travis and appveyor ithe the OS. The former is linux while the second is windows. Then if you want to replicate a failure, you need to create a python environment with the same version of the different required packages.

glemaitre · 2018-02-25T00:03:31Z

sklearn/impute.py


        # Count the zeros
        if missing_values == 0:
-            n_zeros_axis = np.zeros(X.shape[not axis], dtype=int)


not axis should mean 1 if I am not wrong. By the way not axis is really not intuitive use here.

OK! @glemaitre

qinhanmin2014 · 2018-02-25T01:32:00Z

@gilbertoolimpio You can run the tests locally with pytest sklearn (Also see http://scikit-learn.org/dev/developers/advanced_installation.html#testing). Generally you get the same result across different environments. So if you get no error with pytest sklearn, you'll pass Travis and AppVeyor (except for some special cases).
For current PR, please double check your diff to make sure that you didn't change anything when axis=0 (e.g., See above review).

qinhanmin2014 · 2018-02-25T01:34:05Z

doc/whats_new/v0.20.rst

+- Future (and default) behavior is equivalent to ``axis=0``
+  (impute along columns). Row-wise imputation can be performed with
+  FunctionTransformer (e.g.,
+  ``FunctionTransformer(lambda X: Imputer().fit_transform(X.T).T)``).


Please mention the deprecation of axis parameter here and in the deprecation message of preprocessing.Imputer

OK! @qinhanmin2014

jnothman

Tests are failing. Are you able to understand the log output and what to do?

jnothman · 2018-02-25T22:19:03Z

doc/whats_new/v0.20.rst

@@ -376,6 +376,7 @@ Imputer
 - Deprecate :class:`preprocessing.Imputer` and move the corresponding module to
  :class:`impute.SimpleImputer`. :issue:`9726` by :user:`Kumar Ashutosh
  <thechargedneutron>`.
+- The ``axis`` parameter was deprecated in this version.


deprecated in -> dropped from (or removed from)

Please remove the - at the beginning, and in the next line also. We want readers to see all these points as one.

You can also credit yourself for this work, and reference thus pull request

glemaitre · 2018-03-02T16:29:06Z

@gilbertoolimpio Are you able to make those changes?

qinhanmin2014 · 2018-03-13T04:01:38Z

@gilbertoolimpio Are you still working on it or do you have any difficulties? Otherwise we might let someone else to take it.

glemaitre · 2018-03-16T08:01:20Z

@qinhanmin2014 I think that somebody should take over to get the imputer fixed asap

qinhanmin2014 · 2018-03-16T13:42:38Z

@gilbertoolimpio Thanks for your great work so far :)

@qinhanmin2014 I think that somebody should take over to get the imputer fixed asap

@glemaitre Agree, marking as help wanted. Not sure whether there's conflict with MICE. Maybe we should get MICE in first and then take care of this?

glemaitre · 2018-03-16T13:48:55Z

Either way. But MICE is ready to go on my side and the one of @jnothman as well I think. I might have some time in this week to take over this issue in fact.

…

On 16 March 2018 at 14:43, Hanmin Qin ***@***.***> wrote: @gilbertoolimpio <https://github.com/gilbertoolimpio> Thanks for your great work so far :) @qinhanmin2014 <https://github.com/qinhanmin2014> I think that somebody should take over to get the imputer fixed asap @glemaitre <https://github.com/glemaitre> Agree, marking as help wanted. Not sure whether there's conflict with MICE. Maybe we should get MICE in first and then take care of this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10652 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHG9P9f2hYQXrMolxTXPRBF9xAdWr4qGks5te8FqgaJpZM4SJaR4> .

-- Guillaume Lemaitre INRIA Saclay - Parietal team Center for Data Science Paris-Saclay https://glemaitre.github.io/

qinhanmin2014 · 2018-03-16T13:54:05Z

@glemaitre Thanks. Please remove the labels if you've decided to take it yourself :)
I saw your ping in MICE so if it's not merged after a couple of days, I'll try to have a look at the algorithm and give my review there.

giba0 added 8 commits February 14, 2018 22:07

remove axis parameter in impute.rst

5c27cc6

remove de axis parameter in method SimpleImputer

a38f87c

information about the future behavior for axis

14f2f6d

information about the depracation

d5fce03

Complement on axis removal

a3155c7

Remove axis parameter in impute.py file

d99c2fb

Remove axis parameter in test_impute.py file

1ad6659

Merge branch 'master' of https://github.com/scikit-learn/scikit-learn …

3b70cbe

…into Remove-SimpleImputer-axis-parameter-scikit-learn#10636

jnothman reviewed Feb 17, 2018

View reviewed changes

giba0 added 11 commits February 18, 2018 16:59

Revert "information about the future behavior for axis"

bde0f2e

This reverts commit 14f2f6d.

Revert "Complement on axis removal"

f69e8a4

This reverts commit a3155c7.

Revert changens about copy parameter

d53a642

Revert changes in notes.

d3efc4d

revert changes in fit method

1f011be

revert axis=0 in X.sum

a20a207

revert axis=0 in mask_non_zeros

2461301

include axis=0 in np.ma.mean

64ea05c

include axis=0 in na.ma.media method

eb2de3f

include axis=0 in np.sum method

24aaf01

removed tests with axis=1

5720c4d

jnothman reviewed Feb 18, 2018

View reviewed changes

correction in axis atribute exception in line 138

2373654

jnothman approved these changes Feb 19, 2018

View reviewed changes

jnothman changed the title ~~Fix Remove SimpleImputer axis parameter (#10636)~~ [MRG+1] Remove SimpleImputer axis parameter (#10636) Feb 19, 2018

jnothman reviewed Feb 20, 2018

View reviewed changes

remove the self.axis reference

6788a14

remove axis reference

27c795e

correction in X.Shap in imputation.py and impute.py

0f8b420

giba0 added 3 commits February 24, 2018 16:51

new correction in X.shape

d058e66

new corrections

3707fb2

corrections

02183d6

correction in impute and imputation

95da135

glemaitre reviewed Feb 25, 2018

View reviewed changes

qinhanmin2014 reviewed Feb 25, 2018

View reviewed changes

improve the modification

1c46c69

jnothman reviewed Feb 25, 2018

View reviewed changes

qinhanmin2014 added Easy Well-defined and straightforward way to resolve Stalled help wanted labels Mar 16, 2018

jnothman added the Blocker label Mar 17, 2018

glemaitre mentioned this pull request Mar 18, 2018

[MRG+1] Remove axis parameter from SimpleImputer #10829

Merged

lesteve closed this in #10829 Mar 30, 2018

Uh oh!

[MRG] Remove SimpleImputer axis parameter (#10636) #10652

[MRG] Remove SimpleImputer axis parameter (#10636) #10652

Uh oh!

Conversation

giba0 commented Feb 17, 2018

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Any other comments?

Uh oh!

sklearn-lgtm commented Feb 17, 2018

Uh oh!

jnothman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

giba0 commented Feb 18, 2018

Uh oh!

sklearn-lgtm commented Feb 18, 2018

Uh oh!

jnothman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

giba0 commented Feb 19, 2018

Uh oh!

sklearn-lgtm commented Feb 19, 2018

Uh oh!

jnothman commented Feb 19, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jnothman commented Feb 20, 2018

Uh oh!

jnothman commented Feb 20, 2018

Uh oh!

giba0 commented Feb 24, 2018

Uh oh!

giba0 commented Feb 24, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

glemaitre commented Feb 24, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

giba0 Feb 25, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qinhanmin2014 commented Feb 25, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jnothman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

giba0 commented Feb 24, 2018 •

edited

Loading

giba0 Feb 25, 2018 •

edited

Loading