[MRG + 1] Warn on 1D arrays, addresses #4511 #5152

vighneshbirodkar · 2015-08-24T23:55:51Z

Just to reiterate the discussion I had with @amueller in person

I have put a check in check_array and issued a warning. I will proceed to fix the tests now.

jnothman · 2015-08-25T01:13:36Z

sklearn/utils/validation.py

@@ -374,6 +378,9 @@ def check_array(array, accept_sparse=None, dtype="numeric", order=None,
        if force_all_finite:
            _assert_all_finite(array)

+    if array.ndim == 1:


Surely this is appropriate iff ensure_2d is set?

The check is redundant if ensure_2d is set.

Oh. So what's wrong with 1d arrays if ensure_2d=False?

I am a little confused myself now. Here is what I understand

Warn against 1D arrays now and raise an error in later versions.

If ensure_2d is set, nothing to be done now. In later versions ensure_2d parameter is deprecated ?

The problem is that the current ensure_2d does the wrong thing. I think we should change the behavior of ensure_2d from reshaping to warning / erroring.

So should I go ahead with ensure_2d then ? It will warn now and throw an error in later versions

Yes. this is only used internally and we can rename it later.

If I remove the np.atleast_2d call I get 227 errors in the tests. It's my understanding that this PR should fix all of them ?

They are probably many errors caused by the same underlying issues. But yes, indeed.

You should be able to just set the SingleDimensionWarning to "error" though (maybe at the top of the file if it is just for testing).

Many of these will have fixes in #4511.

amueller · 2015-08-25T17:31:11Z

Mentioning #4511 for visibility (mentioning in the title will not show up there).

vighneshbirodkar · 2015-08-26T04:50:15Z

@amueller Down to ~130 errors. I hope these little fixes are correct.

amueller · 2015-08-27T03:10:28Z

sklearn/cluster/birch.py

    dist = euclidean_distances(
-        node.centroids_, Y_norm_squared=node.squared_norm_, squared=True)
+        node.centroids_, Y_norm_squared=squared_norm, squared=True)


this seems odd. why does this need to be 2d? Y_norm_squared should be 1d.

Many things were internally converted to 2D using the np.atleast2d call, without it, many tests are causing errors.

but then this should probably be fixed in euclidean distances. the Y_norm_squared should be ok to be 1d.

Okay, I have made a note of it.

Can euclidian_distances check for 1d and do a conversion to 2d ?

yes, you can do that.

vighneshbirodkar · 2015-08-29T04:03:19Z

@amueller The tests are passing now. There are 9 places where the new warning thrown, I will fix that soon.

However, you also mentioned creating a common test for every estimator and ensure that a 1D input throws the new warning. But as far as I know, due to the 1D array, something most likely will fail later on.

vighneshbirodkar · 2015-08-30T00:34:03Z

sklearn/utils/__init__.py

@@ -351,7 +351,7 @@ def safe_sqr(X, copy=True):
    -------
    X ** 2 : element wise square
    """
-    X = check_array(X, accept_sparse=['csr', 'csc', 'coo'])
+    X = check_array(X, accept_sparse=['csr', 'csc', 'coo'], ensure_2d=False)


This is mainly checking for sparse formats and finiteness. Since the _ensure_sparse_format is an internal function, I have kept the check_array call as it is.

That's fine.

vighneshbirodkar · 2015-08-30T05:13:01Z

@amueller
For the common tests with 1d input, I am leaning towards making an assert_warns call and then catching any exceptions that might occur. Thoughts ?

vighneshbirodkar · 2015-08-31T15:09:17Z

sklearn/utils/validation.py

        if ensure_2d:
-            array = np.atleast_2d(array)
+            if array.ndim == 1:
+                warnings.warn("Array needs atleast 2 dimnesions",


typo on the warning message

This needs a better message, like "Passing 1d arrays as data is deprecated and will be removed in 0.18. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample."

amueller · 2015-08-31T15:58:05Z

sklearn/neighbors/approximate.py

@@ -435,7 +435,12 @@ def kneighbors(self, X, n_neighbors=None, return_distance=True):
        neighbors, distances = [], []
        bin_queries, max_depth = self._query(X)
        for i in range(X.shape[0]):
-            neighs, dists = self._get_candidates(X[i], max_depth[i],
+
+            query = X[i]


wouldn't X[[i]] be easier?

amueller · 2015-08-31T19:19:01Z

ping @jnothman and @GaelVaroquaux who might be interested.
One question: do we also want to deprecate 1d input in the scalers? It was explicitly allowed there up till now.

amueller · 2015-08-31T19:19:09Z

Also maybe ping @ogrisel

giorgiop · 2015-09-02T06:46:42Z

@amueller I was working on this on #5104 for Scalers. So I think the answer is yes.
I have used a 2d-input validator so far (third point in the to do list), but I will remove it to be consistent with the DeprecationWarning

ogrisel · 2015-09-02T10:52:24Z

sklearn/utils/extmath.py

@@ -599,8 +599,10 @@ def log_logistic(X, out=None):
    http://fa.bianp.net/blog/2013/numerical-optimizers-for-logistic-regression/
    """
    is_1d = X.ndim == 1
+    X = np.atleast_2d(X)
    X = check_array(X, dtype=np.float)


Not related to this PR but this should be changed to dtype=np.float64 to be explicitly not platform dependent.

amueller · 2015-09-04T17:02:40Z

sklearn/utils/tests/test_validation.py

@@ -119,6 +121,8 @@ def test_check_array():
    X_csr = sp.csr_matrix(X)
    assert_raises(TypeError, check_array, X_csr)
    # ensure_2d
+    # This might not be needed
+    assert_warns(DeprecationWarning, check_array, [0, 1, 2])


no, this is good. leave it.

Passing 1D arrays to check_array, without setting `ensure_2d` to false now raises a deprecation warning before reshaping it. This will later throw an error. All Scaler classes also throw warnings when 1D arrays are passed. All unit tests/doctests are modified to ensure that no 1D arrays are passed, except in explicit 1D array tests where the warnings have been silenced. Additional tests are also included which check for different 1D array cases. 2D array tests with one samples and one features are also added and where they failed, `check_array` call has been modified to give a more useful error message

amueller · 2015-09-04T19:18:38Z

doc/tutorial/basic/tutorial.rst

@@ -185,7 +185,7 @@ Now you can predict new values, in particular, we can ask to the
 classifier what is the digit of our last image in the ``digits`` dataset,
 which we have not used to train the classifier::

-  >>> clf.predict(digits.data[-1])
+  >>> clf.predict([digits.data[-1]])


should we be consistent and also use [-1:] here?

amueller · 2015-09-04T19:28:00Z

LGTM apart from minor nitpicks

amueller · 2015-09-08T20:23:29Z

@ogrisel wdyt?

ogrisel · 2015-09-09T13:39:53Z

I will squash the commit, merge by rebase and document the change in whats_new.rst.

ogrisel · 2015-09-09T13:51:31Z

Merged as 2f09933. Thanks @vighneshbirodkar!

amueller · 2015-09-09T14:41:26Z

Thanks @vighneshbirodkar, this is really helpful. Thanks for the review @ogrisel.

jnothman reviewed Aug 25, 2015
View reviewed changes

amueller reviewed Aug 27, 2015
View reviewed changes

vighneshbirodkar reviewed Aug 30, 2015
View reviewed changes

vighneshbirodkar reviewed Aug 31, 2015
View reviewed changes

vighneshbirodkar force-pushed the array_1d_fix branch from f52f1a7 to 492e364 Compare August 31, 2015 15:22

amueller reviewed Aug 31, 2015
View reviewed changes

amueller added this to the 0.17 milestone Aug 31, 2015

giorgiop mentioned this pull request Sep 2, 2015

[MRG+2] partial_fit for StandardScaler, MinMaxScaler and MaxAbsScaler #5104

Merged

12 tasks

ogrisel reviewed Sep 2, 2015
View reviewed changes

amueller reviewed Sep 4, 2015
View reviewed changes

vighneshbirodkar force-pushed the array_1d_fix branch from cda640d to f760d27 Compare September 4, 2015 17:55

Improved array indexing for readability and other fixes

81636ee

vighneshbirodkar changed the title ~~Warn on 1D arrays, addresses #4511 [WIP]~~ Warn on 1D arrays, addresses #4511 [MRG] Sep 4, 2015

amueller reviewed Sep 4, 2015
View reviewed changes

amueller changed the title ~~Warn on 1D arrays, addresses #4511 [MRG]~~ [MRG + 1] Warn on 1D arrays, addresses #4511 [MRG] Sep 4, 2015

amueller changed the title ~~[MRG + 1] Warn on 1D arrays, addresses #4511 [MRG]~~ [MRG + 1] Warn on 1D arrays, addresses #4511 Sep 4, 2015

changed some array addressing syntax and added comment

e064d7c

ogrisel closed this Sep 9, 2015

vighneshbirodkar mentioned this pull request Sep 15, 2015

[MRG + 1]Deprecating 1D inputs in fast_mcd #5234

Merged

MechCoder mentioned this pull request Dec 1, 2015

Letting OneHotEncoder encode y? #5930

Closed

raghavrv mentioned this pull request Feb 18, 2016

k-means over 1d data returns wrong error message #6381

Closed

Uh oh!

[MRG + 1] Warn on 1D arrays, addresses #4511 #5152

[MRG + 1] Warn on 1D arrays, addresses #4511 #5152

Uh oh!

Conversation

vighneshbirodkar commented Aug 24, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amueller commented Aug 25, 2015

Uh oh!

vighneshbirodkar commented Aug 26, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vighneshbirodkar commented Aug 29, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vighneshbirodkar commented Aug 30, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amueller commented Aug 31, 2015

Uh oh!

amueller commented Aug 31, 2015

Uh oh!

giorgiop commented Sep 2, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amueller commented Sep 4, 2015

Uh oh!

amueller commented Sep 8, 2015

Uh oh!

ogrisel commented Sep 9, 2015

Uh oh!

ogrisel commented Sep 9, 2015

Uh oh!

amueller commented Sep 9, 2015