BUG: Fix argsort vs sort in Masked arrays #8678

eric-wieser · 2017-02-23T15:38:11Z

This fixes #8664, at the cost of a behavioural change in argsort -- masked values are now sorted to the end by default, whereas before they could end up being placed somewhere arbitrary in the middle. This brings the behaviour of argsort inline with sort.

In particular, argsort would use the default fill value, which for int8 types would be 999999 % 256 = 63. This sorted very misleadingly.

sort made the more sensible choice of using the maximum/minimum possible value.

This also fixes code duplication between the method and function forms of these operations

eric-wieser · 2017-02-23T15:40:50Z

numpy/ma/core.py

-                           indexing='ij')
-        indx[axis] = sindx
-    return a[indx]
+        a.sort(axis=axis, kind=kind, order=order)


There's a choice we need to make here. Right now (and before this patch), np.ma.sort sometimes returns an ma.array, and sometimes an ndarray.

Should we change it to always promote to ma.array, for consistency with the other functions? (Which do this as of #8665 )

eric-wieser · 2017-02-23T15:51:01Z

numpy/ma/core.py

-            self._data.flat = tmp_data
-            self._mask.flat = tmp_mask
-        return
+        self[...] = self[idx]


In theory we could do a little better here, by implementing the in-place sort in terms of the non-in place. Right now (and before this patch), the latter does a redundant copy. So either way, best left for another PR, I think

juliantaylor · 2017-02-23T16:58:33Z

numpy/ma/core.py

@@ -5213,7 +5213,8 @@ def round(self, decimals=0, out=None):
            out.__setmask__(self._mask)
        return out

-    def argsort(self, axis=None, kind='quicksort', order=None, fill_value=None):
+    def argsort(self, axis=-1, kind='quicksort', order=None,
+                endwith=True, fill_value=None):


endwith needs to be after fill_value to not break the api

changing axis default also probably breaks api

Not changing it breaks liskov substitution though. The default values should really match ndarray. Fair call on swapping argument order

it would be nice if they matched but unfortunately that ship has sailed, this default argument is part of our api and we can't change it.

@juliantaylor: Is there a deprecation path for changing it then?

Regarding named argument order, perhaps we should enforce that endwith is a boolean, just to help guard against someone (stupidly) passing unnamed arguments, and assuming they match the order of sort?

I don't see a way we could deprecate the axis argument value besides adding an argsort2 and deprecating the old one.

sort has a different order? that is unfortunate ...
I guess we could try to get away with that change, usage of positional arguments for the ones at the end are probably not very common in the wild.

Also, argsort is documented as having a default argument of -1, even though the actual default is None. So can we update the API to match the documentation?

Re deprecation:

if axis is np._NoValue: if self.ndim > 1: warn("Unlike np.argsort and the documentation for this function, the" "default axis argument is None, not -1. This only matters for 2 or" "higher-dimensional arrays. If this is intended, pass it explicitly to squash this warning") axis = None

eric-wieser · 2017-02-26T14:01:22Z

@juliantaylor: OK, clearly changing the axis default does not belong in this PR. I've created an issue about that at #8701. I think I agree with you that using positional arguments is unlikely in the wild for argsort, especially given how unlikely it is that users pass kind explicitly.

juliantaylor · 2017-02-26T14:34:32Z

numpy/ma/core.py

-                    filler = maximum_fill_value(self)
-            else:
-                filler = fill_value
+            idx = list(np.ix_(*[np.arange(x) for x in self.shape]))


I don't really like these shortcuts very much, the explicit meshgrid is easier to read imo
there isn't really an advantage to using ix_ is there?

Is this really a shortcut though? The two functions are independant, with neither calling the other. Meshgrid doesn't do what we need by default, but ix_ does.

Furthermore, ix_ is used as an indexer in its examples, whereas meshgrid is used to evaluate functions. The former use is what we want here.

mhvk · 2017-03-07T14:45:39Z

@eric-wieser - Overall, this looks good. The main possible issue I see is that in adding the additional endwith argument, you change the order. I think that is fine, since it is unlikely someone would pass in all preceding arguments as positional ones.
A question is whether you wouldn't rather use #8714 (though I guess the order is not so important).
Finally, in reply to your in-line question, yes, I think we should always return a MaskedArray instance, i.e., just to masked_array on the input and call sort.

All this means it is a bit of an API change, so definitely needs an entry in the release notes (and arguably a note to the mailing list).

eric-wieser · 2017-03-07T14:53:04Z

A question is whether you wouldn't rather use #8714

I don't really want a feature needing discussion to hold up a bugfix.

The main possible issue I see is that in adding the additional endwith argument, you change the order.

@juliantaylor seemed to think we could get away with it (hidden comment) as well

yes, I think we should always return a MaskedArray instance

I'll update this accordingly then

so definitely needs an entry in the release notes

Will do

mhvk · 2017-03-07T15:02:48Z

OK, sounds good. I do think this has gone beyond a bug fix (ie., MAINT as well), which I why I set the milestone to 1.13. (I think a bug fix would only touch the default fill value, but do not really think this is worth implementing).

eric-wieser · 2017-03-07T16:11:29Z

yes, I think we should always return a MaskedArray instance

Actually, I'm backtracking on this. While I think that all the np.ma functions should return masked arrays, I think that that change belongs in a PR by itself, along with changes to the other functions.

Release notes added, should be good to go.

mhvk

Only one annoying tidbit, otherwise good to go

doc/release/1.13.0-notes.rst

Previously, these had different rules for unmasking values, and even different arguments to decide how to do so. Fixes numpy#8664

eric-wieser · 2017-03-07T17:35:41Z

Done, I think - just waiting on tests.

eric-wieser · 2017-03-09T20:51:16Z

A question is whether you wouldn't rather use #8714

I've updated #8714 to make that fix

eric-wieser added 03 - Maintenance 00 - Bug component: numpy.ma masked arrays labels Feb 23, 2017

eric-wieser commented Feb 23, 2017

View reviewed changes

eric-wieser force-pushed the fix-argsort-vs-sort branch from f498c2c to 34a02d7 Compare February 23, 2017 15:50

eric-wieser commented Feb 23, 2017

View reviewed changes

juliantaylor reviewed Feb 23, 2017

View reviewed changes

charris changed the title ~~Fix argsort vs sort in Masked arrays~~ BUG: Fix argsort vs sort in Masked arrays Feb 24, 2017

eric-wieser force-pushed the fix-argsort-vs-sort branch from 34a02d7 to b5c4f9a Compare February 26, 2017 13:52

juliantaylor reviewed Feb 26, 2017

View reviewed changes

eric-wieser mentioned this pull request Feb 27, 2017

ENH: Add a function that performs the indexing needed to map argsort to sort #8708

Closed

eric-wieser requested a review from mhvk March 6, 2017 23:27

mhvk added this to the 1.13.0 release milestone Mar 7, 2017

eric-wieser added 2 commits March 7, 2017 16:09

MAINT: Combine ma.sort and MaskedArray.sort

9c865d5

MAINT: Combine ma.argsort and MaskedArray.argsort

f91eb36

eric-wieser force-pushed the fix-argsort-vs-sort branch from 06c40c3 to 37bb166 Compare March 7, 2017 16:10

mhvk approved these changes Mar 7, 2017

View reviewed changes

doc/release/1.13.0-notes.rst Outdated Show resolved Hide resolved

eric-wieser added 4 commits March 7, 2017 17:33

BUG: Make MaskedArray.argsort and MaskedArray.sort consistent

11c5a9f

Previously, these had different rules for unmasking values, and even different arguments to decide how to do so. Fixes numpy#8664

MAINT: Reduce indentation, and remove erroneous return value

8e83849

MAINT: Remove duplicated code between sort and argsort

821293d

MAINT: tidy up sort output stage, avoiding meshgrid and flat

ee90efc

eric-wieser force-pushed the fix-argsort-vs-sort branch from 37bb166 to ee90efc Compare March 7, 2017 17:35

mhvk merged commit 6a3edf3 into numpy:master Mar 7, 2017

MSeifert04 mentioned this pull request Mar 10, 2017

[Don't merge!] fix overwrite=True for ccds generator in ImageFileCollection astropy/ccdproc#454

Closed

4 tasks

eric-wieser mentioned this pull request Apr 6, 2017

API: Change default axis argument for argsort. #8904

Closed

eric-wieser mentioned this pull request Apr 21, 2017

np.ma.argsort uses default fill_value not largest representable value #2890

Closed

rgommers mentioned this pull request May 12, 2021

MAINT: fix unique for masked arrays #18968

Closed

Uh oh!

BUG: Fix argsort vs sort in Masked arrays #8678

BUG: Fix argsort vs sort in Masked arrays #8678

Uh oh!

Conversation

eric-wieser commented Feb 23, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eric-wieser Feb 23, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliantaylor Feb 24, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eric-wieser Feb 24, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eric-wieser commented Feb 26, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eric-wieser Feb 26, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mhvk commented Mar 7, 2017

Uh oh!

eric-wieser commented Mar 7, 2017

Uh oh!

mhvk commented Mar 7, 2017

Uh oh!

eric-wieser commented Mar 7, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mhvk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

eric-wieser commented Mar 7, 2017

Uh oh!

eric-wieser commented Mar 9, 2017

Uh oh!

Uh oh!

eric-wieser commented Feb 23, 2017 •

edited

Loading

eric-wieser Feb 23, 2017 •

edited

Loading

juliantaylor Feb 24, 2017 •

edited

Loading

eric-wieser Feb 24, 2017 •

edited

Loading

eric-wieser Feb 26, 2017 •

edited

Loading

eric-wieser commented Mar 7, 2017 •

edited

Loading