ENH: Catch masked array and invalid x, y to pcolormesh #9723

jklymak · 2017-11-08T21:36:05Z

PR Summary

Docs clearly say that you can't pass masked arrays to pcolor and pcolormesh, yet we were inadvertently allowing it. This catches that error at the arg-check stage.

It also catches invalid numbers, though note that it won't catch invalid objects in ndarrays (i.e. if someone somehow puts a NaN in a datetime array).

If someone does pass a masked array to x or y, but it does not have any masked values, it will still work. I don't think this needs to be documented.

Addresses #9720

PR Checklist

Has Pytest style unit tests
Code is PEP 8 compliant

efiring · 2017-11-08T22:09:24Z

Unlike pcolormesh, pcolor has documented support for masked X and/or Y; pcolor is implemented in a completely different way than pcolormesh, which makes this possible, though it may still be a bit silly.

Kojoley

Is not this check implementation in very costly?

efiring

I think this additional argument checking is reasonable, but I'm not 100% convinced yet.

efiring · 2017-11-11T19:21:15Z

lib/matplotlib/axes/_axes.py

+            for a in X, Y:
+                a = cbook.safe_masked_invalid(a)
+                if funcname == 'pcolormesh' and np.ma.is_masked(a):
+                    raise TypeError(


I would use ValueError rather than TypeError; the problem is more a matter of the information in the container than the precise type of the container.

efiring · 2017-11-11T19:40:58Z

lib/matplotlib/axes/_axes.py

+            if funcname == 'pcolormesh':
+                # if it didn't fail the above it is masked but has no
+                # masked values, so strip the mask for pcolormesh
+                if type(X) is np.ma.core.MaskedArray:


I think all this can be simplified:

C = np.asanyarray(C) X, Y = [cbook.safe_masked_invalid(x) for x in (X, Y)] if funcname == 'pcolormesh': if np.ma.is_masked(X) or np.ma.is_masked(Y): raise ValueError("...") X, Y = X.data, Y.data

efiring · 2017-11-11T19:43:27Z

lib/matplotlib/tests/test_axes.py

@@ -1190,6 +1190,13 @@ def test_pcolorargs():
        ax.pcolormesh(x, y, Z[:-1, :-1], shading="gouraud")
    with pytest.raises(TypeError):
        ax.pcolormesh(X, Y, Z[:-1, :-1], shading="gouraud")
+    with pytest.raises(TypeError):
+        xx = x


Why are you renaming x? Do you mean to use xx = x[:]?

Umm well my understanding of scope and context management is rudimentary. If I change x in this with-block does it not change it for the next with-block?

If I change x in this with-block does it not change it for the next with-block?

It does. But it look suspicious, because xx = x makes xx alias of x, so it is a noop line.

Oh xx = x.copy()?

Actually, I don't need to rename this at all...

efiring · 2017-11-11T19:54:55Z

@Kojoley, the cost is in the checking and masking for nans:

In [1]: import numpy as np
In [2]: from matplotlib import cbook
In [4]: xx = np.random.randn(500, 1000)
In [5]: xx[xx < 0] = np.nan
In [6]: timeit cbook.safe_masked_invalid(xx)
1.36 ms ± 39.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

That means it is adding about 3 ms for handling the X and Y arrays in this case on a 5-year-old Macbook Pro. I think that is negligible--not that I like doing things that make mpl even a little slower.

Kojoley · 2017-11-11T21:19:37Z

lib/matplotlib/tests/test_axes.py

+        xx[0] = np.NaN
+        ax.pcolormesh(xx, y, Z[:-1, :-1])
+    with pytest.raises(TypeError):
+        xx = np.ma.masked(x, mask=(x < 0))


Do this out of a context manager

jklymak · 2017-11-11T22:11:39Z

Thanks @efiring and @Kojoley I think I caught most of your comments.

efiring

Getting close...

efiring · 2017-11-11T22:49:28Z

lib/matplotlib/axes/_axes.py

+            X, Y = [cbook.safe_masked_invalid(a) for a in args[:2]]
+            if funcname == 'pcolormesh':
+                if np.ma.is_masked(X) or np.ma.is_masked(Y):
+                    raise TypeError(


Or ValueError?

Oops, sorry, I changed it, and then accidentally changed it back

efiring · 2017-11-11T22:52:32Z

lib/matplotlib/axes/_axes.py

+                if type(X) is np.ma.core.MaskedArray:
+                    X = X.data
+                if type(Y) is np.ma.core.MaskedArray:
+                    Y = Y.data


The 4 lines above can be one line:

X, Y = X.data, Y.data

Not if they are ndarrays?

I was incorrectly assuming that safe_masked_invalid always returned a masked array--and possibly it should do so for consistency, but that is another matter. Given that it doesn't, you are stuck with the checks, but please use if isinstance(X, np.ma.core.MaskedArray): which is simpler and more idiomatic.
It might be worth replacing your comment above these blocks with:

# safe_masked_invalid() returns an ndarray for dtypes other than floating point

efiring · 2017-11-11T22:57:50Z

lib/matplotlib/tests/test_axes.py

@@ -1190,6 +1190,12 @@ def test_pcolorargs():
        ax.pcolormesh(x, y, Z[:-1, :-1], shading="gouraud")
    with pytest.raises(TypeError):
        ax.pcolormesh(X, Y, Z[:-1, :-1], shading="gouraud")
+    x[0] = np.NaN
+    with pytest.raises(TypeError):


I see that TypeError is used in these earlier tests, so maybe you are using it for consistency. I think that in both cases ValueError would be more consistent with general mpl practice, and the way I think about such things. This is probably something where we need to state guidelines. @tacaswell, your thoughts? (I think that a long time ago I proposed we should define our own exception subclass of ValueError for use in argument validation, something like MPL_ArgumentError. Or maybe that idea never got further than my head.)

Change to just apply to pcolormesh, not pcolor

…mesh

Backport PR #9723 on branch v2.1.x

jklymak changed the title ~~ENH: Catch masked array and invalid x, y to pcolor/mesh~~ WIP: (hoops) ENH: Catch masked array and invalid x, y to pcolor/mesh Nov 8, 2017

jklymak changed the title ~~WIP: (hoops) ENH: Catch masked array and invalid x, y to pcolor/mesh~~ WIP: (oops) ENH: Catch masked array and invalid x, y to pcolor/mesh Nov 8, 2017

jklymak force-pushed the fixpcolorcopy branch from 223e9a9 to 58eb8c2 Compare November 8, 2017 21:47

jklymak changed the title ~~WIP: (oops) ENH: Catch masked array and invalid x, y to pcolor/mesh~~ ENH: Catch masked array and invalid x, y to pcolor/mesh Nov 8, 2017

jklymak added the API: consistency label Nov 8, 2017

jklymak added this to the v2.1.1 milestone Nov 8, 2017

jklymak force-pushed the fixpcolorcopy branch 2 times, most recently from 88e76e8 to b6f8923 Compare November 10, 2017 22:27

Kojoley reviewed Nov 11, 2017

View reviewed changes

efiring requested changes Nov 11, 2017

View reviewed changes

Kojoley reviewed Nov 11, 2017

View reviewed changes

jklymak force-pushed the fixpcolorcopy branch 2 times, most recently from 0d4d3d5 to 9e1ba33 Compare November 11, 2017 22:10

efiring requested changes Nov 11, 2017

View reviewed changes

jklymak force-pushed the fixpcolorcopy branch from 9e1ba33 to f308e0e Compare November 11, 2017 23:02

ENH: Catch masked array and invalid x, y to pcolor/mesh

44d455a

Change to just apply to pcolormesh, not pcolor

jklymak force-pushed the fixpcolorcopy branch from f308e0e to 44d455a Compare November 12, 2017 00:14

efiring approved these changes Nov 13, 2017

View reviewed changes

efiring changed the title ~~ENH: Catch masked array and invalid x, y to pcolor/mesh~~ ENH: Catch masked array and invalid x, y to pcolormesh Nov 13, 2017

tacaswell merged commit 1df45a4 into matplotlib:master Nov 13, 2017

lumberbot-app bot pushed a commit that referenced this pull request Nov 13, 2017

Backport PR #9723: ENH: Catch masked array and invalid x, y to pcolor…

b3f0c28

…mesh

lumberbot-app bot mentioned this pull request Nov 13, 2017

Backport PR #9723 on branch v2.1.x #9769

Merged

tacaswell mentioned this pull request Nov 13, 2017

plt.pcolormesh stopped working with Masked Arrays #9720

Closed

dstansby added a commit that referenced this pull request Nov 14, 2017

Merge pull request #9769 from matplotlib/auto-backport-of-pr-9723

04bc071

Backport PR #9723 on branch v2.1.x

ThorvaldLarsen mentioned this pull request May 20, 2018

Png save fails for sweep with many measurements (~8 or more) qdev-dk/qdev-wrappers#88

Closed

jenshnielsen mentioned this pull request Jun 4, 2018

Add support for Matplotlib 2.2 microsoft/Qcodes#1119

Merged

asreimer mentioned this pull request Feb 12, 2019

[BUG]: Matplotlib change, pcolormesh can't handle nonfinite values in later versions asreimer/visuamisr#12

Closed

timhoffm mentioned this pull request Oct 31, 2020

pcolormesh has been broken #18853

Closed

Uh oh!

ENH: Catch masked array and invalid x, y to pcolormesh #9723

ENH: Catch masked array and invalid x, y to pcolormesh #9723

Uh oh!

Conversation

jklymak commented Nov 8, 2017

PR Summary

PR Checklist

Uh oh!

efiring commented Nov 8, 2017

Uh oh!

Kojoley left a comment

Choose a reason for hiding this comment

Uh oh!

efiring left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efiring commented Nov 11, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jklymak commented Nov 11, 2017

Uh oh!

efiring left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efiring Nov 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

efiring Nov 11, 2017 •

edited

Loading