[WIP] add matrix checking function for quiver input #7461

JunTan · 2016-11-15T05:42:20Z

refer to the problem mentioned in #1558. Add checking condition in quiver.py to ensure that any input except matrix are token.

QuLogic · 2016-11-15T06:15:47Z

I don't understand why isinstance can't just be used directly, so I don't know why the function is really necessary.

anntzer · 2016-11-15T06:17:37Z

lib/matplotlib/cbook.py

@@ -1734,7 +1735,7 @@ def recursive_remove(path):
                os.removedirs(fname)
            else:
                os.remove(fname)
-        #os.removedirs(path)
+        # os.removedirs(path)


If you really want to touch this function, I'd just alias recursive_remove to shutil.rmtree (with onerror set to remove files) :-) Otherwise this is kind of pointless.

I just changed the comment part according to the pep8 style. Nothing is changed within the function.

Yes, I don't think that's really in the scope of this PR.

JunTan · 2016-11-15T06:46:05Z

The purpose of this function is to make the error more clear and force users to use other inputs except matrix. Otherwise the error raised is not clear and points to other places.

QuLogic · 2016-11-15T07:28:35Z

I don't understand how the function makes the error clearer. How is raising from a different function clearer than raising from the method that the user actually called?

JunTan · 2016-11-15T07:51:36Z

If you don't raise this error or don't use this helper function, try this
quiver(np.matrix(np.ones((32, 32))), np.matrix(np.ones((32, 32))))
the output is

Traceback (most recent call last):
  File "/Users/juntan/matplotlib/lib/matplotlib/backends/backend_qt5agg.py", line 183, in __draw_idle_agg
    FigureCanvasAgg.draw(self)
  File "/Users/juntan/matplotlib/lib/matplotlib/backends/backend_agg.py", line 464, in draw
    self.figure.draw(self.renderer)
  File "/Users/juntan/matplotlib/lib/matplotlib/artist.py", line 68, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)
  File "/Users/juntan/matplotlib/lib/matplotlib/figure.py", line 1268, in draw
    renderer, self, dsu, self.suppressComposite)
  File "/Users/juntan/matplotlib/lib/matplotlib/image.py", line 139, in _draw_list_compositing_images
    a.draw(renderer)
  File "/Users/juntan/matplotlib/lib/matplotlib/artist.py", line 68, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)
  File "/Users/juntan/matplotlib/lib/matplotlib/axes/_base.py", line 2383, in draw
    mimage._draw_list_compositing_images(renderer, self, dsu)
  File "/Users/juntan/matplotlib/lib/matplotlib/image.py", line 139, in _draw_list_compositing_images
    a.draw(renderer)
  File "/Users/juntan/matplotlib/lib/matplotlib/artist.py", line 68, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)
  File "/Users/juntan/matplotlib/lib/matplotlib/quiver.py", line 526, in draw
    self._init()
  File "/Users/juntan/matplotlib/lib/matplotlib/quiver.py", line 511, in _init
    self._make_verts(self.U, self.V)
  File "/Users/juntan/matplotlib/lib/matplotlib/quiver.py", line 647, in _make_verts
    X, Y = self._h_arrows(length)
  File "/Users/juntan/matplotlib/lib/matplotlib/quiver.py", line 676, in _h_arrows
    length = length.reshape(N, 1)
ValueError: total size of new array must be unchanged

QuLogic · 2016-11-15T07:59:56Z

Yes, but I'm not asking why the new exception is necessary, but why the extra function is necessary. Also, why the cast with np.asanyarray?

Kojoley · 2016-11-15T19:43:24Z

lib/matplotlib/cbook.py

@@ -2701,3 +2702,15 @@ def __exit__(self, exc_type, exc_value, traceback):
                    os.rmdir(path)
                except OSError:
                    pass
+
+
+def is_matrix(obj):


In my point of view any is_* function must return a boolean value.

I'd indeed rename this check_array

NelleV · 2016-11-15T23:33:58Z

lib/matplotlib/cbook.py

@@ -2701,3 +2702,15 @@ def __exit__(self, exc_type, exc_value, traceback):
                    os.rmdir(path)
                except OSError:
                    pass
+
+
+def is_matrix(obj):


I'd indeed rename this check_array

NelleV · 2016-11-15T23:35:12Z

lib/matplotlib/cbook.py

+
+def is_matrix(obj):
+    '''
+    This is a test for whether the input is a matrix.


Can you please use triple double quotes """: this is the convention for docstring, not triple single quotes.

The docstring is also very unclear on what this function does.

NelleV · 2016-11-15T23:35:59Z

lib/matplotlib/cbook.py

+    cast_result = np.asanyarray(obj)
+    if isinstance(cast_result, np.matrix):
+        raise ValueError("The input cannot be matrix")
+    return obj


You'd want to return the cast objects. Ie, if X is a list, you want the returned object to be an array.

Then should I raise the error in this function or just return false and raise the error inside the function who calls check_array?

NelleV · 2016-11-15T23:37:16Z

Once this is properly done for quiver, this helper function can be used in other places as well.

efiring · 2016-11-16T21:29:19Z

It looks to me like this needs some more thought and discussion at the design stage. For example, here is an (untested) alternative:

def _fail_with_matrix(arglist):
    for name, arr in arglist:
        if isinstance(arr, np.matrix):
            raise ValueError("Input argument %s is a numpy matrix subclass instance, which is not supported." % name)

# example of  usage:
_fail_with_matrix([('X', X), ('Y', Y), ('Z', Z)])

Advantages: a single line handles multiple arguments, and the error message names
the offending argument. Also, in agreement with @QuLogic, this does only the minimal checking; type conversion is left out because it probably needs to be done elsewhere anyway, in a way that may vary from argument to argument and function to function.

In the quiver case, this checking probably should occur inside _parse_args.

NelleV · 2016-11-16T22:03:55Z

I am obviously biased from my experience on sklearn, but I do think that validation and transformation into ndarray could (and should) be done in the same function. On sklearn, we have a check_array function that does some input validation and converts to an ndarray of float (https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/utils/validation.py). This particular code is used everywhere in sklearn and is very useful. It enforces a very consistent API across the project.

I also think that as long as the function we add is private, it will be very easy to replace and extend when needed.

efiring · 2016-11-16T22:27:02Z

Modifying my example to apply asanyarray to each argument and return the tuple of results is a reasonable option.

dopplershift · 2016-11-16T22:28:14Z

It would be nice to standardize our input data handling across our numerous functions, but that's out of scope here.

efiring · 2016-11-16T23:16:55Z

@NelleV, side comment: yes, we can do more to consolidate and standardize argument handling, and we already do quite a bit of it now. But we have much more need for flexibility than sklearn, and blanket conversion to float ndarrays is not appropriate for us.

NelleV · 2016-11-22T19:38:26Z

Hi @efiring
I really don't like the API of the function you suggest. It makes the function much harder to use, for little benefit. In 90% of the cases in matplotlib, we'll only have to check that 1 or 2 elements are not matrices.
It is possible that this function becomes more complex overtime to check for other elements, so I would keep the API simple for now and move the loop outside of the call.

JunTan · 2016-11-22T20:15:55Z

I put the check_array to _parse_args and refactor the _parse_args, does this look nicer?

def _parse_args(*args):
    X, Y, U, V, C = [None] * 5
    args = list(args)

    if len(args) == 2 or len(args) == 4:
        V = np.atleast_1d(args.pop(-1))
        U = np.atleast_1d(args.pop(-1))
    elif len(args) == 3 or len(args) == 5:
        C = np.atleast_1d(args.pop(-1))
        V = np.atleast_1d(args.pop(-1))
        U = np.atleast_1d(args.pop(-1))
    if U.ndim == 1:
        nr, nc = 1, U.shape[0]
    else:
        nr, nc = U.shape
    if len(args) == 2:  # remaining after removing U,V,C
        X, Y = [np.array(a).ravel() for a in args]
        if len(X) == nc and len(Y) == nr:
            X, Y = [a.ravel() for a in np.meshgrid(X, Y)]
    else:
        indexgrid = np.meshgrid(np.arange(nc), np.arange(nr))
        X, Y = [np.ravel(a) for a in indexgrid]
    cbook._check_array(X)
    cbook._check_array(Y)
    cbook._check_array(U)
    cbook._check_array(V)
    cbook._check_array(C)
    return X, Y, U, V, C

efiring · 2016-11-23T19:43:57Z

@NelleV, maybe we can discuss our disagreement some time. Part of my point is that if you really want to make the API user-friendly, it is good to tell the user which argument is failing a test.
@JunTan, as far as I can see, there is no need to check X and Y because every possible code path in _parse_args is already ensuring that X and Y are not matrices.

efiring · 2016-11-29T07:40:32Z

lib/matplotlib/quiver.py

+    if len(args) == 2 or len(args) == 4:
+        V = np.atleast_1d(args.pop(-1))
+        U = np.atleast_1d(args.pop(-1))
+    elif len(args) == 3 or len(args) == 5:


You are increasing the number of lines of code here, and I don't see any benefit to it. If you left it the way it was, all you would have to do is insert your _check_array into each of the lines extracting U, V, and C, like this:

U = np.atleast_1d(cbook._check_array(args.pop(-1)))

Better yet, you could modify your checking function to use np.atleast_1d instead of np.asanyarray, because the former calls the latter internally. Then in the original you would just replace np.atleast1d with cbook._check_array in each of the 3 lines. Much nicer.

I agree that doing the _check_array on U, V, C here (where they are defined) is more suitable than leaving them at the end. But I don't think stuffing the _check_array and atleast_1d together is a good idea, at least, seem less readable to me.

@trpham, I disagree. We need to encapsulate logical chunks of argument validation that typically go together so as to minimize code duplication. The consolidation I am suggesting here is really minimal--it is ensuring that a given argument is some sort of ndarray but not a matrix. That's consistent with the name _check_array (or it could be _ensure_array, etc.) With regard to matrix checking: all of our array-like argument validation should be explicitly blocking (or converting) matrix types because they are known to have odd behavior. This needs to be embedded in the more general validation and "argument scrubbing" functions. We have never claimed to support the matrix subclass, and have always known we didn't want to try to do so--but in the evolution of the library, basic functionality has come first, and ever more stringent validation and scrubbing is gradually being added.

anntzer · 2019-01-14T20:36:11Z

Superseded by #13089. Thanks for the PR!

add matrix checking function for quiver input

d8841a8

anntzer reviewed Nov 15, 2016

View reviewed changes

remove the if statement and use isinstance function

bba1ea6

Kojoley reviewed Nov 15, 2016

View reviewed changes

NelleV requested changes Nov 15, 2016

View reviewed changes

NelleV added the Machine Shop label Nov 15, 2016

NelleV changed the title ~~add matrix checking function for quiver input~~ [WIP] add matrix checking function for quiver input Nov 15, 2016

change doc string and return type in check_matrix

4bdcac3

move checking into _parse_args

22655b6

refactor _parse_ags

73a69a2

efiring requested changes Nov 29, 2016

View reviewed changes

tacaswell added this to the 2.0.1 (next bug fix release) milestone Nov 29, 2016

NelleV removed the Machine Shop label Dec 19, 2016

dstansby added the status: needs revision label Feb 8, 2017

QuLogic modified the milestones: 2.0.1 (next bug fix release), 2.0.2 (next bug fix release) May 3, 2017

QuLogic modified the milestones: 2.0.2 (next bug fix release), 2.0.1 (next bug fix release) May 3, 2017

tacaswell assigned efiring Jul 11, 2017

tacaswell modified the milestones: 2.2 (next next feature release), 2.0.3 (next bug fix release) Jul 11, 2017

anntzer mentioned this pull request Jan 3, 2019

Ensure that arguments to quiver() are not matrices. #13089

Merged

6 tasks

anntzer closed this Jan 14, 2019

QuLogic modified the milestones: needs sorting, v3.1 Jan 15, 2019

QuLogic mentioned this pull request May 14, 2019

Check number of positional arguments passed to quiver() #14084

Merged

3 tasks

Uh oh!

[WIP] add matrix checking function for quiver input #7461

[WIP] add matrix checking function for quiver input #7461

Uh oh!

Conversation

JunTan commented Nov 15, 2016

Uh oh!

QuLogic commented Nov 15, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JunTan commented Nov 15, 2016

Uh oh!

QuLogic commented Nov 15, 2016

Uh oh!

JunTan commented Nov 15, 2016

Uh oh!

QuLogic commented Nov 15, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NelleV commented Nov 15, 2016

Uh oh!

efiring commented Nov 16, 2016

Uh oh!

NelleV commented Nov 16, 2016

Uh oh!

efiring commented Nov 16, 2016

Uh oh!

dopplershift commented Nov 16, 2016

Uh oh!

efiring commented Nov 16, 2016

Uh oh!

NelleV commented Nov 22, 2016

Uh oh!

JunTan commented Nov 22, 2016

Uh oh!

efiring commented Nov 23, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anntzer commented Jan 14, 2019

Uh oh!

Uh oh!