WIP: Fix matplotlib, etc. errors #4804

seberg · 2014-06-11T22:32:44Z

I would like the deprecation warning to show the actual error message. It might also make sense to replace the error again with the original one if the fallback failed as well.

seberg · 2014-06-11T22:39:30Z

While the numpy test obviously fails without change. This should fix all or almost all indexing test failures in matplotlib and the like.

seberg · 2014-06-15T12:32:04Z

Fixed it up to something that should pretty much work. There is one special case here I was not aware of. The boolean index into a 1-d array special case always was strict about shape mismatches (I mean when the nonzero logic works, not wrong number of dimensions). Because my new code does not make a difference, this now is not strict anymore and the erros (if they happen at all) not as nice...

seberg · 2014-06-16T13:51:13Z

Question, considering that for 1-d arrays being indexed with a single boolean index, the shape was always strictly checked, should we at least do that as well in the new code? This requires much less then what is in gh-4353, but maybe less back and forth if we do gh-4353 at some point...

charris · 2014-06-17T03:59:23Z

numpy/core/src/multiarray/mapping.c

+    Py_DECREF(err);
+
+    if (DEPRECATE(
+            "assignment will raise an error in the future, most likely "


The message needs some \n, otherwise it prints as one really long line. I noted that in some of the error messages in the failing tests, so those messages should also be broken into lines

charris · 2014-06-17T04:08:37Z

Looks pretty straight forward. I not clear on whether the a.flat usage is deprecated or not.

seberg · 2014-06-17T07:04:29Z

I didn't ;). I don't think we will deprecate a.flat but in the long run I guess possibly the special case in question here...

charris · 2014-06-30T04:25:30Z

@seberg How does your time look? I confess, some of the code causing problems is doing strange things, and sometimes it looks like there are actual errors in the code. That said, it seems that the 1d vs 2d boolean array indexes are causing most of the remaining problems.

seberg · 2014-06-30T08:46:58Z

Hey, doesn't look to well right now unfortunatly :/. If we only want to fix the 1d boolean case, this could also be done in the boolean special case index parsinig part, by just reshaping the array there.

charris · 2014-06-30T18:18:20Z

Too bad about the time, but everyone needs to make a living :0. Now I suppose I'll need to learn something about the indexing code... Unfortunately, not all projects are as easy to test as NumPy.

charris · 2014-07-04T19:33:03Z

@seberg @juliantaylor The remaining Matplotlib errors look to me to be Matplotlib bugs, as the 2-d arrays that cause the problem are returned by the overloaded _interpolate_single_key function in CubicTriInterpolator that is documented in the base class to return a 1-d array, whereas the actual dimensions are of the form (n, 1). The question is, what is the best work around here? Do we want to break Matplotlib on account of a bug that was previously accepted by Numpy? Perhaps a FutureWarning rather than an error would be more appropriate at this point.

tacaswell · 2014-07-04T20:29:02Z

@charris Are you testing against master of matplotlib? Is there a discussion else where of what failures you are fixing?

@ianthomas23 is the matplotlib export on the triangulation code.

charris · 2014-07-04T20:36:05Z

@tacaswell I'm testing against 1.3.1 from Fedora 20, I didn't want to actually build that beast ;)

charris · 2014-07-04T20:45:46Z

@tacaaswell Christoph Gohlke reported the test failures. There is some discussion on the numpy mailing list in the 1.9.0b1 release thread, and the errors may be found at http://www.lfd.uci.edu/~gohlke/pythonlibs/tests/20140609-win-amd64-py2.7-numpy-1.9.0b1/,

tacaswell · 2014-07-04T22:13:44Z

mpl isn't that bad to build on linux. I ask because the triangulation code got a major overhaul and we are very near (hopefully by/at scipy) to releasing mpl 1.4

charris · 2014-07-04T22:52:39Z

@tacaswell Well, I read the INSTALL file and it discouraged me ;). I used to build it years ago...

seberg · 2014-07-05T14:20:45Z

I am not sure that it actually matters much to us if you fixed this in matplotlib. I will probably have a bit more look at it today. But overall I think this is almost done/OK in the sense that the method is OK and only string formatting/error handling details need to be smoothed out (which is tedious in C...).

One small annoying detail about the new errors is that badly shaped boolean arrays (single array case) currently have worse errors, because the boolean array is converted to an integer one early on (and the error is reported for that). But I think we can live without improving that for 1.9 certainly.

charris · 2014-07-05T14:37:51Z

@seberg I've done the spelling & grammar fixes, so you can skip those if you work on this.

tacaswell · 2014-07-05T15:02:47Z

matplotlib/matplotlib#3186 and matplotlib/matplotlib#3188

seberg · 2014-07-05T15:12:51Z

Quick question, since the ML thread is long and I am not quite sure. So even with this applied matplotlib still has failures?

tacaswell · 2014-07-05T15:16:35Z

with matplotlib/matplotlib#3188 applied the errors that @charris reported are fixed. I am currently re-running the entire test mpl test suite using np 1.9.0.dev-7a2b14a.

seberg · 2014-07-05T15:19:02Z

No, I mean applying this PR, it should already fix all those indexing errors, assuming you are ignoring the deprecation/future warnings (just error reporting isn't very good yet).

charris · 2014-07-05T15:26:02Z

On Sat, Jul 5, 2014 at 9:19 AM, seberg [email protected] wrote:

No, I mean applying this PR, it should already fix all those indexing
errors, assuming you are ignoring the deprecation/future warnings (just
error reporting isn't very good yet).

Yes. There were two types of errors, one involving a flattened rhs, which
this patch fixes, and the other involving an unflattened rhs which should
have been 1-d, with boolean indexing on the left.

Most problems with other packages were of the 1-d size mismatch sort, with
the rhs either too long with excess values discarded, or too short with
repetition used to fill out the data.

The scikit-learn errors are different and I'll be taking a closer look at
them today.

Chuck

charris · 2014-07-05T17:26:56Z

I take that back, matplotlib segfaults.

charris · 2014-07-05T17:43:14Z

Yep, the scikits-learn failures remain, so they are unrelated to this.

seberg · 2014-07-05T19:16:30Z

Any hint on the matplotlib segfault? I guess this code might have a refcount bug or so.

charris · 2014-07-05T19:26:46Z

I'll try for a backtrace. Looks like sklearn is fixed in master.

seberg · 2014-07-05T19:29:53Z

An example would be neat. Frankly, I only got the ubuntu matplotlib right now and that does not include the testing stuff...

charris · 2014-07-05T19:35:03Z

You can clonee the matplotlib repository, cd into it, and do python tests.py. That will run tests against your installed matplotlib.

charris · 2014-07-05T19:43:57Z

This looks like the relevant part of the backtrace.

#0  0x0000003e1449bd5f in ?? () from /lib64/libpython2.7.so.1.0
#1  0x00007fffef7f7e4a in attempt_1d_fallback (self=self@entry=0x4c3e0df0, 
    ind=0x4bc86030, op=op@entry=0x4e9a7940)
    at numpy/core/src/multiarray/mapping.c:1682
#2  0x00007fffef7fa0e0 in array_assign_subscript (self=0x4c3e0df0, 
    ind=0x4bc86030, op=<optimized out>)
    at numpy/core/src/multiarray/mapping.c:1809

seberg · 2014-07-05T19:57:28Z

Oops, crap, I mus thave screwed up editing back and forth. There should be an incref on the error there.

seberg · 2014-07-05T20:02:21Z

OK, pushed that fix. At least I assume this, otherwise there would be no error set, but I don't think that can happen. It seems it just happened to work almost always because of how python does error stuff.

charris · 2014-07-05T20:06:34Z

That won't work, the declarations don't all precede statements.

seberg · 2014-07-05T20:08:49Z

Ops, changed.

seberg · 2014-07-05T20:12:16Z

Though I guess we need to go over all that error handling back and forth anyway, and improve the warnings.

charris · 2014-07-05T20:14:06Z

numpy/core/src/multiarray/mapping.c

+    }
+
+    Py_XDECREF((PyObject *)self_iter);
+    Py_DECREF(err);


Will fail if err is NULL.

If this occurs, there would be a bug elsewhere. But I am not sure if such a bug could not be even outside numpy, so I suppose we should do the sanity check...

charris · 2014-07-05T20:57:58Z

Matplotlib tests pass now. Thanks Sebastian, with this and maybe one other fix I think we can do the next beta release.

charris · 2014-07-06T02:23:39Z

@seberg There seems to be one more indexing issue that affects pandas and pyfits. The pyfits discussion is here along with a bisection and discussion.

juliantaylor · 2014-07-06T12:37:38Z

the pyfits readonly failure looks similar to the scipy failure, I'll have a look

juliantaylor · 2014-07-06T12:59:58Z

minimal testcase:

d = np.rec.array([('NGC1001', 11.100000381469727), ('NGC1002', 12.300000190734863),
       ('NGC1003', 15.199999809265137)], 
      dtype=[('target', 'S20'), ('V_mag', '>f4')])
assert d[np.array([False,  True,  True], dtype=bool)].flags.writeable

juliantaylor · 2014-07-06T13:19:13Z

would this be a fix?

--- a/numpy/core/src/multiarray/mapping.c
+++ b/numpy/core/src/multiarray/mapping.c
@@ -1058,6 +1058,9 @@ array_boolean_subscript(PyArrayObject *self,
             Py_DECREF(ret);
             return NULL;
         }
+        if (_IsWriteable(ret)) {
+            PyArray_ENABLEFLAGS(ret, NPY_ARRAY_WRITEABLE);
+        }
     }

     return ret;

charris · 2014-07-06T16:06:47Z

The commit messages could use a bit of cleanup, but I'm going to put this in as is. Thanks again Sebastian.

WIP: Fix matplotlib, etc. errors

seberg · 2014-07-06T18:44:03Z

This seems like the correct fix:

--- a/numpy/core/src/multiarray/mapping.c
+++ b/numpy/core/src/multiarray/mapping.c
@@ -1569,7 +1569,7 @@ array_subscript(PyArrayObject *self, PyObject *op)
                                       PyArray_SHAPE(tmp_arr),
                                       PyArray_STRIDES(tmp_arr),
                                       PyArray_BYTES(tmp_arr),
-                                      0, /* TODO: Flags? */
+                                      PyArray_FLAGS(self),
                                       (PyObject *)self);

The result now does not own the data necessarily, which was the case before. If that is needed, we would have to change the logic a bit (not set the base, but instead override the owndata flag)

seberg · 2014-07-06T18:44:40Z

Yeah, should have been better, didn't think you would put it in yet :)

seberg · 2014-07-06T18:45:33Z

I guess the flags don't matter if we allocate new data, etc. but since this isn't the case here, they seem to matter.

charris · 2014-07-06T19:06:07Z

Maybe I was a bit hasty.

seberg added this to the 1.9 blockers milestone Jun 11, 2014

WIP: Fix matplotlib, etc. errors

480432b

charris reviewed Jun 17, 2014
View reviewed changes

tacaswell mentioned this pull request Jul 5, 2014

Numpy 1.9 issues. matplotlib/matplotlib#3186

Closed

BUG: Add missing error incref in 1d indexing fallback

9c4d48c

charris reviewed Jul 5, 2014
View reviewed changes

juliantaylor mentioned this pull request Jul 6, 2014

Pyfits issue in numpy 1.9 astropy/astropy#2693

Closed

charris added a commit that referenced this pull request Jul 6, 2014

Merge pull request #4804 from seberg/fancy-ass-1d

8dba040

WIP: Fix matplotlib, etc. errors

charris merged commit 8dba040 into numpy:master Jul 6, 2014

seberg mentioned this pull request Jul 7, 2014

1.9 deprecations: Follow-up ticket #4132

Closed

6 tasks

severin-lemaignan pushed a commit to severin-lemaignan/makehuman-commandline that referenced this pull request Jan 6, 2016

Anticipate a numpy 1.9 deprecation warning (numpy/numpy#4804)

4eca0d2

Uh oh!

WIP: Fix matplotlib, etc. errors #4804

WIP: Fix matplotlib, etc. errors #4804

Uh oh!

Conversation

seberg commented Jun 11, 2014

Uh oh!

seberg commented Jun 11, 2014

Uh oh!

seberg commented Jun 15, 2014

Uh oh!

seberg commented Jun 16, 2014

Uh oh!

charris Jun 17, 2014

Choose a reason for hiding this comment

Uh oh!

charris commented Jun 17, 2014

Uh oh!

seberg commented Jun 17, 2014

Uh oh!

charris commented Jun 30, 2014

Uh oh!

seberg commented Jun 30, 2014

Uh oh!

charris commented Jun 30, 2014

Uh oh!

charris commented Jul 4, 2014

Uh oh!

tacaswell commented Jul 4, 2014

Uh oh!

charris commented Jul 4, 2014

Uh oh!

charris commented Jul 4, 2014

Uh oh!

tacaswell commented Jul 4, 2014

Uh oh!

charris commented Jul 4, 2014

Uh oh!

seberg commented Jul 5, 2014

Uh oh!

charris commented Jul 5, 2014

Uh oh!

tacaswell commented Jul 5, 2014

Uh oh!

seberg commented Jul 5, 2014

Uh oh!

tacaswell commented Jul 5, 2014

Uh oh!

seberg commented Jul 5, 2014

Uh oh!

charris commented Jul 5, 2014

Uh oh!

charris commented Jul 5, 2014

Uh oh!

charris commented Jul 5, 2014

Uh oh!

seberg commented Jul 5, 2014

Uh oh!

charris commented Jul 5, 2014

Uh oh!

seberg commented Jul 5, 2014

Uh oh!

charris commented Jul 5, 2014

Uh oh!

charris commented Jul 5, 2014

Uh oh!

seberg commented Jul 5, 2014

Uh oh!

seberg commented Jul 5, 2014

Uh oh!

charris commented Jul 5, 2014

Uh oh!

seberg commented Jul 5, 2014

Uh oh!

seberg commented Jul 5, 2014

Uh oh!

charris Jul 5, 2014

Choose a reason for hiding this comment

Uh oh!

seberg Jul 5, 2014