BUG: don't silence warnings in ufunc.reduce #9020

eric-wieser · 2017-04-28T23:48:04Z

Needed a file of helper functions so that this was actually visible in reduction.c. This move is in the first commit.

Fix itself is trivial once things are visible.

eric-wieser · 2017-04-29T00:40:29Z

Tests fail right now because we're now being warned about more things. Most of these are correct (ie, testing nan behavior), and need test updates.

I don't think these should warn at all when invoked with (1, np.nan) or .reduce([1, np.nan]), since they are designed to explicitly handle nans:

np.minimum
np.maximum
np.fmin
np.fmax

~~However, I don't know how to disable the warnings there.~~

Also, one test fails due to an overflow that was silenced! Which raises a good point - overflow in np.sum is something we really should have been catching

eric-wieser · 2017-04-29T01:53:27Z

Ok, all fixed I think. Probably easiest to review this commit-by-commit, since the first refactor is noisy

juliantaylor · 2017-04-29T10:07:53Z

maximum/minimum should warn, nan's may lead to unexpected results. We also have the nanmax/nanmin functions for these.

eric-wieser · 2017-04-29T10:09:03Z

What's the history behind having both fmax and nanmax?

eric-wieser · 2017-04-29T12:48:10Z

@juliantaylor: Ok, warnings are now only squashed in fmin and fmax

charris · 2017-05-12T16:01:17Z

Needs rebase.

charris · 2017-05-23T00:30:31Z

What's the history behind having both fmax and nanmax?

fmax is a binary ufunc implementing the standard C fmax function. The nanmax function uses it via the reduce method when it can, but see the note in the implementation. It might can be changed now that __array_ufunc__ is in.

Upon successful completion, these functions shall return the maximum numeric value of their arguments.

If just one argument is a NaN, the other argument shall be returned.

eric-wieser · 2017-06-01T13:46:37Z

@charris: Rebased

mhvk

Mostly nitpicks, but also warnings in the compilation:

    numpy/core/src/umath/ufunc_object.c:3619:15: warning: unused variable 'out_obj' [-Wunused-variable]

    numpy/core/src/umath/reduction.c:562:5: warning: implicit declaration of function 'PyUFunc_clearfperr' [-Wimplicit-function-declaration]

I think the first is likely a result of the rebase; the definition can be removed from l.3619, as it is moved to 3655.

The second I'm not sure about; a missing include perhaps?

mhvk · 2017-06-01T15:05:54Z

numpy/core/src/umath/extobj.c

+    }
+
+    return 0;
+}


Missing end of line.

mhvk · 2017-06-01T15:06:06Z

numpy/core/src/umath/extobj.h

+ufunc_update_use_defaults(void);
+#endif
+
+#endif


Missing end of line.

mhvk · 2017-06-01T15:09:33Z

numpy/core/src/umath/reduction.c

@@ -436,6 +437,7 @@ PyArray_InitializeReduceResult(
 * loop        : The loop which does the reduction.
 * data        : Data which is passed to assign_identity and the inner loop.
 * buffersize  : Buffer size for the iterator. For the default, pass in 0.
+ * errormask   : forwarded from _get_bufsize_errmask


Since it comes after funcname in the signature, move it below it here as well.

mhvk · 2017-06-01T15:10:33Z

numpy/core/src/umath/reduction.h

@@ -149,6 +149,7 @@ PyUFunc_ReduceWrapper(PyArrayObject *operand, PyArrayObject *out,
                      int subok,
                      PyArray_AssignReduceIdentityFunc *assign_identity,
                      PyArray_ReduceLoopFunc *loop,
-                      void *data, npy_intp buffersize, const char *funcname);
+                      void *data, npy_intp buffersize, const char *funcname,
+                      int errormask);


Add errormask to the description above as well.

eric-wieser · 2017-06-02T01:44:52Z

Don't understand how this patch could have introduced the warning about out_obj, since the diff doesn't even contain that word, but all your nitpicks are fair

This makes it visible inside other files, like reduction.c Apart from the includes, everything here is just a direct move - there are no code changes

Previously, these would do so when encountering nans

The two most visible consequences are warnings for: * np.min and np.max when inputs contain nan * np.sum when the summation (of floats) overflows Implementation taken from the code for ufunc.__call__ Fixes numpy#8954

eric-wieser · 2017-06-02T10:11:02Z

Nitpicks addressed, but still not sure about that missing function prototype. Works fine on MSVC though...

mhvk · 2017-06-02T20:30:10Z

This now looks all OK. The one remaining question is whether this is worth backporting. Including the MAINT commits, it is a bit of a large code shuffle, but it does fix a bug. @charris?

charris · 2017-06-02T21:57:51Z

@mhvk That's a lot of code. If the bug is long standing I would leave it for 1.14. I'm OK with backporting fixes to __array_ufunc__, but this doesn't look to qualify for that.

I'm going to release the 1.13.0 final at the beginning of next week.

eric-wieser · 2017-06-03T12:00:16Z

That's a lot of code.

Not really - almost all of it (ie, the first MAINT commit) is just moving code from one file to another. I agree though, this isn't terribly deserving of a backport

charris · 2017-06-24T17:09:24Z

numpy/core/src/umath/extobj.c

+
+#include "ufunc_object.h"  /* for npy_um_str_pyvals_name */
+
+#if USE_USE_DEFAULTS==1


Spaces around operators.

Again, this is verbatim from the other file.

charris · 2017-06-24T17:10:33Z

numpy/core/src/umath/extobj.c

+    int res;
+
+    PyUFunc_NUM_NODEFAULTS += 1;
+    res = PyUFunc_GetPyValues("test", &bufsize, &errmask, &errobj);


Beats me - again, just a maintenance change, moving this from the other file

charris · 2017-06-24T17:12:54Z

numpy/core/src/umath/extobj.c

+        return 0;
+    }
+
+    NPY_ALLOW_C_API;


Does this buy anything? A small speedup in error handling, which should be rare, doesn't seem worth it.

eric-wieser · 2017-06-24T17:27:01Z

Quoting the first commit:

Apart from the includes, everything here is just a direct move - there are no code changes

I'd rather not open the can of poorly-documented worms that is how extobj actually works

eric-wieser · 2017-07-11T21:24:36Z

@charris: Any other remarks here?

charris · 2017-07-12T01:31:08Z

Nope, might as well put it in. Thanks Eric.

rgommers · 2017-07-14T08:32:19Z

Please do not backport this - no need, and it will show warnings in the scipy test suite (and other packages probably) for released versions of scipy for which the test suite should be clean.

eric-wieser added 03 - Maintenance 55 - Needs work component: numpy._core labels Apr 28, 2017

eric-wieser changed the title ~~[WIP] MAINT: Move functions handling ufunc extobj~~ [help needed] MAINT: Move functions handling ufunc extobj Apr 28, 2017

eric-wieser force-pushed the reduce-warn branch from 8273a08 to 9794fd1 Compare April 29, 2017 00:09

eric-wieser changed the title ~~[help needed] MAINT: Move functions handling ufunc extobj~~ BUG: Catch warnings in ufunc.reduce Apr 29, 2017

eric-wieser removed the 55 - Needs work label Apr 29, 2017

eric-wieser force-pushed the reduce-warn branch 3 times, most recently from c00d928 to 06bbb23 Compare April 29, 2017 01:51

eric-wieser changed the title ~~BUG: Catch warnings in ufunc.reduce~~ BUG: don't silence warnings in ufunc.reduce Apr 29, 2017

eric-wieser force-pushed the reduce-warn branch from 06bbb23 to 9ef93b5 Compare April 29, 2017 11:41

This comment has been minimized.

Sign in to view

eric-wieser force-pushed the reduce-warn branch 2 times, most recently from 1656520 to 33c269c Compare May 1, 2017 10:49

This comment has been minimized.

Sign in to view

eric-wieser force-pushed the reduce-warn branch from 33c269c to 02a262b Compare June 1, 2017 13:46

mhvk reviewed Jun 1, 2017

View reviewed changes

eric-wieser added 3 commits June 2, 2017 11:09

MAINT: Move functions handling ufunc extobj

32436ea

This makes it visible inside other files, like reduction.c Apart from the includes, everything here is just a direct move - there are no code changes

MAINT: Do not produce warnings in fmin/fmax functions

1f9ab2c

Previously, these would do so when encountering nans

BUG: Check for warnings in ufunc.reduce

d47ca7b

The two most visible consequences are warnings for: * np.min and np.max when inputs contain nan * np.sum when the summation (of floats) overflows Implementation taken from the code for ufunc.__call__ Fixes numpy#8954

MAINT: Remove unused variable

c659955

eric-wieser force-pushed the reduce-warn branch from 02a262b to c659955 Compare June 2, 2017 10:09

mhvk added this to the 1.14.0 release milestone Jun 2, 2017

charris reviewed Jun 24, 2017

View reviewed changes

charris merged commit 7e1a322 into numpy:master Jul 12, 2017

perimosocordiae mentioned this pull request Jul 13, 2017

BUG: Trimmed stats tests are failing on numpy master scipy/scipy#7598

Closed

This was referenced May 4, 2018

BUG: np.min does not always propagate NaNs #10370

Closed

BUG: reduce using SSE only warns if inside SSE loop #11043

Merged

eric-wieser mentioned this pull request Sep 26, 2018

numpy.maximum since 1.15 raises RuntimeWarning when encountering a NaN even though the docs say it should propagate NaNs #12038

Closed

mattip mentioned this pull request Oct 21, 2018

BUG: maximum, minimum no longer emit warnings on NAN #12236

Merged


		#include "ufunc_object.h" /* for npy_um_str_pyvals_name */

		#if USE_USE_DEFAULTS==1

Uh oh!

BUG: don't silence warnings in ufunc.reduce #9020

BUG: don't silence warnings in ufunc.reduce #9020

Uh oh!

Conversation

eric-wieser commented Apr 28, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eric-wieser commented Apr 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eric-wieser commented Apr 29, 2017

Uh oh!

juliantaylor commented Apr 29, 2017

Uh oh!

eric-wieser commented Apr 29, 2017

Uh oh!

eric-wieser commented Apr 29, 2017

Uh oh!

This comment has been minimized.

This comment has been minimized.

charris commented May 12, 2017

Uh oh!

charris commented May 23, 2017

Uh oh!

eric-wieser commented Jun 1, 2017

Uh oh!

mhvk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eric-wieser commented Jun 2, 2017

Uh oh!

eric-wieser commented Jun 2, 2017

Uh oh!

mhvk commented Jun 2, 2017

Uh oh!

charris commented Jun 2, 2017

Uh oh!

eric-wieser commented Jun 3, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

charris Jun 24, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eric-wieser commented Jun 24, 2017

Uh oh!

eric-wieser commented Jul 11, 2017

Uh oh!

charris commented Jul 12, 2017

Uh oh!

rgommers commented Jul 14, 2017

Uh oh!

Uh oh!

eric-wieser commented Apr 28, 2017 •

edited

Loading

eric-wieser commented Apr 29, 2017 •

edited

Loading

eric-wieser commented Jun 3, 2017 •

edited

Loading

charris Jun 24, 2017 •

edited

Loading