ENH: Further expansion of gufunc signature to allow broadcasting #11179

mhvk · 2018-05-28T18:52:24Z

Builds on #11175 to include allowing broadcasting in gufuncs (which now is surprisingly easy)>

With this, @mattharrigan's all_equal (#8528) would become truly useful, as it also allows things like np.all_equal(ndarray, const, axis=?), i.e., one of the arguments can be broadcast over its core.

EDIT: has decent tests but does still need docs.

eric-wieser · 2018-05-28T21:11:20Z

To be clear, this only enables broadcasting when explicitly requested? I think doing it always would be harmful, and would let bugs happen silently.

mhvk · 2018-05-29T00:36:16Z

@eric-wieser - yes, by adding |1 to the dimension name or number -- clearly I should aim to add some docs! Example: for all_equal, the signature would become (n|1),(n|1)->() (as shown by the new example in _umath.tests.src - though the implementation there is a dumb one, unlike #8528).

shoyer · 2018-05-29T01:38:29Z

doc/source/reference/c-api.generalized-ufuncs.rst

@@ -138,22 +141,35 @@ Notes:
   Each dimension name typically corresponds to one level of looping in the
   elementary function's implementation.
 #. White spaces are ignored.
+#. An integer as a dimension name freezes that dimension to the value,
+#. The name can be suffixed with a question mark, this make the dimension a
+   core dimension only if it exists on the input or output, otherwise 1 is used.


Should be something like: otherwise a dimensions of size 1 is inserted into the inputs and squeezed from the outputs.

Also, we should explain exactly when optional dimensions are used. The rule from matmul is that optional dimensions are used whenever there are enough dimensions for them to be well defined, but matmul doesn't have optional core dimensions that appear on multiple arguments.

For example, how should a signature like (n,m?),(n,m?)->() be interpreted if the two inputs are 1d and 2d? There are two reasonable interpretations I can think of:

Using the signature (n),(n)->(), because not all arguments have enough dimensions.

Raising an error, because the behavior is potentially ambiguous.

O dear, what did I do - this discussion really belongs in @mattip's PR... Though in his PR I'm not exactly sure what the behaviour is...

Here, I do know: the moment any argument has too few dimensions, flexible dimensions are effectively removed altogether from the ufunc, i.e., it follows your option 1. I think this is most consistent with matmul, where, if one were to give an output, one would similarly expect that the dimension would just "not exist".

p.s. And apologies for not adding documentation on this beyond what @mattip had - do look at the tests - I have those for both ? (flexible) and |1 (broadcasting) - but still need to add some for numerical axes; so far, for those there are just tests that the signature gets parsed right.

(I had a separate PR where it was per argument, but it became clear to me that was just going to be a mess...)

mattip · 2018-05-29T17:33:06Z

I would like to see the enhanced core signature functionality broken into a number of PRs. It seems this should be the last one, and needs some discussion on the mailing list.

mhvk · 2018-05-29T18:54:59Z

Just pushed an update to the documentation explaining what the enhancements actually provide (given the rebase conflicts, the tests are currently meaningless, but off-line things pass).

mhvk · 2018-05-29T18:59:27Z

@mattip - completely agree. I pushed this here mostly so that you could see how it worked with flags, and that it was easy to add further features.

Do you have specific suggestions on order? Frozen sizes is by far the easiest, but I'm happy to start with ? as well, since we can then proceed with matmul. But perhaps for the ufunc struct, we can agree that we'll immediately add both core_dim_sizes and core_dim_flags?

njsmith · 2018-05-29T23:42:18Z

The nice thing about ? is that it has a clear use case (matmul) and has been discussed a fair amount on the list and elsewhere. For frozen dimensions and broadcasting core dimensions I'm much less comfortable that we understand the benefits and trade-offs of adding them.

njsmith · 2018-05-29T23:43:05Z

Actually if I was going to suggest adding one thing to gufuncs besides ?, it would be features to let us convert more of the popular numpy functions into gufuncs, like sort.

mhvk · 2018-05-30T00:14:45Z

@njsmith - there is actually a very clear use case for broadcasting, the all_equal gufunc written by @mattharrigan (#8528), which makes much more sense with allowing broadcasting of one of the arguments (especially in combination with the axis keyword of #11018), as it can then truly be the short-circuit, memory-saving equivalent of (a == b).all(axis=...)

What is your other suggestion for an extension?

mhvk · 2018-05-30T00:33:06Z

ps. For the frozen dimensions, I'd really like them - right now, I'm hacking the type-resolver to insist on dimensions of 3 in standards-of-astronomy code I'm wrapping in ufuncs; see https://github.com/mhvk/astropy/blob/88312819cdbf78605dfe5b9eb4c8245cae8bb887/astropy/_erfa/ufunc.c.templ#L587 (and see the code above for other hacks I have to do to make the machinery work...)

So that we can keep track of expansions of the PyUFuncObject.

Goal is allow signatures like (m?,n),(n,p?)->(m?,p?) for matmul.

This allows future expansion and is arguably easier to follow. Also adds tests, of the signature parsing, of the ignoring of dimensions, and of use of the frozen dimensions via cross1d.

It got very long and now matches logically the later call to get core sizes.

…e equal. Was comparison by the string representation so that "03" and "3" would be considered different dimensions. That would be surprising for implementing an inner loop...

Previously, when a core dimension at the end was missing, the strides were not re-ordered accordingly, and the inner loop code needed to check whether the corresponding dimension was zero. But if the strides are passed on correctly, this is not needed at all, and the dimension can just be unity (not indicating broadcasting this time).

That is, parts like core_num_dims which can get changed during execution, e.g., because keepdims is set or a dimension is flexible.

Now op_core_num_dims to make clear it is the number of core dimensions an operand has, rather than that assumed by the ufunc.

But only test the non-broadcast version since that's all that will work here.

mhvk · 2018-07-17T18:26:46Z

Closing since the discussion of the NEP concluded that we would defer on adding broadcasting.

mhvk added 01 - Enhancement 15 - Discussion component: numpy._core labels May 28, 2018

mhvk mentioned this pull request May 29, 2018

ENH: Generalized ufunc signature expansion for frozen and flexible dimensions #11175

Merged

shoyer reviewed May 29, 2018

View reviewed changes

mhvk force-pushed the gufunc-allow-broadcasting branch from a108558 to 3569304 Compare May 30, 2018 17:53

mhvk changed the title ~~WIP, ENH: Further expansion of gufunc signature to allow broadcasting~~ ENH: Further expansion of gufunc signature to allow broadcasting May 30, 2018

mhvk force-pushed the gufunc-allow-broadcasting branch 5 times, most recently from 131f71c to 1ad1a66 Compare June 2, 2018 19:19

mattip added the component: numpy.ufunc label Jun 8, 2018

mhvk force-pushed the gufunc-allow-broadcasting branch from 1ad1a66 to 244682a Compare June 10, 2018 17:21

jaimefrio and others added 7 commits June 11, 2018 15:09

ENH: Add frozen dimensions to gufunc signatures

9bba990

ENH: convert reserved field to version, set new version to 1

c2cb7f3

So that we can keep track of expansions of the PyUFuncObject.

ENH: test, document, implement flexible signature.

7744213

Goal is allow signatures like (m?,n),(n,p?)->(m?,p?) for matmul.

MAINT: Use flags instead of flexible_dims argument.

866dbca

This allows future expansion and is arguably easier to follow. Also adds tests, of the signature parsing, of the ignoring of dimensions, and of use of the frozen dimensions via cross1d.

MAINT: Move dimension logic to its own routine; simplify.

51e4937

It got very long and now matches logically the later call to get core sizes.

MAINT: Remove n_dims_used, and reuse core_num_dims instead.

068ed9a

MAINT: Ensure signatures with frozen sizes written differently compar…

d48ccfd

…e equal. Was comparison by the string representation so that "03" and "3" would be considered different dimensions. That would be surprising for implementing an inner loop...

mhvk and others added 11 commits June 11, 2018 15:10

DOC: updates and fixes for gufunc descriptions.

e3d4886

DOC: Release notes for frozen dimensions.

738deb0

DOC: Release note for flexible dimensions.

2d84594

MAINT: factor out initialization of variable parts of the ufunc.

b350acc

That is, parts like core_num_dims which can get changed during execution, e.g., because keepdims is set or a dimension is flexible.

MAINT: change the name of the actual core_num_dims.

53e84f5

Now op_core_num_dims to make clear it is the number of core dimensions an operand has, rather than that assumed by the ufunc.

ENH: Recognize |1 as can_broadcast in gufunc signature parser.

67a13b0

TST: add all_equal gufunc with can_broadcast signature.

427218a

But only test the non-broadcast version since that's all that will work here.

ENH: Make broadcasting work for simple cases.

6b7178b

ENH: Allow core broadcasting also for missing dimensions.

28f3394

DOC: document broadcastable and fixed-size core dimensions

e7f3139

mhvk force-pushed the gufunc-allow-broadcasting branch from 244682a to e7f3139 Compare June 11, 2018 19:11

mhvk closed this Jul 17, 2018

mhvk deleted the gufunc-allow-broadcasting branch December 21, 2018 02:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

ENH: Further expansion of gufunc signature to allow broadcasting #11179

ENH: Further expansion of gufunc signature to allow broadcasting #11179

Uh oh!

mhvk commented May 28, 2018 •

edited

Loading

Uh oh!

eric-wieser commented May 28, 2018

Uh oh!

mhvk commented May 29, 2018

Uh oh!

shoyer May 29, 2018

Uh oh!

mhvk May 29, 2018

Uh oh!

mattip commented May 29, 2018

Uh oh!

mhvk commented May 29, 2018

Uh oh!

mhvk commented May 29, 2018

Uh oh!

njsmith commented May 29, 2018

Uh oh!

njsmith commented May 29, 2018

Uh oh!

mhvk commented May 30, 2018

Uh oh!

mhvk commented May 30, 2018

Uh oh!

mhvk commented Jul 17, 2018

Uh oh!

Uh oh!

Uh oh!

ENH: Further expansion of gufunc signature to allow broadcasting #11179

ENH: Further expansion of gufunc signature to allow broadcasting #11179

Uh oh!

Conversation

mhvk commented May 28, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eric-wieser commented May 28, 2018

Uh oh!

mhvk commented May 29, 2018

Uh oh!

shoyer May 29, 2018

Choose a reason for hiding this comment

Uh oh!

mhvk May 29, 2018

Choose a reason for hiding this comment

Uh oh!

mattip commented May 29, 2018

Uh oh!

mhvk commented May 29, 2018

Uh oh!

mhvk commented May 29, 2018

Uh oh!

njsmith commented May 29, 2018

Uh oh!

njsmith commented May 29, 2018

Uh oh!

mhvk commented May 30, 2018

Uh oh!

mhvk commented May 30, 2018

Uh oh!

mhvk commented Jul 17, 2018

Uh oh!

Uh oh!

mhvk commented May 28, 2018 •

edited

Loading