[Bug]: Matplotlib 3.5 breaks unyt integration of error bars #21669

neutrinoceros · 2021-11-18T14:28:25Z

Bug summary

Discovered this in unyt's daily CI jobs: https://github.com/yt-project/unyt/actions

Code for reproduction

import matplotlib.pyplot as plt
import unyt as un

un.matplotlib_support.enable()
fig, ax = plt.subplots()

x = un.unyt_array([1, 2], "cm")
y = un.unyt_array([3, 4], "kg")
yerr = un.unyt_array([0.1, 0.2], "kg")
ax.errorbar(x, y, yerr=(yerr, yerr))

Actual outcome

UnitOperationError: The <ufunc 'subtract'> operator for unyt_arrays with units "kg" 
(dimensions "(mass)") and "dimensionless" (dimensions "1") is not well defined.

detailed raceback

╭──────────────────────────── Traceback (most recent call last) ────────────────────────────╮
│                                                                                           │
│ /private/tmp/matplotlib/t.py:10 in <module>                                               │
│                                                                                           │
│    7 x = un.unyt_array([1, 2], "cm")                                                      │
│    8 y = un.unyt_array([3, 4], "kg")                                                      │
│    9 yerr = un.unyt_array([0.1, 0.2], "kg")                                               │
│ ❱ 10 ax.errorbar(x, y, yerr=yerr)                                                         │
│   11                                                                                      │
│ /private/tmp/matplotlib/lib/matplotlib/__init__.py:1348 in inner                          │
│                                                                                           │
│   1345 │   @functools.wraps(func)                                                         │
│   1346 │   def inner(ax, *args, data=None, **kwargs):                                     │
│   1347 │   │   if data is None:                                                           │
│ ❱ 1348 │   │   │   return func(ax, *map(sanitize_sequence, args), **kwargs)               │
│   1349 │   │                                                                              │
│   1350 │   │   bound = new_sig.bind(ax, *args, **kwargs)                                  │
│   1351 │   │   auto_label = (bound.arguments.get(label_namer)                             │
│                                                                                           │
│ /private/tmp/matplotlib/lib/matplotlib/axes/_axes.py:3474 in errorbar                     │
│                                                                                           │
│   3471 │   │   │   │   │   │   xo, yo, marker='|', **eb_cap_style))                       │
│   3472 │   │                                                                              │
│   3473 │   │   if yerr is not None:                                                       │
│ ❱ 3474 │   │   │   lower, upper = extract_err('y', yerr, y, lolims, uplims)               │
│   3475 │   │   │   barcols.append(self.vlines(                                            │
│   3476 │   │   │   │   *apply_mask([x, lower, upper], everymask), **eb_lines_style))      │
│   3477 │   │   │   # select points without upper/lower limits in y and                    │
│                                                                                           │
│ /private/tmp/matplotlib/lib/matplotlib/axes/_axes.py:3434 in extract_err                  │
│                                                                                           │
│   3431 │   │   │   │   │   f"'{name}err' (shape: {np.shape(err)}) must be a scalar "      │
│   3432 │   │   │   │   │   f"or a 1D or (2, n) array-like whose shape matches "           │
│   3433 │   │   │   │   │   f"'{name}' (shape: {np.shape(data)})") from None               │
│ ❱ 3434 │   │   │   return data - low * ~lolims, data + high * ~uplims  # low, high        │
│   3435 │   │                                                                              │
│   3436 │   │   if xerr is not None:                                                       │
│   3437 │   │   │   left, right = extract_err('x', xerr, x, xlolims, xuplims)              │
│                                                                                           │
│ /Users/robcleme/.pyenv/versions/3.9.7/envs/tmp_unit/lib/python3.9/site-packages/unyt/arra │
│ y.py:1764 in __array_ufunc__                                                              │
│                                                                                           │
│   1761 │   │   │   │   │   │   │   │   else:                                              │
│   1762 │   │   │   │   │   │   │   │   │   raise UnitOperationError(ufunc, u0, u1)        │
│   1763 │   │   │   │   │   │   else:                                                      │
│ ❱ 1764 │   │   │   │   │   │   │   raise UnitOperationError(ufunc, u0, u1)                │
│   1765 │   │   │   │   │   conv, offset = u1.get_conversion_factor(u0, inp1.dtype)        │
│   1766 │   │   │   │   │   new_dtype = np.dtype("f" + str(inp1.dtype.itemsize))           │
│   1767 │   │   │   │   │   conv = new_dtype.type(conv)                                    │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
UnitOperationError: The <ufunc 'subtract'> operator for unyt_arrays with units "kg" 
(dimensions "(mass)") and "dimensionless" (dimensions "1") is not well defined.

Expected outcome

This is the output figure with matplotlib 3.4.3

Additional information

The regression bisects to #19526 (8cd22b4)

I note that if replace the last line with

ax.errorbar(x, y, yerr=yerr)

(passing a single array instead of a tuple), it doesn't break on the main branch.

Operating system

Unbuntu

Matplotlib Version

3.5.0

Matplotlib Backend

not relevant (I get an identical error with the 'agg' backend)

Python version

tested on 3.7 to 3.9

Jupyter version

No response

Installation

pip

The text was updated successfully, but these errors were encountered:

neutrinoceros · 2021-11-18T14:50:27Z

(updated because I saw that that my script was overly simplified to a point that it was actually not broken on the main branch)

neutrinoceros · 2021-11-18T14:54:40Z

I think I got a fix, will open a PR shortly !

neutrinoceros · 2021-11-18T15:38:49Z

After digging a bit I'm actually stuck on this comment:

matplotlib/lib/matplotlib/axes/_axes.py

Line 3281 in 552e078

# Casting to object arrays preserves units.

I don't know where this idea comes from
it's not (currently ?) true for unyt arrays

It seems crucial to understand why this assumption was made in matplotlib, and wether it complies with NEP 18 (array_function protocol).
It is entirely possible at this stage that the actual issue is on the unyt side, because it's not up to date with NEP 18.
Also note that unyt's integration is still considered experimental (to the best of my knowledge).

anntzer · 2021-11-18T15:47:42Z

I think(??) I got this belief from pint, but not sure either. OTOH, is there any thing we can do to convert unitful arraylikes to actual arrays that 1) works across unit libs and 2) does not destroy units?
Possibly, it may make sense for numpy to rule on an expected behavior here...

ngoldbaum · 2021-11-18T16:04:20Z

If you’re handed an ndarray subclass, I think casting to a base ndarray would work across libraries? Pint doesn’t implement an ndarray subclass so expecting all unit libraries to work like it does may not be a good expectation. Both unyt and astropy implement ndarray subclasses.

It might also help to have mock unit libraries (or even directly testing extant unit libraries) in matplotlib’s test suite as this kind of change seems to happen relatively often and you’re not depending on downstream libraries to alert you to breakage.

anntzer · 2021-11-18T16:07:16Z

The case pointed out by @neutrinoceros is already (should?) only done when the input is not a ndarray or subclass thereof.

neutrinoceros · 2021-11-18T16:18:25Z

Correct, but the way it's implemented currently behaves weirdly when the input is a tuple of arrays

anntzer · 2021-11-18T16:39:01Z

Ah, fair enough.

jklymak · 2021-11-18T17:46:58Z

It might also help to have mock unit libraries (or even directly testing extant unit libraries) in matplotlib’s test suite as this kind of change seems to happen relatively often and you’re not depending on downstream libraries to alert you to breakage.

We do have a mock unit library. If downstream libraries want to add new ones (and tests) I'm sure we would entertain that. I'm not sure we will accept testing every possible downstream library that may have decided to add units. OTOH we should specify exactly what we will accept in terms of units so downstream libraries have consistent rules. This has been a problem for years, and no one has fixed it, so we play whack-a-mole. Hopefully there is light at the end of the tunnel with the upcoming NASA grant perhaps supporting a specification and interface overhaul. Please

neutrinoceros · 2021-11-18T17:53:20Z

Well unyt isn't new to this game, I think it's maybe the third biggest player after pint and astropy, but clearly we've missed an opportunity to catch this much earlier by testing only against final releases, so I'll try to fix this on our end to try and improve the situation for everyone in the future.

ngoldbaum · 2021-11-18T18:02:57Z

Ah then I guess a concrete improvement would be a test that directly exercises the tuple of unit arrays case for errorbar. Sorry for the off-the-cuff peanut gallery advice.

anntzer · 2021-11-18T19:18:50Z

Going back to the specific problem here: is there a way to cast a 2-tuple of unyt arrays into a (2, N) unyt array? (... that is not unyt-specific...)

ngoldbaum · 2021-11-18T19:46:17Z

@anntzer, perhaps something like this?

In [1]: import unyt as u

In [2]: d = ([1, 2]*u.g, [1, 2]*u.g)

In [3]: type(d[0])(d)
Out[3]:
unyt_array([[1, 2],
            [1, 2]], 'g')

In [4]: import astropy.units as u

In [5]: d = ([1, 2]*u.g, [1, 2]*u.g)

In [6]: type(d[0])(d)
Out[6]:
<Quantity [[1., 2.],
           [1., 2.]] g>

anntzer · 2021-11-18T20:03:42Z

Ah, thanks for the suggestion.

neutrinoceros · 2021-11-18T21:47:09Z

Note that this doesn't work with two unyt arrays unless they have the exact same unit (not just coercible ones), but other than that this was also my best guess, and it's probably good enough indeed.

neutrinoceros · 2021-11-19T06:48:19Z

Update: @ngoldbaum has a working patch for the problem I mentioned in my previous reply here yt-project/unyt#211, so now I have no reserves left to express on this approach

jklymak · 2021-11-19T07:21:43Z

Going back to the specific problem here: is there a way to cast a 2-tuple of unyt arrays into a (2, N) unyt array? (... that is not unyt-specific...)

Do we need the errors to be a 2, N array, versus two N-arrays? I assume the unyt 1-D arrays cast, so isn't the solution from our side to just have errlo and errhi?

anntzer · 2021-11-19T08:37:30Z

Technically, the docs state that yerr can be a (2, N) array-like which I think means "can be cast to a (2, N) array" (and it should be possible to cast a tuple[1d_unyt_array, 1d_unyt_array] into that). Trying to unpack immediately into errlo, errhi is probably also possible, but just requires much more wrangling than a straightforward np.broadcast_to(err, (2, N))...

closes matplotlib#21669

neutrinoceros mentioned this issue Nov 18, 2021

Regression in matplotlib interface with matplotlib 3.5 yt-project/unyt#208

Closed

tacaswell added this to the v3.5.1 milestone Nov 18, 2021

dstansby added the topic: units and array ducktypes label Nov 18, 2021

tacaswell added the Release critical For bugs that make the library unusable (segfaults, incorrect plots, etc) and major regressions. label Dec 1, 2021

tacaswell added a commit to tacaswell/matplotlib that referenced this issue Dec 8, 2021

FIX: be more careful about coercing unit-full containers to ndarray

783c822

closes matplotlib#21669

tacaswell mentioned this issue Dec 8, 2021

FIX: be more careful about coercing unit-full containers to ndarray #21884

Merged

6 tasks

tacaswell added a commit to tacaswell/matplotlib that referenced this issue Dec 8, 2021

FIX: be more careful about coercing unit-full containers to ndarray

2024498

closes matplotlib#21669

QuLogic closed this as completed in #21884 Dec 10, 2021

Uh oh!

[Bug]: Matplotlib 3.5 breaks unyt integration of error bars #21669

[Bug]: Matplotlib 3.5 breaks unyt integration of error bars #21669

Comments

neutrinoceros commented Nov 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bug summary

Code for reproduction

Actual outcome

Expected outcome

Additional information

Operating system

Matplotlib Version

Matplotlib Backend

Python version

Jupyter version

Installation

neutrinoceros commented Nov 18, 2021

Uh oh!

neutrinoceros commented Nov 18, 2021

Uh oh!

neutrinoceros commented Nov 18, 2021

Uh oh!

anntzer commented Nov 18, 2021

Uh oh!

ngoldbaum commented Nov 18, 2021

Uh oh!

anntzer commented Nov 18, 2021

Uh oh!

neutrinoceros commented Nov 18, 2021

Uh oh!

anntzer commented Nov 18, 2021

Uh oh!

jklymak commented Nov 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

neutrinoceros commented Nov 18, 2021

Uh oh!

ngoldbaum commented Nov 18, 2021

Uh oh!

anntzer commented Nov 18, 2021

Uh oh!

ngoldbaum commented Nov 18, 2021

Uh oh!

anntzer commented Nov 18, 2021

Uh oh!

neutrinoceros commented Nov 18, 2021

Uh oh!

neutrinoceros commented Nov 19, 2021

Uh oh!

jklymak commented Nov 19, 2021

Uh oh!

anntzer commented Nov 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

neutrinoceros commented Nov 18, 2021 •

edited

Loading

jklymak commented Nov 18, 2021 •

edited

Loading

anntzer commented Nov 19, 2021 •

edited

Loading