At our root we are a numpy based library. If you can't cleanly convert your data to a numpy numerical array with np.asanyarray we probably can't plot it.

dd = df.astype('Int64')
print(np.asanyarray(dd).dtype)

returns an object array, which we obviously cannot pcolormesh because each element could be anything.

I would say this is an upstream issue, and pandas and numpy need to work out what they want to do here, versus us adding a conversion shim.

I think the root issue here is that pd.NA cannot be cast to float:

float(pd.NA)

TypeError: float() argument must be a string or a real number, not 'NAType'

I do find this a little surprising; you might expect float(pd.NA) -> np.nan. And in fact, if you do df.astype("Int64").astype(float) it round trips back to np.nan. So I think it probably is fair to call this upstream of matplotlib too :/

This would also require changing

to

c = np.asanyarray(args[0], dtype=float)

Together with

In [1]: float(pd.NA)
Out[1]: nan

it looks like this solves the issue

Would that be acceptable?

The reason this is necessary is that a Series of type Int64, after to_numpy(), currently becomes of dtype object regardless of whether it contains missing values, see this comment

I guess this is OK? It will mean that an integer array ax.pcolormesh([[0, 1], [2, 3]]) will get converted to float earlier in the pipeline, but I assume that is OK. I think other object arrays will still fail to convert.

Someone will have to go through and do this for all the methods that take a 2-D array (pcolormesh, imshow, contour, contourf, streamline all pop to mind, but likely more).

I think its a bit stubborn to not just convert the array to float, but I can see the idea that you want to preserve the ints as ints.

it looks like this solves the issue

I don't think so?

df = pd.DataFrame([[1, 2], [3, pd.NA]]).astype("Int64")
np.asanyarray(df, float)

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In [72], line 2
      1 df = pd.DataFrame([[1, 2], [3, pd.NA]]).astype("Int64")
----> 2 np.asanyarray(df, float)

File ~/miniconda/envs/py310/lib/python3.10/site-packages/pandas/core/generic.py:2069, in NDFrame.__array__(self, dtype)
   2068 def __array__(self, dtype: npt.DTypeLike | None = None) -> np.ndarray:
-> 2069     return np.asarray(self._values, dtype=dtype)

TypeError: float() argument must be a string or a real number, not 'NAType'

@mwaskom sorry I meant with the float(pd.NA) returning nan change as well (I was trying this out here pandas-dev/pandas#48885)

I think its a bit stubborn to not just convert the array to float

Yeah TBH I'm not keen on the current solution of to_numpy() converting everything to object by default. I've opened pandas-dev/pandas#48891 to discuss

Oh, sorry, I somehow thought it worked straight as np.asanyarray(df.astype('Int64'), dtype=np.float64) (I thought I'd tested it, but apparently my coffee hadn't kicked in). If that doesn't work, I'm not clear what you mean by

the float(pd.NA) returning nan change as well

I meant changing pandas so that float(pd.NA) returns np.nan

If I make that change in pandas, and also change

c = np.asanyarray(args[0], dtype=float)

in

Shouldn't this be handled via the unit code?

I don't think we pass any mappable data through a units machinery, despite some attempts/desire to do so. I guess if this was x or y data it could be handled via units, but I'm not aware of special units for pandas objects.

oh, fair point I missed that this was the color channel not the x/y 🐑

There are handler registered someplace (I think by pandas) to correctly handle their datetime types, I think it would make sense to also handle the nullable integers the same way (we treat them as a "unit" that needs casting to floats somehow).

Agreed. OTOH asanyarray should work and all that is being suggested here is that we further cast to float, which makes sense to me. It may actually catch user type errors sooner in the pipeline.

fair, but explicit type casts is a bit of a code-smell to me (which may be just be due to the persistent issues with the unit code which are not currently used here).

explicit type casts is a bit of a code-smell

Hi @tacaswell - would love your thoughts on pandas-dev/pandas#48891 if possible

In the case of e.g. pd.Series([1,2,3], dtype='Int64').to_numpy(), then it could be transformed to int64 and there'd be no need to type cast.
On the other hand, pd.Series([1,2,pd.NA], dtype='Int64').to_numpy() can't get converted to int64, and there's a desire in pandas to avoid value-dependent behaviour. So, this would raise, unless the user specifies something like pd.Series([1,2,pd.NA], dtype='Int64').to_numpy('int64', na_value=-1) or pd.Series([1,2,pd.NA], dtype='Int64').to_numpy('float64').
If you didn't want to cast by default in matplotlib, I think that might have to look like

try:
    c = np.asanyarray(args[0])
except ValueError:
    c = np.asanyarray(args[0], dtype='float64')

Could that be acceptable?

Well I'd argue the code smell here (had to look that phrase up) is that pandas didn't push their NA concept upstream to numpy.

From a practical point of view I was going to claim that we implicitly cast to float pretty early here. However, that is not true - we have BoundaryNorm that maps to integers. So we have another example where things have developed too many paths to easily infer behaviour. In this case I think our best bet is to short term tell folks to explicitly cast their data to something that works.

Longterm, I think the idea of unit conversion for mapables may be ok, but maybe via passing a converter in explicitly and/or allowing norms to handle the unit conversion. The problem as always is if the units are triggered via the container object or the type of elements in an array.

Do we have any other methods that can succesfully plot this data type? (e.g. plot, scatter)