You'll get more help w/ this if you minimize it and get rid of extra stuff like mpld3 (whatever that is). Its hard for me to tell if the issue is matplotlib, or that pd.to_datetime is not returning what you think it is...

#9779 is about the plot function. Here you use the bar function.

A code for reproduction seems to be the following:

import matplotlib.pyplot as plt
import numpy as np
from datetime import datetime
import pandas as pd

dates = [datetime(2018,7,i) for i in range(1,15)]
values = np.cumsum(np.random.rand(len(dates)))

df = pd.DataFrame({"dates":dates, "values" : values})
df["dates"] = pd.to_datetime(df["dates"])

plt.bar(df["dates"],  df["values"].values)

plt.show()

This fails with the same error TypeError: ufunc subtract cannot use operands with types dtype('<M8[ns]') and dtype('float64').

The solution (for now) is to use the numpy array provided by .values instead of the pandas.Series.

plt.bar(df["dates"].values,  df["values"].values)

works as expected.

But I totally agree that it is somehow confusing that

plt.plot(df["dates"], df["values"].values)

works fine, while

plt.bar(df["dates"], df["values"].values)
# or
plt.scatter(df["dates"], df["values"].values)

does not. Essentially the same issue, but for scatter, is raised in #11391.

Okay....

plot eventually does:

print('pre x', x, type(x), type(x[0]))
if x.ndim == 1:
    x = x[:, np.newaxis]
print('x after', x, type(x), type(x[0, 0]))

and this converts x from a pandas time series to a numpy datetime64 array:

pre x 0   2018-07-01
1   2018-07-02
2   2018-07-03
3   2018-07-04
Name: dates, dtype: datetime64[ns] <class 'pandas.core.series.Series'> <class 'pandas._libs.tslib.Timestamp'>

final x [['2018-07-01T00:00:00.000000000']
 ['2018-07-02T00:00:00.000000000']
 ['2018-07-03T00:00:00.000000000']
 ['2018-07-04T00:00:00.000000000']] <class 'numpy.ndarray'> <class 'numpy.datetime64'>

So for plot the pandas time series actually gets converted to numpy.datetime64 and we use our converter.

For scatter, bar, and everything else, we use panda's converter, which doesn't appear to convert to matplotlib datetimes (at least not natively).

So, not sure what the right thing to do here is. We could easily pre-process x and y the way plot does, but that seems risky. In fact I'd kind of argue plot is doing the wrong thing and should not be automatically converting to a np array so early. But I don't know why the processing is done that way.

Checked blame, and the numpy typecast came into plot before the great split of axes.py, so its pretty hard to know where it came from...

Wow it's kind of depressing to see what I always considered as a great feature turning out to be pure coincidence.

In any case, this "feature" is used a lot in the wild, so even though I somehow agree that the conversion may be wrong in fact, I would argue that one should not break many people's codes by removing it.

In general, I could imagine that adding a couple of pandas examples to the matplotlib gallery would be really helpful; but the problem is that due to pandas not being a dependency, those couldn't be run through sphinx-gallery, producing any output figures.

Thanks,

The solution proposed by Elan Ernest indeed converted dates correctly but as a side effect I got got weird behavior with the bars.

Before (not using .values, not doing df["dates"] = pd.to_datetime(df["dates"])

After

However this seems to be a problem out of the scope of this topic.

Does the workaround proposed by Elan Ernest deserves an entry in https://matplotlib.org/gallery/recipes/common_date_problems.html? I was trying to find a solution for this problem there before opening this issue

Thank you very much again

Anyway I would appreciate if someone has a tip for solving both problems as I used to have this problem before and this was one of the reasons I stopped using values :)

@ImportanceOfBeingErnest - I guess I think the behaviour of plot versus scatter (for instance) was just masking the fact that the Pandas date converter isn't doing its job. When it gets called, it needs to take the pandas time series and turn it into a numpy array of Matplotlib datenums. It really doesn't seem to be doing that, so maybe someone should open a bug report there (I'm hesitant to, just because I don't use pandas)

If that is fixed, then maybe we can talk about making sure plot preserves its x-data in the original form until convert is called, rather than casting to a numpy data type so early.

Getting this right should all be thought about as part of the units MEP. ping @dopplershift @tacaswell

Does the workaround proposed by Elan Ernest deserves an entry in https://matplotlib.org/gallery/recipes/common_date_problems.html? I was trying to find a solution for this problem there before opening this issue

@danieljmr you are using pandas... As you can probably tell, pandas is not a native part of matplotlib; its a little hard for us to specify "common problems" for all the downstream packages (though of course we can/should try and smooth out any problems if we can help on our end).

@danieljmr you are using pandas... As you can probably tell, pandas is not a native part of matplotlib; its a little hard for us to specify "common problems" for all the downstream packages (though of course we can/should try and smooth out any problems if we can help on our end).

Pandas and tabular data is a big use case for our users. Even though we do not directly depend on pandas, we need to make sure we work well with it. Pandas is defiantly a special case of our down-stream consumers.

In any case, this "feature" is used a lot in the wild, so even though I somehow agree that the conversion may be wrong in fact, I would argue that one should not break many people's codes by removing it.

I do not understand the details of this bug report, but in general I agree. There is a difference between the "official public" API and the "what users rely on so it is defacto public" API which is one of the biggest challenges of long-term maintenance of Matplotlib.

I agree it’d be nice for us to work well w pandas; I guess the question is how much do we do natively and how much do we ask them to do. On this bug, which is pretty long standing from what I can tell, I think the ball is in pandas court because it is their converter that isn’t converting. That plot works is because we never give their converter the chance to work in plot.

This needs to be a collaborative effort with pandas (users don't care whos problem it is, they just want it to work!). I have added this to the to-do list for the scipy sprints. Hopefully we can got a mpl dev and a pandas dev to sit together and sort out where this is going sideways.

attn @TomAugspurger

I won't be available during sprints, but if people need pandas questions answered they can post here and I'll try to answer when I'm back online.

@danieljmr To solve the remaining problem of your bars being too wide, you need to set a useful width for them. The units of matplotlib datetime plots are days, so the width needs to be set in fractions of a day. E.g. if you wanted to have hourly wide bars, you'd set width=1./24, if you wanted minute wide bars width=1./24/60 etc. You may also use an array for different width bars.

@TomAugspurger There isn't really any question to be answered here; it's more that it needs to be discussed where matplotlib and pandas can meet when it comes to datetime plotting.
I'd say ideally all of the following should produce the same plot

• plt.scatter(df.x.values,df.y.values)
• plt.scatter(df.x,df.y)
• plt.scatter(x="x", y="y", data=df)
• df.plot.scatter(x="x", y="y")

even with datetimes involved.
In principle the same with bars, although this is a bit harder, because pandas treats those as categorical per se (which is also a major point of confusion judging from the numerous stackoverflow questions on that topic - half the people wonder why matplotlib bar plots are numeric, the other half wonders why pandas bar plots are categorical - on all levels of understanding).
One point to start would be to discuss in whose responsibility the datetime conversion lies when it comes to plt.bar(df["dates"], df["values"].values) failing as in this issue.

Thank you Elan Ernest 🥇 ,

Passing width=0.00001 (seconds scale) to axes.bar worked! Now I got the dates converted and my bars were nicely rendered.

Thank you contributors for your support.
pandas and matplotlib is a powerful combination. A good interaction between both libraries will be beneficial for both ends... and of course for us... the users

Gentlemen,

At this point, having all my questions answered, I will proceed with the closure.

Great support!

This still identifies a bug that we need to talk about.

Note that if we deregister the pandas converters this works fine:

import matplotlib.pyplot as plt
import numpy as np
from datetime import datetime
import pandas as pd

pd.plotting.deregister_matplotlib_converters()

dates = [datetime(2018,7,i) for i in range(1, 5)]
values = np.cumsum(np.random.rand(len(dates)))

df = pd.DataFrame({"dates":dates, "values" : values})

plt.plot(df["dates"],  df["values"])
plt.scatter(df["dates"],  df["values"])

plt.show()

Sorry, I will close this in favour of #11391 where the use cross section is more complete....