Cannot plot fully masked array against datetimes #11337

logdog · 2018-05-29T19:05:23Z

Bug report

Bug summary
Cannot plot a fully masked array agaisnt an array of datetimes. A ValueError is raised for a seemingly unrelated reason. Surprisingly, matplotlib has no problem plotting a fully masked array by itself.

Code for reproduction

import matplotlib.pyplot as plt
from datetime import datetime
import numpy as np

x = np.array([
    datetime(2017, 1, 1), 
    datetime(2017, 1, 2), 
    datetime(2017, 1, 3),
    datetime(2017, 1, 4),
    datetime(2017, 1, 5)
])

y = np.array([1,2,3,4,5])
m = np.ma.masked_greater(y, 0)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(x, m)
plt.show()

Actual outcome

[super big traceback, only showing the most recent to save space...]
/usr/lib64/python3.5/dist-packages/matplotlib/dates.py in _from_ordinalf(x, tz)
    258 
    259     ix = int(x)
--> 260     dt = datetime.datetime.fromordinal(ix).replace(tzinfo=UTC)
    261 
    262     remainder = float(x) - ix
ValueError: ordinal must be >= 1

Expected outcome
An empty graph. If you comment out x and don't plot against it, then matplotlib has no problem displaying an empty graph. It is only when you plot against a datetime array this is an issue.

Matplotlib version

Operating system: GNU/Linux
Matplotlib version: 2.1.1
Matplotlib backend (print(matplotlib.get_backend())): module://ipykernel.pylab.backend_inline
Python version: 3.5.5
Jupyter version (if applicable): 4.3.0
Other libraries: numpy, datetime

The text was updated successfully, but these errors were encountered:

logdog · 2018-05-29T19:10:24Z

Here is the full traceback if it is needed:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/usr/local/lib/python3.5/site-packages/IPython/core/formatters.py in __call__(self, obj)
    330                 pass
    331             else:
--> 332                 return printer(obj)
    333             # Finally look for special method names
    334             method = get_real_method(obj, self.print_method)

/usr/local/lib/python3.5/site-packages/IPython/core/pylabtools.py in <lambda>(fig)
    235 
    236     if 'png' in formats:
--> 237         png_formatter.for_type(Figure, lambda fig: print_figure(fig, 'png', **kwargs))
    238     if 'retina' in formats or 'png2x' in formats:
    239         png_formatter.for_type(Figure, lambda fig: retina_figure(fig, **kwargs))

/usr/local/lib/python3.5/site-packages/IPython/core/pylabtools.py in print_figure(fig, fmt, bbox_inches, **kwargs)
    119 
    120     bytes_io = BytesIO()
--> 121     fig.canvas.print_figure(bytes_io, **kw)
    122     data = bytes_io.getvalue()
    123     if fmt == 'svg':

/usr/lib64/python3.5/dist-packages/matplotlib/backend_bases.py in print_figure(self, filename, dpi, facecolor, edgecolor, orientation, format, **kwargs)
   2214                     orientation=orientation,
   2215                     dryrun=True,
-> 2216                     **kwargs)
   2217                 renderer = self.figure._cachedRenderer
   2218                 bbox_inches = self.figure.get_tightbbox(renderer)

/usr/lib64/python3.5/dist-packages/matplotlib/backends/backend_agg.py in print_png(self, filename_or_obj, *args, **kwargs)
    505 
    506     def print_png(self, filename_or_obj, *args, **kwargs):
--> 507         FigureCanvasAgg.draw(self)
    508         renderer = self.get_renderer()
    509         original_dpi = renderer.dpi

/usr/lib64/python3.5/dist-packages/matplotlib/backends/backend_agg.py in draw(self)
    428             if toolbar:
    429                 toolbar.set_cursor(cursors.WAIT)
--> 430             self.figure.draw(self.renderer)
    431         finally:
    432             if toolbar:

/usr/lib64/python3.5/dist-packages/matplotlib/artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
     53                 renderer.start_filter()
     54 
---> 55             return draw(artist, renderer, *args, **kwargs)
     56         finally:
     57             if artist.get_agg_filter() is not None:

/usr/lib64/python3.5/dist-packages/matplotlib/figure.py in draw(self, renderer)
   1297 
   1298             mimage._draw_list_compositing_images(
-> 1299                 renderer, self, artists, self.suppressComposite)
   1300 
   1301             renderer.close_group('figure')

/usr/lib64/python3.5/dist-packages/matplotlib/image.py in _draw_list_compositing_images(renderer, parent, artists, suppress_composite)
    136     if not_composite or not has_images:
    137         for a in artists:
--> 138             a.draw(renderer)
    139     else:
    140         # Composite any adjacent images together

/usr/lib64/python3.5/dist-packages/matplotlib/artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
     53                 renderer.start_filter()
     54 
---> 55             return draw(artist, renderer, *args, **kwargs)
     56         finally:
     57             if artist.get_agg_filter() is not None:

/usr/lib64/python3.5/dist-packages/matplotlib/axes/_base.py in draw(self, renderer, inframe)
   2435             renderer.stop_rasterizing()
   2436 
-> 2437         mimage._draw_list_compositing_images(renderer, self, artists)
   2438 
   2439         renderer.close_group('axes')

/usr/lib64/python3.5/dist-packages/matplotlib/image.py in _draw_list_compositing_images(renderer, parent, artists, suppress_composite)
    136     if not_composite or not has_images:
    137         for a in artists:
--> 138             a.draw(renderer)
    139     else:
    140         # Composite any adjacent images together

/usr/lib64/python3.5/dist-packages/matplotlib/artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
     53                 renderer.start_filter()
     54 
---> 55             return draw(artist, renderer, *args, **kwargs)
     56         finally:
     57             if artist.get_agg_filter() is not None:

/usr/lib64/python3.5/dist-packages/matplotlib/axis.py in draw(self, renderer, *args, **kwargs)
   1131         renderer.open_group(__name__)
   1132 
-> 1133         ticks_to_draw = self._update_ticks(renderer)
   1134         ticklabelBoxes, ticklabelBoxes2 = self._get_tick_bboxes(ticks_to_draw,
   1135                                                                 renderer)

/usr/lib64/python3.5/dist-packages/matplotlib/axis.py in _update_ticks(self, renderer)
    972 
    973         interval = self.get_view_interval()
--> 974         tick_tups = list(self.iter_ticks())
    975         if self._smart_bounds and tick_tups:
    976             # handle inverted limits

/usr/lib64/python3.5/dist-packages/matplotlib/axis.py in iter_ticks(self)
    915         Iterate through all of the major and minor ticks.
    916         """
--> 917         majorLocs = self.major.locator()
    918         majorTicks = self.get_major_ticks(len(majorLocs))
    919         self.major.formatter.set_locs(majorLocs)

/usr/lib64/python3.5/dist-packages/matplotlib/dates.py in __call__(self)
   1059     def __call__(self):
   1060         'Return the locations of the ticks'
-> 1061         self.refresh()
   1062         return self._locator()
   1063 

/usr/lib64/python3.5/dist-packages/matplotlib/dates.py in refresh(self)
   1079     def refresh(self):
   1080         'Refresh internal information based on current limits.'
-> 1081         dmin, dmax = self.viewlim_to_dt()
   1082         self._locator = self.get_locator(dmin, dmax)
   1083 

/usr/lib64/python3.5/dist-packages/matplotlib/dates.py in viewlim_to_dt(self)
    837             vmin, vmax = vmax, vmin
    838 
--> 839         return num2date(vmin, self.tz), num2date(vmax, self.tz)
    840 
    841     def _get_unit(self):

/usr/lib64/python3.5/dist-packages/matplotlib/dates.py in num2date(x, tz)
    443         tz = _get_rc_timezone()
    444     if not cbook.iterable(x):
--> 445         return _from_ordinalf(x, tz)
    446     else:
    447         x = np.asarray(x)

/usr/lib64/python3.5/dist-packages/matplotlib/dates.py in _from_ordinalf(x, tz)
    258 
    259     ix = int(x)
--> 260     dt = datetime.datetime.fromordinal(ix).replace(tzinfo=UTC)
    261 
    262     remainder = float(x) - ix

ValueError: ordinal must be >= 1

<matplotlib.figure.Figure at 0x7f50819d3780>

fredrik-1 · 2018-05-30T11:50:45Z

The problem seems to be that a fully masked array is "plotted" around (0, 0) independent of the x values and this doesn't work with datetimes because they should always be larger than 1.

I get the following value error in 2.2.2 that explains more.
ValueError: view limit minimum -0.001 is less than 1 and is an invalid Matplotlib date value. This often happens if you pass a non-datetime value to an axis that has datetime units

ImportanceOfBeingErnest · 2018-05-30T13:16:55Z

I think the problem is more that an artist without any visual representation is attempted to be drawn.

I would say this tracks down to the question of why the two below codes give different xlimits:

plt.gca()
print(plt.gca().get_xlim())  # (0.0, 1.0)

plt.gca()
plt.plot([])
print(plt.gca().get_xlim())  # (-0.055000000000000007, 0.055000000000000007)

In the second case, even though the artist has no extent on the axes, the axes limits are changed. I would argue that this shouldn't be the case - and if fixed, would also solve this issue here automagically.

fredrik-1 · 2018-05-30T13:39:51Z

I believe that is consistent with my thoughts. plt.plot([]) sets axis around 0 and that does not work with a datetime x axis.

I also saw that loglog with fully masked data worked and that the limits where not around zero in that case.

tacaswell · 2018-05-30T13:42:11Z

I think @ImportanceOfBeingErnest diagnosed this right. The bug is someplace in the auto-scale logic for Line2D objects with no valid data defaulting to min/max of 0 (which is then getting expanded to +/- 0.05) rather than being skipped.

logdog · 2018-10-03T21:09:47Z

I just checked that the array was not completely masked with an if statement before I plotted the data. If it was fully masked, it just didn't plot the data and displayed a small graphic saying the data was fully masked for that particular graph.

…

On Sun, Sep 30, 2018 at 7:10 AM Timothy Pulliam ***@***.***> wrote: Is there a work around for this? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#11337 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/APE1VrPK5yBXLp_9sf462nVtKZGPdf7-ks5ugNC8gaJpZM4USDVm> .

jonnyhtw · 2018-10-12T03:16:22Z

I'm also seeing this error using this conda environment...

> cat environment.yml
name: previous_env
channels:
- conda-forge
- defaults
dependencies:
- python=2.7
- iris>=1.12.0, <2
- numpy
- scipy
- nose
- matplotlib
- pep8
- sphinx
- sphinx_rtd_theme
- image-meta-tag >=0.6

jklymak · 2019-03-04T17:24:00Z

Note this will still error if xmin=0. xmin>=1 for num2date to work.

jklymak · 2019-03-04T17:48:52Z

I think that path does the "right" thing, and doesn't update the view limits based on points that won't plot.

I think its possible to argue that we should be able to handle num2date(0) better than we do. Right now we use datetime.datetime.toordinal(0), and that fails because datetime ordinals cannot be less than 0. However, numpy.datetime64 has no such restriction. I don't think its ridiculously hard to rewrite dates.py to make the internal representation of dates datetime64 objects. However, there may b pitfalls. I know @efiring and @pganssle have opinions on date times - maybe they have some thoughts here. But I don't see fundamentally why matplotlib should not be able to plot dates before 0000.

pganssle · 2019-03-04T18:43:59Z

But I don't see fundamentally why matplotlib should not be able to plot dates before 0000.

I can think of many reasons why you'd want to plot very old dates, but it's pretty unusual to want to represent them in a proleptic Gregorian calendar (and even for those when you do want that, it's pretty rare to care about the time portion of it). Most of the functionality you gain by using a datetime is essentially useless or misleading for very old datetimes, but there's no special transition at 1 AD, as you go back in time the datetime class just sort of progressively decays into something of dubious value because it doesn't match contemporary timekeeping systems. I think the main reason datetime itself drops support there is that it's a natural point to drop support so that you can make assumptions about the sign of the components.

So what I will say is that support for out-of-bounds datetime objects should probably be as tightly scoped as possible. There are a number of bugs reported to dateutil and other projects relating to old datetimes, and I don't think I've ever seen an example of one where someone had a specific use case for these things. Most of the bugs come from people using hypothesis in their test suite or situations like this, where some default value doesn't totally fit in with the way datetime is intended to be used. I'd be willing to change my mind if I could identify a set of users who have an actual use for very old proleptic Gregorian datetimes.

In this particular situation, I would want to avoid a solution that too-eagerly converts from datetime to some ordinal number, as I was concerned about in (I think) #12863. I haven't looked at the relevant code paths, but I think maybe either changing the default value to 1970-01-01T00:00:00 or a somewhat more complicated mechanism that uses ordinal dates only when a datetime value can't be represented as datetime would be the right way to go here.

jklymak · 2019-03-04T19:07:53Z

The problem here is the converse - the default limits are 0-1 but the user supplied datetime to the x-axis so the converter is datetime, and then when ticking, it tries to convert 0 to a datetime and that fails.

I guess the correct fix is when the axis first gets registered as datetime its default limits get changed to something other than 0-1.

anntzer · 2019-03-04T19:56:05Z

Hummm... possibly #11337 (comment) is related to #13463 (the comment mentioning that dates Locators likely need view_limits)?

jklymak · 2019-03-04T20:00:33Z

Yeah I was just looking at it... The default view interval gets set by dates, but not used in the ticking.

tacaswell added this to the v3.0 milestone May 30, 2018

tacaswell modified the milestones: v3.0, v3.1 Aug 11, 2018

jklymak added the topic: date handling label Mar 4, 2019

jklymak modified the milestones: v3.1.0, v3.2.0 Mar 4, 2019

jklymak mentioned this issue Mar 4, 2019

FIX: fallback to viewlims if no data #13588

Merged

6 tasks

jklymak modified the milestones: v3.2.0, v3.1.1 Mar 4, 2019

efiring closed this as completed in #13588 Mar 18, 2019

QuLogic modified the milestones: v3.1.1, v3.1.0 Mar 19, 2019

anntzer mentioned this issue Sep 12, 2019

Large memory growth with log scaling and linear ticking #15251

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Cannot plot fully masked array against datetimes #11337

Cannot plot fully masked array against datetimes #11337

logdog commented May 29, 2018

logdog commented May 29, 2018

Uh oh!

fredrik-1 commented May 30, 2018

Uh oh!

ImportanceOfBeingErnest commented May 30, 2018

Uh oh!

fredrik-1 commented May 30, 2018

Uh oh!

tacaswell commented May 30, 2018

Uh oh!

logdog commented Oct 3, 2018 via email

Uh oh!

jonnyhtw commented Oct 12, 2018

Uh oh!

jklymak commented Mar 4, 2019

Uh oh!

jklymak commented Mar 4, 2019

Uh oh!

pganssle commented Mar 4, 2019 •

edited

Loading

Uh oh!

jklymak commented Mar 4, 2019

Uh oh!

anntzer commented Mar 4, 2019

Uh oh!

jklymak commented Mar 4, 2019

Uh oh!

Uh oh!

Cannot plot fully masked array against datetimes #11337

Cannot plot fully masked array against datetimes #11337

Comments

logdog commented May 29, 2018

Bug report

logdog commented May 29, 2018

Uh oh!

fredrik-1 commented May 30, 2018

Uh oh!

ImportanceOfBeingErnest commented May 30, 2018

Uh oh!

fredrik-1 commented May 30, 2018

Uh oh!

tacaswell commented May 30, 2018

Uh oh!

logdog commented Oct 3, 2018 via email

Uh oh!

jonnyhtw commented Oct 12, 2018

Uh oh!

jklymak commented Mar 4, 2019

Uh oh!

jklymak commented Mar 4, 2019

Uh oh!

pganssle commented Mar 4, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jklymak commented Mar 4, 2019

Uh oh!

anntzer commented Mar 4, 2019

Uh oh!

jklymak commented Mar 4, 2019

Uh oh!

pganssle commented Mar 4, 2019 •

edited

Loading