Different result, slower runtime of heatmap between 2.0.0 and 2.0.1 #8947

NickG123 · 2017-07-27T20:31:33Z

Bug report

Bug summary

I am using Matplotlib with Basemap to create heatmaps of data.
A change introduced between version 2.0.0 and 2.0.1 caused my code to:

Produce a different result
Run significantly slower

The problem still seems to exist in 2.0.2.

I also ran some profiling to have a look at the differences, and the increase in time seems to be in the resample function (from 0.473s total time to 19.042s total time)

Code for reproduction

import csv
import time

import numpy as np
import matplotlib
import matplotlib.pyplot as plt
import matplotlib.cm as cm
from mpl_toolkits.basemap import Basemap


def create_basemap():
    m = Basemap(projection='cyl', llcrnrlat=-90, urcrnrlat=90,
                llcrnrlon=-180, urcrnrlon=180, resolution='c')
    m.drawmapboundary(fill_color='#b4d0d0')
    m.drawcoastlines(linewidth=0.25, color="#ffffff")
    m.fillcontinents(color="grey", lake_color='#b4d0d0')

    m.drawparallels(np.arange(-90., 91., 30.), linewidth="0.25", color="#333333")
    m.drawmeridians(np.arange(-180., 181., 60.), linewidth="0.25", color="#555555")
    return m


def create_heatmap(positions):
    heatmap = np.zeros((180, 360))
    extents = [[-90, 90], [-180, 180]]

    lats, lons = map(list, zip(*positions))

    subheatmap, xedges, yedges = np.histogram2d(lats, lons, bins=[180, 360], range=extents)
    extent = [xedges[0], xedges[-1], yedges[0], yedges[-1]]
    heatmap = np.add(heatmap, subheatmap)

    fig = plt.figure()
    m = create_basemap()

    cmap = cm.jet
    cmap.set_bad(alpha=0.0)

    im = m.imshow(heatmap, cmap=cmap, interpolation='bicubic', extent=extent, origin='lower', alpha=1.0, norm=matplotlib.colors.LogNorm(), vmin=1, vmax=2, zorder=100)
    cb = fig.colorbar(im, shrink=0.5, format="%d")

    plt.title("Test")
    plt.savefig("test.png", dpi=500, bbox_inches='tight', pad_inches=0.1)
    plt.close()


positions = [(50, i) for i in range(50)]
start = time.time()
create_heatmap(positions)
print "Runtime: {0}".format(time.time() - start)

Version 2.0.0 outcome

Runtime: 2.31200003624

python -m cProfile --sort=tottime script.py:

         667764 function calls (659117 primitive calls) in 3.555 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        2    0.702    0.351    0.702    0.351 {matplotlib._png.write_png}
        2    0.473    0.237    0.473    0.237 {matplotlib._image.resample}
        1    0.138    0.138    0.138    0.138 {_tkinter.create}
      890    0.117    0.000    0.117    0.000 {method 'is_valid' of '_geoslib.BaseGeometry' objects}
       15    0.106    0.007    0.632    0.042 __init__.py:1(<module>)
        1    0.090    0.090    0.135    0.135 __init__.py:14(<module>)
      296    0.083    0.000    0.083    0.000 {method 'intersection' of '_geoslib.BaseGeometry' objects}
        2    0.068    0.034    0.681    0.340 image.py:275(_make_image)
    11413    0.055    0.000    0.058    0.000 {numpy.core.multiarray.array}
      300    0.050    0.000    0.072    0.000 {method 'draw_path' of 'matplotlib.backends._backend_agg.RendererAgg' objects}
...

Version 2.0.1 outcome

Runtime: 21.0169999599

python -m cProfile --sort=tottime script.py:

         1763720 function calls (1754827 primitive calls) in 23.725 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        4   19.042    4.761   19.043    4.761 {matplotlib._image.resample}
        2    0.735    0.367    0.735    0.367 {matplotlib._png.write_png}
        2    0.609    0.304    1.377    0.688 font_manager.py:558(createFontList)
       60    0.138    0.002    0.336    0.006 afm.py:181(_parse_char_metrics)
        1    0.135    0.135    0.135    0.135 {_tkinter.create}
        2    0.133    0.067   19.309    9.655 image.py:275(_make_image)
      890    0.115    0.000    0.115    0.000 {method 'is_valid' of '_geoslib.BaseGeometry' objects}
        1    0.107    0.107    0.159    0.159 __init__.py:14(<module>)
       15    0.106    0.007    0.637    0.042 __init__.py:1(<module>)
      559    0.104    0.000    0.104    0.000 {method 'get_sfnt' of 'matplotlib.ft2font.FT2Font' objects}

Matplotlib version

Operating System: Windows 10
Matplotlib Version: 2.0.0/2.0.1 (from pip)
Python Version: 2.7.13
Jupyter Version (if applicable): N/A
Other Libraries: basemap 1.1.0, numpy 1.13.1

The text was updated successfully, but these errors were encountered:

afvincent · 2017-10-17T21:39:34Z

@NickG123 Could you test if the problem is still there with Matplotlib 2.1?

On my own workstation (Fedora 26), with Python 3.6.3 and Matplotlib 2.1 (both from conda), the results are more similar to yours with Matplotlib 2.0.0 than those with Matplotlib 2.0.1:

Runtime: 2.199585437774658
         744304 function calls (733696 primitive calls) in 2.669 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        2    0.703    0.352    0.703    0.352 {built-in method matplotlib._png.write_png}
        6    0.520    0.087    0.521    0.087 {built-in method matplotlib._image.resample}
      890    0.075    0.000    0.075    0.000 {method 'is_valid' of '_geoslib.BaseGeometry' objects}
       46    0.067    0.001    0.067    0.001 {built-in method numpy.core.multiarray.copyto}
    10851    0.066    0.000    0.067    0.000 {built-in method numpy.core.multiarray.array}
        6    0.061    0.010    0.113    0.019 colors.py:985(__call__)
      296    0.055    0.000    0.055    0.000 {method 'intersection' of '_geoslib.BaseGeometry' objects}
    46/45    0.053    0.001    0.056    0.001 {built-in method _imp.create_dynamic}
        2    0.053    0.026    0.082    0.041 __init__.py:14(<module>)
      447    0.051    0.000    0.076    0.000 {method 'draw_path' of 'matplotlib.backends._backend_agg.RendererAgg' objects}

afvincent · 2017-10-17T21:45:21Z

I did not perform a proper git bisect but this might have been fixed by @tacaswell in #8966 (more precisely 12c27f3)

NickG123 · 2017-10-18T13:03:57Z

@afvincent Thanks for having a look.

I do see a significant improvement in performance with Matplotlib 2.1, back down to roughly the same speed as 2.0.0.

However, the resulting image is very strage:

I do get a couple of warnings that might be related, but they are coming out of the Basemap library, not my code:

C:\Python27\lib\site-packages\mpl_toolkits\basemap\__init__.py:1704: MatplotlibDeprecationWarning: The axesPatch function was deprecated in version 2.1. Use Axes.patch instead.
  limb = ax.axesPatch
C:\Python27\lib\site-packages\mpl_toolkits\basemap\__init__.py:1707: MatplotlibDeprecationWarning: The axesPatch function was deprecated in version 2.1. Use Axes.patch instead.
  if limb is not ax.axesPatch:

WeatherGod · 2017-10-18T13:42:14Z

The warnings can be fixed by updating basemap using conda-forge, or installing from source.

…

On Wed, Oct 18, 2017 at 9:03 AM, Nick Gerow ***@***.***> wrote: @afvincent <https://github.com/afvincent> Thanks for having a look. I do see a significant improvement in performance with Matplotlib 2.1, back down to roughly the same speed as 2.0.0. However, the resulting image is very strage: [image: test] <https://user-images.githubusercontent.com/8438951/31719927-0d7bc45a-b3e3-11e7-96af-259aee4730dd.png> I do get a couple of warnings that might be related, but they are coming out of the Basemap library, not my code: C:\Python27\lib\site-packages\mpl_toolkits\basemap\__init__.py:1704: MatplotlibDeprecationWarning: The axesPatch function was deprecated in version 2.1. Use Axes.patch instead. limb = ax.axesPatch C:\Python27\lib\site-packages\mpl_toolkits\basemap\__init__.py:1707: MatplotlibDeprecationWarning: The axesPatch function was deprecated in version 2.1. Use Axes.patch instead. if limb is not ax.axesPatch: — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#8947 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AARy-FbmOvg9QJWTDcZh6fgtlhxtqF9zks5stfc_gaJpZM4OlzQ-> .

WeatherGod · 2017-10-18T13:46:45Z

Also, I find it interesting that your 2.0.0 image has dashed grid lines, but 2.0.1 has solid grid lines... Can you try some different interpolations? I wonder if this is the result of some of the masking work Thomas was doing?

…

On Wed, Oct 18, 2017 at 9:41 AM, Benjamin Root ***@***.***> wrote: The warnings can be fixed by updating basemap using conda-forge, or installing from source. On Wed, Oct 18, 2017 at 9:03 AM, Nick Gerow ***@***.***> wrote: > @afvincent <https://github.com/afvincent> Thanks for having a look. > > I do see a significant improvement in performance with Matplotlib 2.1, > back down to roughly the same speed as 2.0.0. > > However, the resulting image is very strage: > [image: test] > <https://user-images.githubusercontent.com/8438951/31719927-0d7bc45a-b3e3-11e7-96af-259aee4730dd.png> > > I do get a couple of warnings that might be related, but they are coming > out of the Basemap library, not my code: > > C:\Python27\lib\site-packages\mpl_toolkits\basemap\__init__.py:1704: MatplotlibDeprecationWarning: The axesPatch function was deprecated in version 2.1. Use Axes.patch instead. > limb = ax.axesPatch > C:\Python27\lib\site-packages\mpl_toolkits\basemap\__init__.py:1707: MatplotlibDeprecationWarning: The axesPatch function was deprecated in version 2.1. Use Axes.patch instead. > if limb is not ax.axesPatch: > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#8947 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AARy-FbmOvg9QJWTDcZh6fgtlhxtqF9zks5stfc_gaJpZM4OlzQ-> > . >

tacaswell · 2017-10-18T14:38:04Z

The fuzz over the oceans is almost certainly the same as the fuzz we saw at the edges of #8966 Trying with a lower-order interpolation might help.

NickG123 · 2017-10-18T17:47:18Z

Looks like that's the problem
Both interpolation='none' and interpolation='nearest' generate the expected image (the same one as version 2.0.0), while all other options produce varying degrees of "fuzz" across the whole map.
Thanks,
Nick

tacaswell · 2018-08-11T21:07:34Z

@NickG123 can you reproduce this with 2.2.3?

NickG123 · 2018-08-13T12:34:44Z

Hi @tacaswell, It looks like v2.2.3 still produces the same result as with 2.1 above.

However, my use case is satisfied by using a lower order interpolation, so I guess it is up to you how much of a priority this is.

Thanks,
Nick

jklymak · 2019-02-12T01:03:02Z

I think we solved the original problem, so I'm closing, but feel free to re-open...

tacaswell added this to the v2.1.1 milestone Oct 18, 2017

tacaswell added the Release critical For bugs that make the library unusable (segfaults, incorrect plots, etc) and major regressions. label Oct 18, 2017

tacaswell self-assigned this Oct 18, 2017

tacaswell modified the milestones: v2.1.1, v2.2 Dec 6, 2017

tacaswell mentioned this issue Dec 8, 2017

imshow interpolation generating artifacts #9961

Closed

tacaswell modified the milestones: v2.2, v3.0 Feb 5, 2018

tacaswell modified the milestones: v3.0, v3.1 Aug 13, 2018

tacaswell removed the Release critical For bugs that make the library unusable (segfaults, incorrect plots, etc) and major regressions. label Aug 13, 2018

jklymak closed this as completed Feb 12, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Different result, slower runtime of heatmap between 2.0.0 and 2.0.1 #8947

Different result, slower runtime of heatmap between 2.0.0 and 2.0.1 #8947

NickG123 commented Jul 27, 2017

afvincent commented Oct 17, 2017

Uh oh!

afvincent commented Oct 17, 2017

Uh oh!

NickG123 commented Oct 18, 2017

Uh oh!

WeatherGod commented Oct 18, 2017 via email

Uh oh!

WeatherGod commented Oct 18, 2017 via email

Uh oh!

tacaswell commented Oct 18, 2017

Uh oh!

NickG123 commented Oct 18, 2017

Uh oh!

tacaswell commented Aug 11, 2018

Uh oh!

NickG123 commented Aug 13, 2018

Uh oh!

jklymak commented Feb 12, 2019

Uh oh!

Uh oh!

Different result, slower runtime of heatmap between 2.0.0 and 2.0.1 #8947

Different result, slower runtime of heatmap between 2.0.0 and 2.0.1 #8947

Comments

NickG123 commented Jul 27, 2017

Bug report

afvincent commented Oct 17, 2017

Uh oh!

afvincent commented Oct 17, 2017

Uh oh!

NickG123 commented Oct 18, 2017

Uh oh!

WeatherGod commented Oct 18, 2017 via email

Uh oh!

WeatherGod commented Oct 18, 2017 via email

Uh oh!

tacaswell commented Oct 18, 2017

Uh oh!

NickG123 commented Oct 18, 2017

Uh oh!

tacaswell commented Aug 11, 2018

Uh oh!

NickG123 commented Aug 13, 2018

Uh oh!

jklymak commented Feb 12, 2019

Uh oh!