Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Animation of contourf becomes extremely slow #6985

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
erichlf opened this issue Aug 26, 2016 · 27 comments
Closed

Animation of contourf becomes extremely slow #6985

erichlf opened this issue Aug 26, 2016 · 27 comments

Comments

@erichlf
Copy link

erichlf commented Aug 26, 2016

When creating an animation with FuncAnimation and contourf I have run into an issue where the more frames I have the slower things become. This is illustrated in the code below. I print a progress bar, which displays both progress and the time required for the current step. As the animation creation progresses you will see that each step takes longer and longer. I was able to mitigate this by creating a derived class MyFuncAnimation, where I overloaded the function save. In the new version of save I go through and clear all collections associated with self._drawn_artists, after writer.grab_frame, in the following way:
for collection in self._drawn_artists.collections: gca().collections.remove(collection)

I am not sure if there is a better way to do this, but it was the only way I could figure out how to address the issue.

Matplot Version = 1.5.1
Python Version = 3.5.2
Platform = MAC OS 10.11.6

Installed matplotlib through pip

Example:

#!/usr/bin/env python3

import numpy as np
import matplotlib.pyplot as plt
from matplotlib import mlab, cm
from matplotlib.animation import FuncAnimation
import sys
import time

barWidth = 40
n_steps = 100

class ProgressBar():
    '''
    Provides a simple progress bar class
    '''
    def __init__(self, nsteps, width=barWidth):
        self._start = self._stop = time.time()
        self._nsteps = nsteps
        self._width = width
        self._status = ""

    def update(self, step):
        '''
        This function produces and updates a progress bar.
        It was stolen and modified from
        http://stackoverflow.com/a/15860757/1552338
        '''
        self._start = self._stop
        self._stop = time.time()
        self._status = self._stop - self._start

        progress = float(step)/float(self._nsteps - 1)
        if progress >= 1:
            progress = 1
            self._status = "Complete...\r\n"
        block = int(round(self._width * progress))
        text = "\rProcessing: [{}] {:.1%} {:.3}".format("#" * block
                                                        + "-" * (self._width
                                                                 - block),
                                                        progress,
                                                        self._status)
        sys.stdout.write(text)
        sys.stdout.flush()

# Create x, y data
delta = 0.5

extent = (-3, 4, -4, 3)

x = np.arange(-3.0, 4.001, delta)
y = np.arange(-4.0, 3.001, delta)
X, Y = np.meshgrid(x, y)

# Boost the upper limit to avoid truncation errors.
levels = np.arange(-2.0, 1.601, 0.4)

fig, ax = plt.subplots(1, 1)  # create our figure and axes
progress = ProgressBar(n_steps)  # initialize the progress bar


def update(frame_number):
    '''
    This is where our contour is creating
    '''
    sigma = np.random.uniform(0, 1, 4)
    Z1 = mlab.bivariate_normal(X, Y, sigma[0], sigma[1], 0.0, 0.0)
    Z2 = mlab.bivariate_normal(X, Y, sigma[2], sigma[3], 1, 1)
    Z = (Z1 - Z2) * 10

    norm = cm.colors.Normalize(vmax=abs(Z).max(), vmin=-abs(Z).max())
    cmap = cm.PRGn

    contf = plt.contourf(X, Y, Z, levels,
                         cmap=cm.get_cmap(cmap, len(levels) - 1),
                         norm=norm)

    progress.update(frame_number)

    return contf

# Construct the animation, using the update function as the animation
# director.
anim = FuncAnimation(fig, update, interval=n_steps)
anim.save("AnimContourf.mp4")
plt.close(fig)
sys.stdout.write("\n")

@WeatherGod
Copy link
Member

The reason it is taking progressively longer is because the previous
contourf's aren't getting cleared out. So, each "save()" call is rendering
more and more contourf's that aren't actually visible because the newer
ones are on top.

On Fri, Aug 26, 2016 at 12:49 PM, Erich L Foster [email protected]
wrote:

When creating an animation with FuncAnimation and contourf I have run into
an issue where the more frames I have the slower things become. This is
illustrated in the code below. I print a progress bar, which displays both
progress and the time required for the current step. As the animation
creation progresses you will see that each step takes longer and longer. I
was able to mitigate this by creating a derived class MyFuncAnimation,
where I overloaded the function save. In the new version of save I go
through and clear all collections associated with self._drawn_artists,
after writer.grab_frame, in the following way:
for collection in self._drawn_artists.collections:
gca().collections.remove(collection)

I am not sure if there is a better way to do this, but it was the only way
I could figure out how to address the issue.

Matplot Version = 1.5.1
Python Version = 3.5.2
Platform = MAC OS 10.11.6

Installed matplotlib through pip

Example:

#!/usr/bin/env python3

import numpy as np
import matplotlib.pyplot as plt
from matplotlib import mlab, cm
from matplotlib.animation import FuncAnimation
import sys
import time

barWidth = 40
n_steps = 100

class ProgressBar():
'''
Provides a simple progress bar class
'''
def init(self, nsteps, width=barWidth):
self._start = self._stop = time.time()
self._nsteps = nsteps
self._width = width
self._status = ""

def update(self, step):
    '''
    This function produces and updates a progress bar.
    It was stolen and modified from
    http://stackoverflow.com/a/15860757/1552338
    '''
    self._start = self._stop
    self._stop = time.time()
    self._status = self._stop - self._start

    progress = float(step)/float(self._nsteps - 1)
    if progress >= 1:
        progress = 1
        self._status = "Complete...\r\n"
    block = int(round(self._width * progress))
    text = "\rProcessing: [{}] {:.1%} {:.3}".format("#" * block
                                                    + "-" * (self._width
                                                             - block),
                                                    progress,
                                                    self._status)
    sys.stdout.write(text)
    sys.stdout.flush()

Create x, y data

delta = 0.5

extent = (-3, 4, -4, 3)

x = np.arange(-3.0, 4.001, delta)
y = np.arange(-4.0, 3.001, delta)
X, Y = np.meshgrid(x, y)

Boost the upper limit to avoid truncation errors.

levels = np.arange(-2.0, 1.601, 0.4)

fig, ax = plt.subplots(1, 1) # create our figure and axes
progress = ProgressBar(n_steps) # initialize the progress bar

def update(frame_number):
'''
This is where our contour is creating
'''
sigma = np.random.uniform(0, 1, 4)
Z1 = mlab.bivariate_normal(X, Y, sigma[0], sigma[1], 0.0, 0.0)
Z2 = mlab.bivariate_normal(X, Y, sigma[2], sigma[3], 1, 1)
Z = (Z1 - Z2) * 10

norm = cm.colors.Normalize(vmax=abs(Z).max(), vmin=-abs(Z).max())
cmap = cm.PRGn

contf = plt.contourf(X, Y, Z, levels,
                     cmap=cm.get_cmap(cmap, len(levels) - 1),
                     norm=norm)

progress.update(frame_number)

return contf

Construct the animation, using the update function as the animation

director.

anim = FuncAnimation(fig, update, interval=n_steps)
anim.save("AnimContourf.mp4")
plt.close(fig)
sys.stdout.write("\n")


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#6985, or mute the thread
https://github.com/notifications/unsubscribe-auth/AARy-OwemAlgHkdMVMVOHQ4rGtw8de-Rks5qjxkMgaJpZM4JuQxC
.

@erichlf
Copy link
Author

erichlf commented Aug 26, 2016

This is what I understood, but what is the proper method to do this?

@WeatherGod
Copy link
Member

oh, and I think there is a tiny memory leak that is fixed in master with
respect to contourf(), but it wouldn't be the main reason for the slowdown.

On Fri, Aug 26, 2016 at 12:53 PM, Benjamin Root [email protected]
wrote:

The reason it is taking progressively longer is because the previous
contourf's aren't getting cleared out. So, each "save()" call is rendering
more and more contourf's that aren't actually visible because the newer
ones are on top.

On Fri, Aug 26, 2016 at 12:49 PM, Erich L Foster <[email protected]

wrote:

When creating an animation with FuncAnimation and contourf I have run
into an issue where the more frames I have the slower things become. This
is illustrated in the code below. I print a progress bar, which displays
both progress and the time required for the current step. As the animation
creation progresses you will see that each step takes longer and longer. I
was able to mitigate this by creating a derived class MyFuncAnimation,
where I overloaded the function save. In the new version of save I go
through and clear all collections associated with self._drawn_artists,
after writer.grab_frame, in the following way:
for collection in self._drawn_artists.collections:
gca().collections.remove(collection)

I am not sure if there is a better way to do this, but it was the only
way I could figure out how to address the issue.

Matplot Version = 1.5.1
Python Version = 3.5.2
Platform = MAC OS 10.11.6

Installed matplotlib through pip

Example:

#!/usr/bin/env python3

import numpy as np
import matplotlib.pyplot as plt
from matplotlib import mlab, cm
from matplotlib.animation import FuncAnimation
import sys
import time

barWidth = 40
n_steps = 100

class ProgressBar():
'''
Provides a simple progress bar class
'''
def init(self, nsteps, width=barWidth):
self._start = self._stop = time.time()
self._nsteps = nsteps
self._width = width
self._status = ""

def update(self, step):
    '''
    This function produces and updates a progress bar.
    It was stolen and modified from
    http://stackoverflow.com/a/15860757/1552338
    '''
    self._start = self._stop
    self._stop = time.time()
    self._status = self._stop - self._start

    progress = float(step)/float(self._nsteps - 1)
    if progress >= 1:
        progress = 1
        self._status = "Complete...\r\n"
    block = int(round(self._width * progress))
    text = "\rProcessing: [{}] {:.1%} {:.3}".format("#" * block
                                                    + "-" * (self._width
                                                             - block),
                                                    progress,
                                                    self._status)
    sys.stdout.write(text)
    sys.stdout.flush()

Create x, y data

delta = 0.5

extent = (-3, 4, -4, 3)

x = np.arange(-3.0, 4.001, delta)
y = np.arange(-4.0, 3.001, delta)
X, Y = np.meshgrid(x, y)

Boost the upper limit to avoid truncation errors.

levels = np.arange(-2.0, 1.601, 0.4)

fig, ax = plt.subplots(1, 1) # create our figure and axes
progress = ProgressBar(n_steps) # initialize the progress bar

def update(frame_number):
'''
This is where our contour is creating
'''
sigma = np.random.uniform(0, 1, 4)
Z1 = mlab.bivariate_normal(X, Y, sigma[0], sigma[1], 0.0, 0.0)
Z2 = mlab.bivariate_normal(X, Y, sigma[2], sigma[3], 1, 1)
Z = (Z1 - Z2) * 10

norm = cm.colors.Normalize(vmax=abs(Z).max(), vmin=-abs(Z).max())
cmap = cm.PRGn

contf = plt.contourf(X, Y, Z, levels,
                     cmap=cm.get_cmap(cmap, len(levels) - 1),
                     norm=norm)

progress.update(frame_number)

return contf

Construct the animation, using the update function as the animation

director.

anim = FuncAnimation(fig, update, interval=n_steps)
anim.save("AnimContourf.mp4")
plt.close(fig)
sys.stdout.write("\n")


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#6985, or mute the
thread
https://github.com/notifications/unsubscribe-auth/AARy-OwemAlgHkdMVMVOHQ4rGtw8de-Rks5qjxkMgaJpZM4JuQxC
.

@QuLogic
Copy link
Member

QuLogic commented Aug 26, 2016

Set blit=True and return the list of artists to clear, but unfortunately, that doesn't work with contourf due to #6139.

@WeatherGod
Copy link
Member

The proper way is not to do blitting. The QuadContourSet object doesn't
have the ability to reuse itself in the same way that an image could. You
just need to remove the artist before computing the next contourf(). All
artists should come with a .remove() method, so you'd just need to store
that artist somewhere and do a conditional remove() on it before calling
contourf() and replacing the previous object with the new contourf object.

This usually means having a global variable of some sort to store the
object between frames, initialized to None for the first frame (which is
why you would need to conditionally call .remove()).

Does that help?

On Fri, Aug 26, 2016 at 5:39 PM, Elliott Sales de Andrade <
[email protected]> wrote:

Set blit=True and return the list of artists to clear, but unfortunately,
that doesn't work with contourf due to #6139
#6139.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#6985 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AARy-C0pk-haLauALF6QcS_IFEg1IdBTks5qj10JgaJpZM4JuQxC
.

@erichlf
Copy link
Author

erichlf commented Aug 26, 2016

Seems like this would be handled better by the Animation class, but I can certainly do it the other way too.

@QuLogic
Copy link
Member

QuLogic commented Aug 26, 2016

It's simpler to pass the artists to the update function via fargs and update the existing artists, though:

def init():
    '''
    This is where our contour is created
    '''
    sigma = np.random.uniform(0, 1, 4)
    Z1 = mlab.bivariate_normal(X, Y, sigma[0], sigma[1], 0.0, 0.0)
    Z2 = mlab.bivariate_normal(X, Y, sigma[2], sigma[3], 1, 1)
    Z = (Z1 - Z2) * 10

    norm = cm.colors.Normalize(vmax=abs(Z).max(), vmin=-abs(Z).max())
    cmap = cm.PRGn

    contf = plt.contourf(X, Y, Z, levels,
                         cmap=cm.get_cmap(cmap, len(levels) - 1),
                         norm=norm)

    return contf


def update(frame_number, contf):
    '''
    This is where our contour is updated
    '''
    sigma = np.random.uniform(0, 1, 4)
    Z1 = mlab.bivariate_normal(X, Y, sigma[0], sigma[1], 0.0, 0.0)
    Z2 = mlab.bivariate_normal(X, Y, sigma[2], sigma[3], 1, 1)
    Z = (Z1 - Z2) * 10

    norm = cm.colors.Normalize(vmax=abs(Z).max(), vmin=-abs(Z).max())
    cmap = cm.PRGn

    contf.set_array(Z)
    contf.set_cmap(cm.get_cmap(cmap, len(levels) - 1))
    contf.set_norm(norm)

    progress.update(frame_number)

# Construct the animation, using the update function as the animation
# director.
contf = init()
anim = FuncAnimation(fig, update, interval=n_steps, fargs=(contf, ))
anim.save("AnimContourf.mp4")
plt.close(fig)
sys.stdout.write("\n")

though I'm not sure changing the cmap like that is the best idea. Depending on what you're doing, the init bit could be just adding a nearly empty contourf instead of the full stuff.

@QuLogic
Copy link
Member

QuLogic commented Aug 26, 2016

Sorry, I may have been thrown by the randomness; alas, I think it doesn't update the contour. I guess the longer method is the only option until contourf returns a more normal artist.

@WeatherGod
Copy link
Member

Correct. Just updating the Z data isn't sufficient. It still has to
calculate the polygons and finalize them. Unfortunately, we never really
cleaned up that code too well, so the only place that handles everything
correctly is in contourf().

On Fri, Aug 26, 2016 at 6:12 PM, Elliott Sales de Andrade <
[email protected]> wrote:

Sorry, I may have been thrown by the randomness; alas, I think it doesn't
update the contour. I guess the longer method is the only option until
contourf returns a more normal artist.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#6985 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AARy-INCEOqMSSFzC_50oX7RcUg_SDdQks5qj2TkgaJpZM4JuQxC
.

@erichlf
Copy link
Author

erichlf commented Aug 29, 2016

So how would the solution given by @QuLogic work for a contourf which uses tri=True? In this case the z data is not in matrix form. The example I gave isn't my real example, but my real example is too complicated and I use a netcdf file to fill in data.

@WeatherGod
Copy link
Member

It wouldn't work. That's what his apology was about because the update data stuff doesn't update the entire object, and as you point out, the other contour objects don't work the same way.

You would have to follow what I originally suggested, which is to keep track of the objects via a global variable (or some other specialized class that is passed in through fargs that manages the lifecycle of the contourf object). Some psuedo-code:

def update(num, ...):
    if update.theobj is not None:
        update.theobj.remove()
    update.theobj = plt.contourf(data, ....)
    return (update.theobj,)
update.theobj = None

Something to that effect.
By the way, I think you are supposed to return an iterable of objects from the update function, not a single object.

@efiring
Copy link
Member

efiring commented Aug 29, 2016

With contour, just clear the axes and remake the contour plus whatever else needs to go there. I don't see any point in trying to track and remove objects associated with the contours.

@erichlf
Copy link
Author

erichlf commented Aug 29, 2016

@WeatherGod When I do
return (contf,)
I get AttributeError: 'tuple' object has no attribute 'autoscale_None'
Maybe I am not doing this correctly.

@efiring: In my code removing collections is about 8% faster than ax.clear(). Also, ax.clear forces me to redraw continents in basemap. In fact, the 8% slower was without even redrawing the continents.

@WeatherGod
Copy link
Member

Hmm, that's strange. If you are not blitting at all, you can get away with
not returning anything at all. But that error definitely shouldn't happen,
I don't think.

On Mon, Aug 29, 2016 at 1:47 PM, Erich L Foster [email protected]
wrote:

@WeatherGod https://github.com/WeatherGod When I do
return (contf,)
I get AttributeError: 'tuple' object has no attribute 'autoscale_None'
Maybe I am not doing this correctly.

@efiring https://github.com/efiring: In my code removing collections is
about 8% faster than ax.clear().


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#6985 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AARy-JMkjzpnTRvEvjatRaY3PlzZjocGks5qkxsmgaJpZM4JuQxC
.

@erichlf
Copy link
Author

erichlf commented Aug 29, 2016

So what is different between self._drawn_artists.collections and ax.collections? If I go through and remove collections from ax this will of course remove my basemap continents and I lose about 10% in time, but using this method on self._drawn_artists.collections doesn't remove the continents. Right now, for efficiency, using my derived class with overloaded save is definitely the fastest.

@efiring
Copy link
Member

efiring commented Aug 29, 2016

If you want to remove only the collections added by contourf, can't you just cycle through the list you saved in your contf variable?

@erichlf
Copy link
Author

erichlf commented Aug 29, 2016

I tried (before):

if contf is not None:
    for collection in contf.collections:
        gca().collections.remove(collection)

But,

File "./plotCDAT.py", line 149, in animate
gca().collections.remove(collection)
ValueError: list.remove(x): x not in list

I understand what the error is saying, but I don't understand why collection wouldn't be in gca().collections.

@WeatherGod
Copy link
Member

No, the artist objects themselves have a remove() method. I wasn't
referring to the list's remove() method. You want the artist's remove()
method because the artist knows everything it needs to do to clean itself
up.

On Mon, Aug 29, 2016 at 4:15 PM, Erich L Foster [email protected]
wrote:

I tried (before):

if contf is not None:
for collection in contf.collections:
gca().collections.remove(collection)

But,

File "./plotCDAT.py", line 149, in animate
gca().collections.remove(collection)
ValueError: list.remove(x): x not in list

I understand what the error is saying, but I don't understand why
collection wouldn't be in gca().collections.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#6985 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AARy-GpIOeZd-iHUMqKEu1FJSO8u3a_dks5qkz3LgaJpZM4JuQxC
.

@erichlf
Copy link
Author

erichlf commented Aug 29, 2016

Maybe I am still confused, but using

for collection in contf.collections:
    collection.remove

Causes things to be as slow as if I hadn't removed the collections at all, i.e. 10x slower. So, for some reason removing the collection from gca().collections and then adding back in the coastlines is actually much faster.

@efiring
Copy link
Member

efiring commented Aug 29, 2016

You need collection.remove() with the parentheses.

@erichlf
Copy link
Author

erichlf commented Aug 30, 2016

@efiring Then it says

ValueError: list.remove(x): x not in list

So clearly, I must be accessing the wrong object.

@jenshnielsen
Copy link
Member

Does it really do that if you execute

for collection in contf.collections:
    collection.remove()

That seems very strange?

@WeatherGod
Copy link
Member

Does the following work for you?

Python 2.7.12 (default, Jul 19 2016, 14:17:21) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> import matplotlib.pyplot as plt
>>> z = np.random.random((10, 10))
>>> foo = plt.contourf(z)
>>> foo
<matplotlib.contour.QuadContourSet object at 0x7fc97aff4210>
>>> foo.collections
<a list of 7 mcoll.PathCollection objects>
>>> for c in foo.collections:
...     c.remove()
... 
>>> plt.show()

With that, you should get a blank plot with limits from 0 to 9.

As a note to self, perhaps we should augment ContourSet so that it has its own remove() method so that it can clear out not just the PatchCollections, but also any contour labels and such if they exist.

@erichlf
Copy link
Author

erichlf commented Aug 30, 2016

After some debugging I discovered that a collection was being removed more than once. However, doing the contf.collection method results again in an extremely slow code. Anyway, I am just going to use the method of overriding save.

I hope you guys consider blitting for contours.

@tacaswell
Copy link
Member

The time consuming step (as I understand it) with contours is actually computing the contours.

Blitting only helps if you only need to re-draw a single artist and the slowest part of the process is redrawing everything else in the figure.

Blitting is also only used when displaying to the screen, anim.save("AnimContourf.mp4") always fully-rerenders the figure.

@jfainberg
Copy link

Thanks for the discussion above! Is there a way to deal with this if you're using ArtistAnimation and not FuncAnimation? If I remove artists after every iteration of my algorithm, there's nothing to plot at the end, e.g. you get:

    if artist.axes.figure not in figs:
AttributeError: 'NoneType' object has no attribute 'figure'

But otherwise it becomes painfully slow (~hours).

@efiring
Copy link
Member

efiring commented Nov 25, 2016

I don't see how ArtistAnimation helps anything; it just puts the creation of your contours for each frame into an initial step to create the list of lists of artists.

There is simply no way to get around the time taken for the contouring calculation itself.

A penalty of 8% for a simpler algorithm is not worth spending a lot of programming time on.

It sounds like the real problem is that you don't want to recreate the Basemap from scratch each time. The solution is to initialize the Basemap object (which includes the time-consuming step of extracting the coastlines), save it, and reuse it as the starting point for each frame. Unfortunately I don't have a simple ready-to-go example to point to, because all of my code along these lines predates the animation module (or at least my use of it), involves working with remote data sources, and uses my Basemap wrapper code.

In the interest of reducing the number of open issues we have to keep scanning and checking, I am going to close this now.

@efiring efiring closed this as completed Nov 25, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants