MNT: overhaul stale handling #4738

tacaswell · 2015-07-18T01:31:23Z

Only propagate stale state to the figure from it's direct children, not all artists. This is to prevent double call backs when using interactive mode with backends which do not implement an asynchronous draw_idle.

This is to match the behavior of artists being uniquely associated with a single Axes.

If a draw has been forced, but either the figure or axes is not stale some of the draw processes will cause them to become scale (because the current implementation is naive and does no check if values have changed so no-op sets still mark the artist as stale). By setting _stale to be True the stale events will not propagate past the axes/figure and schedule another redraw. This only really matters for Canvas classes which do not implement an asynchronous `draw_idle`. If there is an asynchronous `draw_idle` the additional draw requests will be ignored until the current draw is finished anyway.

WeatherGod · 2015-07-24T13:46:28Z

This PR actually only has a single failure (the same in all python versions):

======================================================================
FAIL: matplotlib.tests.test_offsetbox.test_offsetbox_clip_children
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/matplotlib/matplotlib/venv/lib/python2.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/travis/build/matplotlib/matplotlib/venv/lib/python2.6/site-packages/matplotlib-1.5.dev1-py2.6-linux-x86_64.egg/matplotlib/testing/decorators.py", line 110, in wrapped_function
    func(*args, **kwargs)
  File "/home/travis/build/matplotlib/matplotlib/venv/lib/python2.6/site-packages/matplotlib-1.5.dev1-py2.6-linux-x86_64.egg/matplotlib/tests/test_offsetbox.py", line 83, in test_offsetbox_clip_children
    assert_true(fig.stale)
AssertionError

When setting the axes of an offset box to propagate to it's children.

tacaswell · 2015-07-31T01:19:41Z

Turns out the issue was that funny things were happening with the offsetbox subclasses which was getting masked by the fact that before artists were pinging both the axes and the figure, now they just hit the figure.

There are probably a bunch more issues like this that we will slowly find...

Do not re-use a much too broad callback registry for propagating stale state.

tacaswell · 2015-08-01T04:22:42Z

@rmorshea the stale callbacks have been simplified to not use pchanged anymore. I think this will make replicating the current proposed behavior easier.

Always propagate stale up (even if immediate parent is stale)

mdboom · 2015-08-06T20:44:40Z

I definitely see this as an improvement, modulo investigating the Travis failure. Possibly more to do over time, but I don't see any reason not to do this much now.

tacaswell · 2015-08-06T21:08:09Z

The travis issue is triggering draws in a draw. This is caused by:

not checking if anything actually changed, just setting stale on every setter so stale gets triggered during draws
the stale state now propagating past existing stale = True states
Agg not having an asynchronous draw method and draw_idle just passing through to draw.

@mdehoon makes the very good point that due to our late validation of values if you set an invalid value you don't find out until draw time. If the stale state does not propagate past the existing stale artists and you are using plain python then a re-draw won't be triggered the next time the value is changed (as implemented with IPython it will work fine as-is).

The solutions to this that I see are:

put back in the short circuit logic back in and tell people not to use plain python (I am +0 on this, and @mdehoon is -1)
do proper validation and change determination in ever setter (I am -1 on this as traitlets will take care of this)
make Agg's draw_idle re-entrant (I am +0 and am guessing @mdehoon is +1)

I am working on adding a commit to make the base draw_idle re-entrant.

Calling `draw_idle` while executing a `draw_idle` will no-longer infinitely recurse.

tacaswell · 2015-08-07T02:12:32Z

I think this is ready to go again.

WeatherGod · 2015-08-12T15:40:53Z

Still, I don't see the value in using sets here, and I don't like that it works now on a technicality. A list would work just fine. It isn't like we are expecting to have many figures to check against. At the extreme end, i would imagine someone doing something fancy with 10 figures, but typically, only 1 or 3 figures.

dopplershift · 2015-08-12T15:47:57Z

I disagree about it working on a "technicality"--this is documented python behavior, essentially making the in check work using is. In fact, using in (which uses __eq__) ends up on the same code path as hash:

User-defined classes have __eq__() and __hash__() methods by default; with them,
all objects compare unequal (except with themselves) and x.__hash__() returns an
appropriate value such that x == y implies both that x is y and hash(x) == hash(y).

(https://docs.python.org/3/reference/datamodel.html#object.__eq__)

However, I do agree that there is essentially no performance difference in our real-world scenarios here.

WeatherGod · 2015-08-12T18:23:13Z

Ok, just tried out this branch. I was doing an imshow of a very large array (3600x7000), and a basemap plot and using the TkAgg backend. Watching "top" in another terminal, I see the memory usage increasing every time my mouse passes over the figure window. It never goes back down. The amount of memory increases by hundreds of MB for a quick pass over the window.

I don't know yet if it is because of this PR, or if it is due to some unfixed issue with stale flags, or something else entirely. Just noting it here for the moment.

WeatherGod · 2015-08-12T18:27:37Z

Joys... this is on master and this branch. Furthermore, once you pass over the figure window, the memory usage keeps creeping up, even though the mouse is no longer over the window.

tacaswell · 2015-08-12T18:29:09Z

Try commenting out the mouse hover logic in backed base.

On Wed, Aug 12, 2015, 2:27 PM Benjamin Root [email protected]
wrote:

Joys... this is on master and this branch. Furthermore, once you pass over
the figure window, the memory usage keeps creeping up, even though the
mouse is no longer over the window.

—
Reply to this email directly or view it on GitHub
#4738 (comment)
.

WeatherGod · 2015-08-12T18:41:26Z

Yup, looks like commenting out the contents of NavigationToolbar2.mouse_move "fixed" the problem. Do you think it is a bug to be dealt with here or shall I file a new issue?

tacaswell · 2015-08-12T18:44:07Z

Tack it on to the pr where i am working on removing hit list

On Wed, Aug 12, 2015, 2:41 PM Benjamin Root [email protected]
wrote:

Yup, looks like commenting out the contents of
NavigationToolbar2.mouse_move "fixed" the problem. Do you think it is a
bug to be dealt with here or shall I file a new issue?

—
Reply to this email directly or view it on GitHub
#4738 (comment)
.

- disconnect the stale_callback (this assumes that matplotlib#4738 goes in) - remove the artist from the Axes.mousover_set - marks the parent axes/figure as stale - resets the artist's axes/figure to None

Artists do not properly guard against being removed from figures or Axes when tracking 'stale'. This will keep them from throwing an exception if they become stale after being removed from an Axes or Figure.

Look at figures, not axes, and use a set

- short circuit if not really chaning state - don't fire stale trigger on changing animation state

When using blitting in animation make sure to set the blitted artsts as animated so that changing their state does not trigger the normal stale -> draw_idle cascade.

jenshnielsen · 2015-08-14T20:09:09Z

I can still reproduce the original issue in #4732 when interactivity is on i.e. plt.ion()
However I can also reproduce the issue on 1.4.3 so I not really cure what to make of this

jenshnielsen · 2015-08-14T20:25:29Z

I.e if I call set_dashes('--') it triggers that exception on every draw until call set_dashes((1,1)) or whatever but surely that is just missing input validation?

tacaswell · 2015-08-14T20:31:44Z

That is the expected behavior, the problem was that previously correcting
the value would not trigger a redraw at all and the was no obvious way to
do so.

On Fri, Aug 14, 2015, 4:25 PM Jens Hedegaard Nielsen <
[email protected]> wrote:

I.e if I call set_dashes('--') it triggers that exception on every draw
until call set_dashes((1,1)) or whatever but surely that is just missing
input validation?

—
Reply to this email directly or view it on GitHub
#4738 (comment)
.

jenshnielsen · 2015-08-14T20:46:45Z

@tacaswell That makes sense. However, I can't reproduce that on current master.

set_dashes((3,'3)) after set_dashes(('--')) correctly triggers the redraw so it seems that either this has already been fixed by something else or I am doing something stupid. In any case this is an improvement to the stale login so I will go ahead and merge it

MNT: overhaul stale handling

tacaswell · 2015-08-14T20:48:34Z

Are you using ipython or plain python? The redraw logic is different in the
two cases.

On Fri, Aug 14, 2015, 4:46 PM Jens Hedegaard Nielsen <
[email protected]> wrote:

Merged #4738 #4738.

—
Reply to this email directly or view it on GitHub
#4738 (comment).

jenshnielsen · 2015-08-14T20:49:13Z

plain python

FIX: attempting to draw marks figure as not-stale

02158b1

Closes matplotlib#4732

tacaswell added the status: needs review label Jul 18, 2015

tacaswell added this to the next point release milestone Jul 18, 2015

tacaswell added 6 commits July 18, 2015 02:11

PEP: remove extra lines

b0c3741

FIX: do not propogate stale on animated artists

7f3ef65

MNT: Limit what propagates stale to figure

a677634

Only propagate stale state to the figure from it's direct children, not all artists. This is to prevent double call backs when using interactive mode with backends which do not implement an asynchronous draw_idle.

API: forbid changing figure of Artist instance

6d6e39a

This is to match the behavior of artists being uniquely associated with a single Axes.

DOC: added explanatory comments to example

1a58f17

WeatherGod mentioned this pull request Jul 24, 2015

Non-interactive backend calls draw more than 100 times #4780

Closed

tacaswell mentioned this pull request Jul 24, 2015

DOC: Start to document interactive figures #4779

Merged

FIX: make sure that offset boxes propagate axes

72c5ce2

When setting the axes of an offset box to propagate to it's children.

tacaswell added 2 commits July 30, 2015 23:18

TST: don't reuse artists in tests

4cc9a74

FIX: protect against None child

b39fd90

tacaswell force-pushed the fix_repl_ensure_fig_unstale branch from d168e59 to b39fd90 Compare July 31, 2015 04:05

tacaswell added 3 commits August 1, 2015 00:04

MNT: use direct callback for stale, not pchanged

0b49a55

Do not re-use a much too broad callback registry for propagating stale state.

MNT: still only propagate True up the chain

59eaad6

MNT: fix vanilla python repl hook

69b1b36

tacaswell added 2 commits August 1, 2015 00:55

FIX: remove excessive stale cleaning

770b8d5

MNT: don't need try/except anymore

aa810b6

Always propagate stale up (even if immediate parent is stale)

tacaswell mentioned this pull request Aug 1, 2015

Mnt direct call draw idle [WIP] #4849

Closed

tacaswell added 2 commits August 1, 2015 01:00

FIX: fix stupid bug in pyplot

0af89d7

FIX: don't push race condition on un-pickling

14b5a36

MNT: make base draw_idle re-entrant

252d7a7

Calling `draw_idle` while executing a `draw_idle` will no-longer infinitely recurse.

WeatherGod mentioned this pull request Aug 12, 2015

PRF: only check some artists on mousemove #4878

Merged

James Evans and others added 7 commits August 13, 2015 21:42

FIX: stale propagation in artist without fig/ax

13e7c09

Artists do not properly guard against being removed from figures or Axes when tracking 'stale'. This will keep them from throwing an exception if they become stale after being removed from an Axes or Figure.

MNT: make sorting out what to re-draw faster

83ea020

Look at figures, not axes, and use a set

MNT: make set_animated smarter

a5f5ab5

- short circuit if not really chaning state - don't fire stale trigger on changing animation state

MNT: when blitting in make artist animated

c75f75d

When using blitting in animation make sure to set the blitted artsts as animated so that changing their state does not trigger the normal stale -> draw_idle cascade.

MNT: set animated to blit state

51cc0ca

MNT: setting cmap/norm marks as stale

0e201f9

MNT: set_data on FigureImage marks as stale

25079bf

tacaswell force-pushed the fix_repl_ensure_fig_unstale branch from ce80568 to 25079bf Compare August 14, 2015 01:43

jenshnielsen added a commit that referenced this pull request Aug 14, 2015

Merge pull request #4738 from tacaswell/fix_repl_ensure_fig_unstale

212b0ff

MNT: overhaul stale handling

jenshnielsen merged commit 212b0ff into matplotlib:master Aug 14, 2015

tacaswell removed the status: needs review label Aug 14, 2015

jenshnielsen mentioned this pull request Aug 16, 2015

Memory leak in NavigationToolbar2.mouse_over #4936

Closed

tacaswell deleted the fix_repl_ensure_fig_unstale branch August 17, 2015 01:15

pwuertz mentioned this pull request Aug 18, 2015

qt backend has more draws than necessary #4943

Merged

dstansby mentioned this pull request Nov 16, 2017

Error with plot_date and agg backend: ValueError: ordinal must be >= 1 #6023

Closed

Uh oh!

MNT: overhaul stale handling #4738

MNT: overhaul stale handling #4738

Uh oh!

Conversation

tacaswell commented Jul 18, 2015

Uh oh!

WeatherGod commented Jul 24, 2015

Uh oh!

tacaswell commented Jul 31, 2015

Uh oh!

tacaswell commented Aug 1, 2015

Uh oh!

mdboom commented Aug 6, 2015

Uh oh!

tacaswell commented Aug 6, 2015

Uh oh!

tacaswell commented Aug 7, 2015

Uh oh!

WeatherGod commented Aug 12, 2015

Uh oh!

dopplershift commented Aug 12, 2015

Uh oh!

WeatherGod commented Aug 12, 2015

Uh oh!

WeatherGod commented Aug 12, 2015

Uh oh!

tacaswell commented Aug 12, 2015

Uh oh!

WeatherGod commented Aug 12, 2015

Uh oh!

tacaswell commented Aug 12, 2015

Uh oh!

jenshnielsen commented Aug 14, 2015

Uh oh!

jenshnielsen commented Aug 14, 2015

Uh oh!

tacaswell commented Aug 14, 2015

Uh oh!

jenshnielsen commented Aug 14, 2015

Uh oh!

tacaswell commented Aug 14, 2015

Uh oh!

jenshnielsen commented Aug 14, 2015

Uh oh!

Uh oh!