Make it easier to improve UI event metadata. #16931

anntzer · 2020-03-27T10:42:53Z

PR Summary

Currently, UI events (MouseEvent, KeyEvent, etc.) are generated by
letting the GUI-specific backends massage the native event objects into
a list of args/kwargs and then call
FigureCanvasBase.motion_notify_event/.key_press_event/etc. This
makes it a bit tricky to improve the metadata on the events, because one
needs to change the signature on both the FigureCanvasBase method and
the event class. Moreover, the motion_notify_event/etc. methods are
directly bound as event handlers in the gtk3 and tk backends, and thus
have incompatible signatures there.

Instead, the native GUI handlers can directly construct the relevant
event objects and trigger the events themselves; a new Event.process
helper method makes this even shorter (and allows to keep factoring some
common functionality e.g. for tracking the last pressed button or key).

As an example, this PR also updates figure_leave_event to always
correctly set the event location based on the current cursor position,
instead of the last triggered location event (which may be outdated);
this can now easily be done on a backend-by-backend basis, instead of
coordinating the change with FigureCanvasBase.figure_leave_event.

This also exposed another (minor) issue, in that resize events often
trigger two calls to draw_idle -- one in the GUI-specific handler, and
one in FigureCanvasBase.draw_idle (now moved to ResizeEvent.process, but
should perhaps instead be a callback autoconnected to "resize_event") --
could probably be fixed later.

As an example, this strategy would have made the change at #9814 easier (by just passing the correct additional arguments to LocationEvent).

This would also make #6159 easier (right now #6159 only fixes modifiers for button_press_event, but it would also nice to have it for motion_notify_event and button_release_event).

Also closes #8715 (by deprecating the relevant handlers).

Another small commit updates .gitattributes so that diffs in _macosx.m get the correct context when displayed locally.

PR Checklist

Has Pytest style unit tests
Code is Flake 8 compliant
New features are documented, with examples if plot related
Documentation is sphinx and numpydoc compliant
Added an entry to doc/users/next_whats_new/ if major new feature (follow instructions in README.rst there)
Documented in doc/api/api_changes.rst if API changed in a backward-incompatible way

tacaswell · 2020-03-28T21:23:13Z

lib/matplotlib/backend_bases.py

    def __init__(self, name, canvas, key, x=0, y=0, guiEvent=None):
        LocationEvent.__init__(self, name, canvas, x, y, guiEvent=guiEvent)
        self.key = key

+    def process(self):


This (or the init) should update the location based on the last-known mouse location if the user does not supply one.

Did that occur before? One of the points of this PR is also to stop trying to fill in this "bad" metadata, which can easily be wrong. (If it did occur, sure, I can put that back for now, but I don't think it did?)

That is my understanding of

matplotlib/lib/matplotlib/backend_bases.py

Lines 1691 to 1692 in e9e1c87

event = KeyEvent(

s, self, key, self._lastx, self._lasty, guiEvent=guiEvent)

and

matplotlib/lib/matplotlib/backend_bases.py

Lines 1700 to 1701 in e9e1c87

event = KeyEvent(

s, self, key, self._lastx, self._lasty, guiEvent=guiEvent)

tacaswell · 2020-03-28T21:36:18Z

I'm mixed on this, but leaning 👍.

On the pro side, this makes it slightly more explicit what is going on (as you create the Event object rather than calling a function that creates it for you). I have always found the method name (pick_event) to be confusing so I'm not particularly sad to see them go and I hope that these are not being widely used by non-backend authors.

On the con-side having the process method push it's self through the callback registry seems a bit odd. This also moves management of (private) state on the canvas out of canvas methods and into the EventSubclass.process which is also a bit odd. I think this will require changes to atleast 3 extrernal packages (mplcairo, ipympl, the kivy backend).

The process for figure_leave_event should clear LocationEevent.lastevent

If we are going to do this the on-init (!?) processing chain that happen in LocationEvent.__init__ (via _update_enter_leave should be moved to process.

anntzer · 2020-03-28T23:55:45Z

On the pro side, this makes it slightly more explicit what is going on (as you create the Event object rather than calling a function that creates it for you). I have always found the method name (pick_event) to be confusing so I'm not particularly sad to see them go and I hope that these are not being widely used by non-backend authors.

Especially, these methods are (effectively) backend-dependent, so you can't write backend-independent code using them (well, unless you're careful to always call the base class implementation FigureCanvasBase.foo_event instead of canvas.foo_event...).

On the con-side having the process method push it's self through the callback registry seems a bit odd.

I'm not super-wedded to the API; initially I had the somewhat more obvious(?) def FigureCanvasBase.process_event(event): self.callbacks.process(event.name, event), which is a bit more verbose but works too. Or we can just not add any helper and force backend implementers to call everywhere event = FooEvent(...); canvas.callbacks.process(event.name, event) which is even more verbose, but again works too. On the other hand, having a helper is also useful for e.g. testing code which wants to trigger arbitrary events (e.g., mplcursors).

This also moves management of (private) state on the canvas out of canvas methods and into the EventSubclass.process which is also a bit odd.

I agree, but really I think these things should be just plain callbacks always connected at canvas setup (e.g. adding a callback that draws on resize).

I think this will require changes to atleast 3 extrernal packages (mplcairo, ipympl, the kivy backend).

mplcairo doesn't need to change anything because it fully relies on Matplotlib's base classes for UI integration (that's basically why I wrote the foocairo backends for Matplotlib -- making sure that GUI and rendering were orthogonal); IOW it only overrides paintEvent (Qt), on_draw_event (GTK3), etc.

A quick look suggests that yes, Kivy and ipympl will need updates, but they can at least use the no-helper API (event = FooEvent(...); canvas.callbacks.process(event.name, event)) which should be safe in all cases.

The process for figure_leave_event should clear LocationEevent.lastevent
If we are going to do this the on-init (!?) processing chain that happen in LocationEvent.init (via _update_enter_leave should be moved to process.

~~I need to look into it, but again I'd rather not have this lastevent at all and just have the GUI backends query the cursor position when the event is triggered and fill in that info correctly.~~
Edit: now done in a separate commit; this is needed for axes_{enter,leave}_event to work.

tacaswell · 2020-03-30T01:00:04Z

but they can at least use the no-helper API

But then they lose the axes enter/leave state management. They are going to have to carry some version gating code for a while.

I guess we could pull all of the "extra" logic back into the canvas and add a "pre-process" step to the call back registry?

so you can't write backend-independent code using them

Why are you trying to write code against these methods? My sense of the design is that the canvas.foo_event were "internal" and should only be called by the backends (which is why the API drifted between the backends) and that users should be writing code against the other side of the callback registry.

timhoffm · 2020-03-30T01:02:06Z

Just a quick thought (not an expert in the event system): Aren‘t events sort of data classes (if that had existed back then)? It feels a bit strange to add process functionality there.

anntzer · 2020-03-30T10:07:14Z

But then they lose the axes enter/leave state management. They are going to have to carry some version gating code for a while.

I guess we could pull all of the "extra" logic back into the canvas and add a "pre-process" step to the call back registry?

Fair enough, I guess we can have that code both in LocationEvent.process (like I did in this PR) and in each of the FigureCanvasBase.location_event(). (In any case, that seems better than having it in the LocationEvent constructor -- big side effects in a class constructor seem just... awkward.)

so you can't write backend-independent code using them

Why are you trying to write code against these methods? My sense of the design is that the canvas.foo_event were "internal" and should only be called by the backends (which is why the API drifted between the backends) and that users should be writing code against the other side of the callback registry.

I'm not actually trying to write code against these methods; my point here was just that end-users can't rely on it (as you say they're basically internal).

The original motivation was to improve event metadata in the case of MouseEvents (#6159). Basically, right now the GUI backends don't fill in the .key attribute of MouseEvents -- because indeed Qt/GTK/Tk/wx expose no API to tell what keys are pressed when a mouse event occurs. Instead, the key attribute is filled based on the last key_{press,release}_event. But that's actually quite brittle, because key presses/releases can also occur while the canvas doesn't have the focus, and these won't be recorded (this isn't an abstract problem, but one I actually encountered, and quite puzzling in fact, because most of the time .key will be correct... but fail in a few times, then you repress the key a few times and "weirdly" it works again, and you wonder whether your keyboard is dying). However, Qt/... expose an API to tell which modifier keys (Ctrl/Alt/Shift) are pressed when a mouse event occurs, so MouseEvent could gain a .modifiers attribute recording that info (and then we may choose later to deprecate .key and tell the users to listen to key_{press,release}_event themselves if they want, but that's another story). With the approach of this PR, one would just need to add a .modifiers parameter/attribute to MouseEvent, and then the backend could just set it and call process() on it. In fact a third-party backend could even choose to implement that independently of Matplotlib by creating its own MouseEvent subclass and then push it to the callback registry. But with the current API this is slightly more annoying, because it needs to go through button_{press,release}_event/etc., which all need to have their signature extended. And then is it a bug if a backend exposes a button_press_event which has no modifiers parameter? (On the other hand it's not really a problem because these methods already have incompatible signatures anyways.)

The situation is similar wrt. figure_leave_event, which currently fills its location based on the last LocationEvent, but really the GUI backends could/should just query the cursor position when the LocationEvent is triggered and fill in that info themselves.

Just a quick thought (not an expert in the event system): Aren‘t events sort of data classes (if that had existed back then)? It feels a bit strange to add process functionality there.

As noted above, initially I had a def FigureCanvasBase.process_event(event): self.canvas.process(event.name, event). One of the issues, though, is that with that approach you need to put all the event-specific process() logic into a big switchyard in that single method, rather than in each separate process() implementation.

Also I don't think that dataclasses means "no methods".

QuLogic · 2020-05-05T03:01:37Z

Seems like this still needs some discussion?

anntzer · 2020-05-05T07:24:18Z

Likely, yes. (I'll wait for the rebase until we can agree on whether we want this or not.)

anntzer · 2020-10-25T13:54:28Z

On a similar note, the approach here would also help with reporting of pressed buttons in the case of motion_notify_event: currently, we just report the button corresponding to the last button_press_event that occurred over the canvas, but all GUI frameworks can actually report all of the buttons that are being pressed while a cursor motion occurs (which may not even include the last button_press_event, if the mouse motion started out of the canvas).

... and rebased.

QuLogic · 2022-07-07T01:25:23Z

So was this ever on a call?

anntzer · 2022-07-07T06:29:38Z

A few times, but last time was a while ago.

tacaswell · 2022-07-11T01:57:05Z

doc/api/next_api_changes/deprecations/16931-AL.rst

+difficult to improve event metadata.
+
+In order to trigger an event on a canvas, directly construct an `.Event` object
+of the correct class and call ``canvas.callbacks.process(event.name, event)``.


Include a note about _process existing?

tacaswell · 2022-07-11T03:03:37Z

My attempt to summarize:

the idea of having a method on the base class per event is / was sound
using the same methods do directly connect to the UI toolkits caused signature instability in between the subclasses
Our events know a lot about the context they are in and can "self actuate" because they know which canvas they came from. On one hand this feels a bit weird, but on the other hand it is a way to make sure that we do not pass events from one canvas to callbacks on the other (might be better if they Events knew less, but it is too late for that)
we are still doing "dead reckoning" of a few things (the last event so we can track leaving / entering axes and figures, attaching keys to button events and buttons to key events)
this gives us a path get let the UI toolkit track that information for us, but does not get us there yet

I'ma little wary of documenting to use canvas.callbacks.process(event.name, event) when that is not what we do internally (because the dead reckoning book keeping is done in Event._process (in the sub-classes implementation or because we actually emit some events 2x but with different names for the enter/leave events (it was like this before). Although in #16931 (comment) it is noted the consensus was to make _process private, I'm leaning the other way now and think we should make it public. It might also make sense to have a single proccess_ui_event(name, evt) on the canvas base-class that merges all the special casing into one place?

I am happy with the motion_notify subscription, doing that work in __init__ (maybe triggering other callbacks!) was not great. There might be some subtle differences in the order of events (I think that you used to get the enter/leave event then the location event, now you will get the location event and then the enter / leave event). I really hope no one is relying on that ordering and do not think it is worth more effort to restore the old behavior (but we should note it).

Over all 👍🏻 and think we should get this in for 3.6.

anntzer · 2022-07-12T20:48:34Z

Thanks for the review. I moved all special-casing in the various _process methods into plain event handler callbacks (in separate commits, which should be hopefully self-contained enough for review), which seems cleaner. How does that look to you? I can still make _process public if you prefer and/or revert back to putting the special-casing into _process overrides.

As for event ordering, I believe user callbacks registered for the axes_enter_event will still be called before user callbacks registered for the motion_notify_event, because the order is now 1) we process the motion_notify_event, and call the the first callback registered to it, which is _axes_enter_leave_emitter 2) this synthesizes and processes the axes_enter/leave_event, and calls all user callbacks registered to it, and finally 3) we return to prcessing the motion_notify_event, calling all remaining callbacks, which are the user callbacks.

lib/matplotlib/figure.py

QuLogic

This seems to work in the backends I tested; TkAgg, QtAgg, Gtk3Agg.

lib/matplotlib/backend_bases.py

Currently, UI events (MouseEvent, KeyEvent, etc.) are generated by letting the GUI-specific backends massage the native event objects into a list of args/kwargs and then call `FigureCanvasBase.motion_notify_event`/`.key_press_event`/etc. This makes it a bit tricky to improve the metadata on the events, because one needs to change the signature on both the `FigureCanvasBase` method and the event class. Moreover, the `motion_notify_event`/etc. methods are directly bound as event handlers in the gtk3 and tk backends, and thus have incompatible signatures there. Instead, the native GUI handlers can directly construct the relevant event objects and trigger the events themselves; a new `Event._process` helper method makes this even shorter (and allows to keep factoring some common functionality e.g. for tracking the last pressed button or key). As an example, this PR also updates figure_leave_event to always correctly set the event location based on the *current* cursor position, instead of the last triggered location event (which may be outdated); this can now easily be done on a backend-by-backend basis, instead of coordinating the change with FigureCanvasBase.figure_leave_event. This also exposed another (minor) issue, in that resize events often trigger *two* calls to draw_idle -- one in the GUI-specific handler, and one in FigureCanvasBase.draw_idle (now moved to ResizeEvent._process, but should perhaps instead be a callback autoconnected to "resize_event") -- could probably be fixed later.

These were only needed back when axes_enter_events were generated in the LocationEvent constructor, but this is now done by a standard callback.

CallbackRegistry already replaces exceptions by printed tracebacks, which seems better than fully suppressing everything.

Backends can call draw_idle themselves. Note that 1) this was already done by the gtk backends, and 2) this may actually be unneeded, as figure.set_size_inches (which is always called a bit earlier by the various resize handlers) also marks the figure as stale, which should trigger a redraw too. Still, let's add the draw_idle calls to be safe, they shouldn't be costly as both draws should get compressed together; we can always investigate removing them later.

tacaswell · 2022-07-21T21:40:58Z

I did the rebase, anyone (including @anntzer ) can merge this when green.

anntzer added this to the v3.3.0 milestone Mar 27, 2020

anntzer force-pushed the reevents branch 11 times, most recently from 44b6055 to b5763fa Compare March 28, 2020 00:12

tacaswell reviewed Mar 28, 2020

View reviewed changes

tacaswell mentioned this pull request Mar 30, 2020

solution: All subclasses of LocationEvent could be used in cbook.callbacks before being fully initialized - issue 15139 #16948

Merged

6 tasks

QuLogic added status: needs comment/discussion needs consensus on next step status: needs rebase labels May 5, 2020

QuLogic modified the milestones: v3.3.0, v3.4.0 May 5, 2020

anntzer force-pushed the reevents branch from b7c09d5 to ee5710c Compare October 25, 2020 19:37

anntzer removed the status: needs rebase label Oct 25, 2020

anntzer force-pushed the reevents branch 2 times, most recently from d2ceeef to 684663c Compare October 27, 2020 11:14

anntzer force-pushed the reevents branch from cb3516a to 6c8773c Compare July 5, 2022 06:22

tacaswell self-assigned this Jul 7, 2022

tacaswell reviewed Jul 11, 2022

View reviewed changes

anntzer force-pushed the reevents branch from 6c8773c to a1b7b7b Compare July 12, 2022 20:42

tacaswell reviewed Jul 13, 2022

View reviewed changes

lib/matplotlib/figure.py Outdated Show resolved Hide resolved

anntzer force-pushed the reevents branch 2 times, most recently from 84f69e0 to 151de3c Compare July 13, 2022 20:52

anntzer mentioned this pull request Jul 16, 2022

[Bug]: Constrained pan/zoom doesn't work on GNOME/Wayland #23436

Open

QuLogic approved these changes Jul 21, 2022

View reviewed changes

lib/matplotlib/backend_bases.py Outdated Show resolved Hide resolved

anntzer force-pushed the reevents branch from 151de3c to eb9bef4 Compare July 21, 2022 11:43

tacaswell approved these changes Jul 21, 2022

View reviewed changes

anntzer added 6 commits July 21, 2022 17:34

Move emission of axes_enter/leave_event to a separate handler.

e26189b

Remove unneeded delayed super-inits.

f265579

These were only needed back when axes_enter_events were generated in the LocationEvent constructor, but this is now done by a standard callback.

Move key/mouse dead reckoning to standard callbacks.

f293fc6

Remove exception suppression in CloseEvent processing.

6fe42ad

CallbackRegistry already replaces exceptions by printed tracebacks, which seems better than fully suppressing everything.

tacaswell force-pushed the reevents branch from eb9bef4 to ff21516 Compare July 21, 2022 21:40

anntzer removed the status: needs comment/discussion needs consensus on next step label Jul 21, 2022

jklymak merged commit 33a0599 into matplotlib:main Jul 22, 2022

anntzer deleted the reevents branch July 22, 2022 16:54

anntzer mentioned this pull request Jul 22, 2022

Separately track modifier keys for mouse events. #23473

Merged

6 tasks

greglucas mentioned this pull request Sep 17, 2022

FIX: use process_event in dpi changes on macosx backend #23925

Merged

dhomeier mentioned this pull request Oct 10, 2022

Fix matplotlib and other deprecations glue-viz/glue#2329

Merged

tacaswell mentioned this pull request Oct 20, 2022

[Bug]: pick events do not forward mouseevent-key on Linux #24199

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make it easier to improve UI event metadata. #16931

Make it easier to improve UI event metadata. #16931

anntzer commented Mar 27, 2020 •

edited

Loading

tacaswell Mar 28, 2020

anntzer Mar 28, 2020

tacaswell Mar 30, 2020

tacaswell commented Mar 28, 2020

anntzer commented Mar 28, 2020 •

edited

Loading

tacaswell commented Mar 30, 2020

timhoffm commented Mar 30, 2020

anntzer commented Mar 30, 2020

QuLogic commented May 5, 2020

anntzer commented May 5, 2020 •

edited

Loading

anntzer commented Oct 25, 2020 •

edited

Loading

QuLogic commented Jul 7, 2022

anntzer commented Jul 7, 2022

tacaswell Jul 11, 2022

tacaswell commented Jul 11, 2022

anntzer commented Jul 12, 2022

QuLogic left a comment

tacaswell commented Jul 21, 2022

	event = KeyEvent(
	s, self, key, self._lastx, self._lasty, guiEvent=guiEvent)

Make it easier to improve UI event metadata. #16931

Make it easier to improve UI event metadata. #16931

Conversation

anntzer commented Mar 27, 2020 • edited Loading

PR Summary

PR Checklist

tacaswell Mar 28, 2020

Choose a reason for hiding this comment

anntzer Mar 28, 2020

Choose a reason for hiding this comment

tacaswell Mar 30, 2020

Choose a reason for hiding this comment

tacaswell commented Mar 28, 2020

anntzer commented Mar 28, 2020 • edited Loading

tacaswell commented Mar 30, 2020

timhoffm commented Mar 30, 2020

anntzer commented Mar 30, 2020

QuLogic commented May 5, 2020

anntzer commented May 5, 2020 • edited Loading

anntzer commented Oct 25, 2020 • edited Loading

QuLogic commented Jul 7, 2022

anntzer commented Jul 7, 2022

tacaswell Jul 11, 2022

Choose a reason for hiding this comment

tacaswell commented Jul 11, 2022

anntzer commented Jul 12, 2022

QuLogic left a comment

Choose a reason for hiding this comment

tacaswell commented Jul 21, 2022

anntzer commented Mar 27, 2020 •

edited

Loading

anntzer commented Mar 28, 2020 •

edited

Loading

anntzer commented May 5, 2020 •

edited

Loading

anntzer commented Oct 25, 2020 •

edited

Loading