Arbitrary figure customization hooks. #22316

anntzer · 2022-01-25T14:22:45Z

PR Summary

MEP22 was intended to provide a way to customize user interactions,
in particular with the toolbar via the implementation to "tools". In
practice, it remains currently difficult to add general customizations;
for example, the color-vision deficiency simulator recently proposed
does not fit in the MEP22 framework. Moreover, there is no way to
register MEP22 tools to be added to all figures.

This patch proposes an alternative approach for toolbar customization:
it adds a rcParam (figure.hooks) which is a list of callables
(actually, of "modulename:functioname" strings, so that they can
be specified in the matplotlibrc file) that get called whenever
plt.figure() creates a figure is instantiated; each of the callable gets
the figure as parameter and can modify it as it sees fit (note that it
is equivalent to pass the figure or the toolbar as parameter, as one
can reach one from the other; passing the figure was deemed nicer).
This makes it easy to distribute such customizations as plain python
modules than can be installed from PyPI. Also note that figure hooks
are intentionally not applied when figures are created without going
through plt.figure -- when embedding, one can easily explicitly call
the hooks directly on the figure (this is one reason why passing the
figure is nicer than passing the toolbar).

As an example, the color-vision deficiency simulator is modified to use
this hook (see docstring of the user_interfaces/mplcvd.py example).
The advantage of this approach is that arbitrary modifications to the
toolbar become possible (here, adding a menu-in-a-toolbar), but this
naturally means that each GUI toolkit needs its own code. Likely we
will need to provide "template" implementations that can be copied
by third-parties. (There is also some functionality currently only
available as private API, as mentioned in comments; one such API is
_icon, which provides theme-dependent icon recolorization. These APIs
should be made public in some form, but that should be doable.)

(One should check to what extent this approach is generalizable to
the macos and notebook backends -- customizations to the former could
possibly be implemented via PyObjC(?), and the latter via js injection?)

PR Checklist

Tests and Styling

Has pytest style unit tests (and pytest passes).
Is Flake 8 compliant (install flake8-docstrings and run flake8 --docstring-convention=all).

Documentation

New features are documented, with examples if plot related.
New features have an entry in doc/users/next_whats_new/ (follow instructions in README.rst there).
API changes documented in doc/api/next_api_changes/ (follow instructions in README.rst there).
Documentation is sphinx and numpydoc compliant (the docs should build without error).

timhoffm · 2022-01-25T21:00:45Z

I'm somewhat worried that this is too broad and cannot live up to expectations.

First, a hook into toolbar creation lets the user do anything. Do we really need that? It feels a bit like the sys.set_coroutine_wrapper issue in the standard library (https://bugs.python.org/issue32591).

Then, do we need multiple functions? I assume the idea is that multiple should be added to the toolbar and they should not compete for the single hook function. OTOH since they can do anything with the toolbar, hooks from different sources can still interact badly. - This is calling for "the toolbar is broken" bug reports. I have no solution for this problem either.

Implementors need to explicitly write code for each backend. That's a big obstacle. Likely very few people know multiple UI frameworks.

Overall, a hook is temptingly simple to implement on our side. And we can tell people "you can do anything you want, just implement this hook". But I see potential issues and I question how useful it actually is. Maybe this is one of the cases where we should be defensive and say "sorry this is beyond the scope of core matplotlib. If you really need fancy toolbars, embed a figure in your own GUI application."

anntzer · 2022-01-26T11:09:06Z

I'm somewhat worried that this is too broad and cannot live up to expectations.

I agree this is a big hammer but thought it was worthwhile at least bringing it up for discussion.

First, a hook into toolbar creation lets the user do anything. Do we really need that? It feels a bit like the sys.set_coroutine_wrapper issue in the standard library (bugs.python.org/issue32591).

A more targeted approach was tried (MEP22), but I personally think its API is quite limiting (for example, the CVD tool UI cannot be implemented with it).

Then, do we need multiple functions? I assume the idea is that multiple should be added to the toolbar and they should not compete for the single hook function. OTOH since they can do anything with the toolbar, hooks from different sources can still interact badly. - This is calling for "the toolbar is broken" bug reports. I have no solution for this problem either.

I certainly think this would be much more useful with support for multiple hooks. Another example of tool would be control of savefig options (#7696), or some kind of ruler tool (#7216), and it would seem quite arbitrary to only allow one of them at a time. (Yes, some of these tools could be bound to keyboard shortcuts only and not have a toolbar entry at all, but the same is true for everything on the toolbar: having a clickable button is more discoverable.)

Implementors need to explicitly write code for each backend. That's a big obstacle. Likely very few people know multiple UI frameworks.

You don't have to support all UI frameworks to distribute your hook; it is entirely acceptable to to write hooks that only support a single framework. Matplotlib can provide generic helpers for common tasks (adding a button at the right place, colorizing the icon while respecting the desktop theme), but trying to fit everything into a generic helper API runs into the MEP22 limitations mentioned above.

Overall, a hook is temptingly simple to implement on our side. And we can tell people "you can do anything you want, just implement this hook". But I see potential issues and I question how useful it actually is. Maybe this is one of the cases where we should be defensive and say "sorry this is beyond the scope of core matplotlib. If you really need fancy toolbars, embed a figure in your own GUI application."

There are tools which seem generic enough (CVD, savefig options, ruler) that it seems rather limiting to say "you can only have them in a full GUI application". In any case, there seems to have been demand for toolbar customization previously (i.e., MEP22) and the point of the proposal here is that IMO (and as I already argued elsewhere), the API of MEP22 was not really optimal; let's just say this is how I would have skinned this cat.

jklymak · 2022-01-26T12:31:33Z

I guess if the user has to specifically opt into the toolbar by importing a module and editing an rcParam, that would be manageable from the expectations point of view. i.e. bug reports would go to toolbar plugin authors. I guess I am somewhat convinced that it would be worthwhile to have an ecosystem of plugins that can add GUI features, so long as we do not need to support them directly.

Is it clear to folks that mplcvd:setup will import mycvd? Why does the user need to know about setup? If the user doesn't specify a module, will it try a method from the current namespace?

anntzer · 2022-01-26T12:38:21Z

Is it clear to folks that mplcvd:setup will import mycvd? [...] If the user doesn't specify a module, will it try a method from the current namespace?

We can spell that module://mplcvd:setup if we want to be extra explicit that this is a third-party import (like for backends); I don't really care. But note that in this case every entry would be an import (we don't have to special-case builtin tools), so perhaps the module:// is redundant.
We also don't need to support moduleless cases. (Well, we can, but we don't have to -- right now I explicitly only support strings (names) as inputs, and they have to be fully qualified.)

Why does the user need to know about setup?

We could instead say "you should give a module name and we will call a function named setup (or mpl_setup_toolbar...) in that module" but that seems more constraining, e.g. this prevents providing two tools in the same module.

timhoffm · 2022-01-26T22:29:41Z

I see the arguments. OTOH there's only so much you can reasonably do with a toolbar beyond adding buttons. We should be sure the big hammer is actually needed. If it's only the menu thing, we could consider providing that ourselves. Yes, it's a bit more work on our side, but would make it easier to use for tool developers. Overall this needs very careful consideration.

anntzer · 2022-01-27T21:09:38Z

General agreement during the call is to make this figure.hook and get called with the figure as sole argument (from which you can get access to the toolbar) at the end of new_figure_manager(). Also needs some docs/guidelines. One thing to mention is that in a figure hook you can also register a single-shot callback for the first draw time.

anntzer · 2022-01-27T23:34:16Z

Switching to figure.hooks was easy enough. Docs still todo.

anntzer · 2022-02-07T08:17:18Z

Any thoughts on where the docs should go? (This connects back to the discussion of a few weeks ago about having a place specifically to document rcParams.) Right now I'm considering explaining this in the plt.figure docstring (hence #22420), as that's after all the only place where figurehooks will be applied.

anntzer · 2022-02-08T23:23:04Z

Added some (very) minimal documentation.

tacaswell · 2022-04-22T22:09:55Z

Taking a broader view, the goal here is to make it easier to add buttons to our toolbar . This is something there is definitely demand for (I suspect that this is one of the biggest drivers for people who go down the full embedding route) and have a couple of example from my domain. Any application where you are using the figure as a data input (feature picking, manual masking, measurement, etc) you are going to want the ability to hook up a bunch of stateful callbacks to multiple of the event types we expose and then have a toggle to turn the lot of them on/off while you are using the UI.

The UX idioms around this sort of thing are pretty well established (push buttons for one-off things, toggle buttons for things you want to be state fully on/off and radio groups for mutually exclusive things) and we already have code in the library to implement all of these things for all of the toolkits we support (as we use all of the idioms in the stock toolbar), however we do not expose this to the users as public API (because writing a cross-GUI framework toolkit is a slippery slope and not something we want to get too far into).

Currently to get a customized toolbar users have a couple of options:

extend their favorite backend to replace the tool bar class + use pyplot with a custom backend
write a function like enrich_figure(fig) (and monkey patch plt.figure to run it interactively) which grabs the toolbar and does things to it dynamically
go full embedding and use their own toolbar (maybe extending ours?)
subclass figure to be aware of the toolbar

None of these are particularly nice, but people make it work. In all of these cases the user is currently specializing to a particular backend. User may also side-step this issue and always use the prompt to set up their tool via a function and/or wrapping up figure creation, initial plotting, and tool plumbing into one function

Providing an easier way to get these sort of things working is a good idea. That said, while we are targeting updating the toolbar here, I am already starting to think what else you could do with this. The main thing I can think of is installing callbacks that do not necessarily get a toolbar entry, but I could also imagine adding whole other toolbars, auto-docking the figures into another window (only makes sense in some cases), or controlling exactly where it goes on the screen etc

I do see the comparison to the hook on coroutines, however I think we can afford to be more permissive that the interpreter. coroutine objects fundamentally belong to the interpreter / asyncio framework and from the user side there is not a whole lot you want to do with them. On the other hand in our case I think of the Figure objects as something that belongs to the user so we can give them more power on the creation.

I also think that this has shadows of the long running rcparams vs some python settings discussion and shades of declarative vs imperative programming. By going more declarative (like rcparams) we can constrain the user and be more sure that the code will work as expected, but if the user wants to do something we did not give them a knob for they can not (well, it is harder). On the other hand if we have a lot of extension points that are "give us a function with " then we are way more flexible, but we give up a lot of the gaurentees we can make about "it will work" because 🤷🏻 who knows what code the user just gave us.

Likely very few people know multiple UI frameworks.

and we have collected most of them to work on mpl already 😬

In practice, it remains currently difficult to add general customizations;
for example, the color-vision deficiency simulator recently proposed
does not fit in the MEP22 framework.

I do not understand why this is. It does not support a drop-down out of the box (this example implements all of them) but it looks like it has a way to spell radio buttons out of the box. I will take a shot at getting this to work...

Moreover, there is no way to register MEP22 tools to be added to all figures.

This seems fixable via an rcParams and/or some way to control

I think I am sold on the figure.hooks with'module:function' as the entries in the list, but I am not sold that this is an alternative to MEP22 rather than complimentary.

anntzer · 2022-04-23T10:49:32Z

I do not understand why this is. It does not support a drop-down out of the box (this example implements all of them) but it looks like it has a way to spell radio buttons out of the box. I will take a shot at getting this to work...

Agreed this is also doable with radio buttons, although the UI would not look as nice (but that's a relatively less crucial point).

Moreover, there is no way to register MEP22 tools to be added to all figures.

This seems fixable via an rcParams and/or some way to control

I guess what would be needed is to transform

matplotlib/examples/user_interfaces/toolmanager_sgskip.py

Lines 77 to 90 in e077394

    
           # Add the custom tools that we created 
        
           fig.canvas.manager.toolmanager.add_tool('List', ListTools) 
        
           fig.canvas.manager.toolmanager.add_tool('Show', GroupHideTool, gid='mygroup') 
        
           # Add an existing tool to new group `foo`. 
        
           # It can be added as many times as we want 
        
           fig.canvas.manager.toolbar.add_tool('zoom', 'foo') 
        
           # Remove the forward button 
        
           fig.canvas.manager.toolmanager.remove_tool('forward') 
        
           # To add a custom tool to the toolbar at specific location inside 
        
           # the navigation group 
        
           fig.canvas.manager.toolbar.add_tool('Show', 'navigation', 1)

into some kind of declarative language (and even then, ListTools and GroupHideTools would have to become (effectively) string references to classes provided by third-parties. Certainly not impossible, but (as you may know) I personally tend to prefer "just writing code" instead of inventing new DSLs :-)

I think I am sold on the figure.hooks with'module:function' as the entries in the list, but I am not sold that this is an alternative to MEP22 rather than complimentary.

IIRC the general agreement at the last call where we discussed this was to indeed keep trying on providing MEP22ish generic helpers that could be reused by third parties to implement (simpler) tools.

jklymak · 2022-06-23T07:47:08Z

This looks like it has consensus? However, it needs docs etc, so I'll move to draft...

anntzer · 2022-06-23T10:16:35Z

There is a reference doc at https://github.com/matplotlib/matplotlib/pull/22316/files#diff-d1edb5a528e7423f485b24cac2083d05d365a19352f886bf15af3ef442bdda92R767, which also links to the example. I don't think we really have a better place to document rcParams (so putting this back to "ready for review"), but would be happy to be proven wrong?... Feel free to put it back to draft if you disagree.

tacaswell

The footprint on the library is tiny. I am in favor of getting this in for 3.7.

MEP22 was intended to provide a way to customize user interactions, in particular with the toolbar via the implementation to "tools". In practice, it remains currently difficult to add general customizations; for example, the color-vision deficiency simulator recently proposed does not fit in the MEP22 framework. Moreover, there is no way to register MEP22 tools to be added to all figures. This patch proposes an alternative approach for toolbar customization: it adds a rcParam (`figure.hooks`) which is a list of callables (actually, of "modulename:functioname" strings, so that they can be specified in the matplotlibrc file) that get called whenever plt.figure() creates a figure is instantiated; each of the callable gets the figure as parameter and can modify it as it sees fit (note that it is equivalent to pass the figure or the toolbar as parameter, as one can reach one from the other; passing the figure was deemed nicer). This makes it easy to distribute such customizations as plain python modules than can be installed from PyPI. Also note that figure hooks are intentionally *not* applied when figures are created without going through `plt.figure` -- when embedding, one can easily explicitly call the hooks directly on the figure (this is one reason why passing the figure is nicer than passing the toolbar). As an example, the color-vision deficiency simulator is modified to use this hook (see docstring of the `user_interfaces/mplcvd.py` example). The advantage of this approach is that arbitrary modifications to the toolbar become possible (here, adding a menu-in-a-toolbar), but this naturally means that each GUI toolkit needs its own code. Likely we will need to provide "template" implementations that can be copied by third-parties. (There is also some functionality currently only available as private API, as mentioned in comments; one such API is `_icon`, which provides theme-dependent icon recolorization. These APIs should be made public in some form, but that should be doable.) (One should check to what extent this approach is generalizable to the macos and notebook backends -- customizations to the former could possibly be implemented via PyObjC(?), and the latter via js injection?)

anntzer · 2022-12-17T15:21:58Z

Thanks for the bump and additions. I also got rid of trying to set mplcvd.__version__.

anntzer added New feature status: waiting for other PR topic: toolbar status: needs comment/discussion needs consensus on next step status: needs documentation labels Jan 25, 2022

anntzer force-pushed the toolbarhooks branch from 895a30e to 5f895a8 Compare January 27, 2022 09:56

anntzer removed the status: waiting for other PR label Jan 27, 2022

anntzer changed the title ~~Arbitrary toolbar customization hooks.~~ Arbitrary figure customization hooks. Jan 27, 2022

anntzer force-pushed the toolbarhooks branch from 5f895a8 to f1d2f2c Compare January 27, 2022 23:33

tacaswell added this to the v3.6.0 milestone Jan 28, 2022

jklymak marked this pull request as draft January 28, 2022 08:15

anntzer force-pushed the toolbarhooks branch from f1d2f2c to 6515cbc Compare February 8, 2022 23:19

anntzer force-pushed the toolbarhooks branch from 6515cbc to b85b9ed Compare February 8, 2022 23:23

anntzer marked this pull request as ready for review April 21, 2022 20:00

anntzer force-pushed the toolbarhooks branch from b85b9ed to 19f5d8a Compare April 21, 2022 20:00

tacaswell self-requested a review April 21, 2022 20:01

jklymak marked this pull request as draft June 23, 2022 07:47

anntzer marked this pull request as ready for review June 23, 2022 10:16

tacaswell modified the milestones: v3.6.0, v3.7.0 Aug 19, 2022

github-actions bot added the status: needs rebase label Oct 18, 2022

anntzer force-pushed the toolbarhooks branch from 19f5d8a to 2c054de Compare October 22, 2022 16:45

github-actions bot removed the status: needs rebase label Oct 22, 2022

tacaswell modified the milestones: v3.7.0, v3.8.0 Dec 16, 2022

github-actions bot added the status: needs rebase label Dec 16, 2022

tacaswell approved these changes Dec 16, 2022

View reviewed changes

tacaswell force-pushed the toolbarhooks branch from a3a4957 to 424d1ce Compare December 16, 2022 22:23

tacaswell removed the status: needs documentation label Dec 16, 2022

github-actions bot removed the status: needs rebase label Dec 16, 2022

anntzer removed the status: needs comment/discussion needs consensus on next step label Dec 17, 2022

anntzer and others added 3 commits December 17, 2022 16:21

DOC: add demo code to example

e6eecba

TST: add a test of figure.hooks

ac5f694

anntzer force-pushed the toolbarhooks branch from 424d1ce to ac5f694 Compare December 17, 2022 15:21

ksunden approved these changes Dec 23, 2022

View reviewed changes

ksunden merged commit a9e4f38 into matplotlib:main Dec 23, 2022

anntzer deleted the toolbarhooks branch December 23, 2022 20:43

anntzer mentioned this pull request Feb 2, 2023

Add colour vision deficiency simulation #20649

Closed

7 tasks

Uh oh!

Arbitrary figure customization hooks. #22316

Arbitrary figure customization hooks. #22316

Uh oh!

Conversation

anntzer commented Jan 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

PR Checklist

Uh oh!

timhoffm commented Jan 25, 2022

Uh oh!

anntzer commented Jan 26, 2022

Uh oh!

jklymak commented Jan 26, 2022

Uh oh!

anntzer commented Jan 26, 2022

Uh oh!

timhoffm commented Jan 26, 2022

Uh oh!

anntzer commented Jan 27, 2022

Uh oh!

anntzer commented Jan 27, 2022

Uh oh!

anntzer commented Feb 7, 2022

Uh oh!

anntzer commented Feb 8, 2022

Uh oh!

tacaswell commented Apr 22, 2022

Uh oh!

anntzer commented Apr 23, 2022

Uh oh!

jklymak commented Jun 23, 2022

Uh oh!

anntzer commented Jun 23, 2022

Uh oh!

tacaswell left a comment

Choose a reason for hiding this comment

Uh oh!

anntzer commented Dec 17, 2022

Uh oh!

Uh oh!

anntzer commented Jan 25, 2022 •

edited

Loading