WebAgg backend #1426

mdboom · 2012-10-22T18:27:15Z

This adds a WebAgg backend based on the approach I described here:

http://mdboom.github.com/blog/2012/10/11/matplotlib-in-the-browser-its-coming/

It's not complete, but I thought I would post this early to give it lots of browser testing and make sure that the fundamental approach is sound.

This does not integrate with IPython (though that's the plan) -- it is just a standalone backend, and the only additional dependency is Tornado.

To try it out, set your backend to WebAgg, make a figure and call pyplot.show(). By default, a webbrowser tab will be opened showing the figure (though the webbrowser behavior is configurable). Some fun examples to try are examples/animation/double_pendulum_animated.py and examples/event_handling/lasso.py.

Each of the existing figures is available on its own page (just like their own window in a GUI backend), and the URLs are of the form:

http://127.0.0.1:8888/1/
http://127.0.0.1:8888/2/
etc...

I don't have a lot of experience with browser programming, so the CSS and layout is a little clunky. I'd encourage improvements there, as well as any reports of any examples that don't seem to work.

mdboom · 2012-10-22T18:36:06Z

I should add that the "Zoom to Rect" functionality isn't drawing a rectangle yet, but that's coming.

pelson · 2012-10-23T07:56:32Z

lib/matplotlib/animation.py

@@ -500,7 +500,7 @@ class Animation(object):
    '''
    def __init__(self, fig, event_source=None, blit=False):
        self._fig = fig
-        self._blit = blit
+        self._blit = blit and fig.canvas.supports_blit


This seems a little unfriendly. If I ask for blitting, I would expect to get it - otherwise I should get an error. What do you think?

This deserves some explanation. Some backends, such as WebAgg, do not support blitting. In the case of WebAgg, it's because blitting is not possible -- we don't have direct access to the window/screen buffers anyway, so there is no advantage to it. This allows code that was written to expect blitting (such as all of the examples/animation examples) to still work. There should be consistent behavior across backends, even when certain features are not available.

pelson · 2012-10-23T09:42:22Z

Note: Tornado should probably get a mention in the setup "optional dependencies" list.

pelson · 2012-10-23T10:04:40Z

lib/matplotlib/backends/backend_webagg.py

+
+
+class FigureCanvasWebAgg(backend_agg.FigureCanvasAgg):
+    supports_blit = False


As far as I can tell, this isn't a fundamental property of this approach (i.e. in the future, we will be able to blit). Is that correct?

No -- there is no way to blit directly to the screen/window hardware so there is no advantage to that approach. It could be faked, but I'm not sure why...

pelson · 2012-10-23T10:11:37Z

Sadly, using web sockets means I can't use this backend with Firefox 10 (released January 31, 2012) - see http://en.wikipedia.org/wiki/WebSocket#Browser_support

I was able to test your previous tech demo, so I wonder if we can provide a fallback capability to less able browsers?

pelson · 2012-10-23T10:15:54Z

lib/matplotlib/backends/backend_webagg.py

+            self.write_message(diff, binary=True)
+
+    def __init__(self):
+        super(WebAggApplication, self).__init__([


We should add an index page (and maybe a 404 error page), which lists all of the available figures (for bonus points, it would be nice to use the figure.get_label for that).

Sure. I planned to do this, but forgot to mention it.

My plan here is to provide a page with "thumbnails" of all of the open figures (with the figure.get_label beneath it).

Nice idea! Extra bonus points for that! 😄

pelson · 2012-10-23T10:24:51Z

Mike: This is awesome stuff! The approach you have taken here is pragmatic in that it achieves 80% of what is really required (interactive figures in the web browser which are easily shareable to local users). Obviously, the approach falls short of being able to have the figures disconnected from a python/matplotlib server and globally shareable (securely), but realistically that requirement can only be met by re-writing large chunks of matplotlib in JavaScript which really does not sit well for a community driven Python package.

I've left quite a few comments: I hope you find them useful - the majority of them are relating to documentation/commenting. Other than that, I would be in favour of merging this early and labelling it an experimental feature (and getting it out for users to try in 1.3.x).

Great stuff!

mdboom · 2012-10-23T13:57:04Z

I hadn't considered the minimum requirements for WebSockets. Unfortunately, the approach used in the original tech demo made sharing these plots a lot more complicated -- since there was no way to "push" images to the browser, the browsers had to poll occasionally, and to stay in sync (i.e. to not drop frames) full frames would have to be pushed on polling. This approach with WebSockets is a lot cleaner and less likely to fail. But sure, if supporting the older browsers is deemed important, we do have a possible solution for a backward compatibility mode -- but I'd prefer not to have the complication if we can avoid it.

I should also add that the IPython notebook requires websockets, so there is incentive on a number of fronts for users to need a more recent browser.

jenshnielsen · 2012-10-23T14:29:06Z

Note that IPython is considering to replace pure websockets with SockJS which should increase the browser coverage: ipython/ipython#2321

mdboom · 2012-10-23T14:36:03Z

@jenshnielsen: Thanks for the pointer. I'll look into that.

…lib#1426.

mdboom · 2012-10-23T15:30:18Z

Regarding WebSocket browser coverage: It is specifically the binary websocket messages (that are used for the image data) that don't work with Firefox < 11. The text-based messages appear to work fine, with a simple Firefox compatibility hack.

SockJS looks like a nice solution, and the built-in support for broadcasting messages is a nice convenience we could use here. However, it doesn't yet support binary messages, so it doesn't really address our needs. Even if it does support binary messages, I'm not sure there's any way to do it without base64 encoding, which will probably add a lot of overhead. I'm also not too keen, as IPython has done, or including a bunch of external code to make this work... (There might be a way to get pip to require it etc.) I don't want to grow a dependency on IPython for this basic usage, either (nothing wrong with IPython, but I do like that it is easy/simple to get this going without it).

Anyway, I haven't decided either way yet about using SockJS -- that will require some experimentation.

WeatherGod · 2012-10-24T14:04:43Z

Yes, all of the above was done in Firefox 10. The context menu issue is going to be tricky. I am not particularly keen on mapping to the middle button, but that may have to be what happens. We would need to advertise that fact. This may also cause conflicts with any existing code that utilized the middle mouse button already (I have to check to see if mplot3d was using it at all...).

The hanging issue with Firefox might have been spurious. I only noticed it happening very badly once, so it could have been unrelated.

…lib#1426.

pelson · 2012-12-10T09:49:33Z

@mdboom - I'm a big fan of getting new features of such value in quickly for early adopters to pick up and try out (and report bugs before the next release). I'd be willing to put some effort in this week to get this merged - if your keen, would you mind getting this into a state where you are happy with it, and I will do a thorough once over before merging (assuming everybody else is happy with that).

After that, I presume it is about getting similar functionality into the ipython notebook? Have you had an attempt, or are you hopeful that someone from the ipython dev team will meet us half way?

Cheers,

mdboom · 2012-12-10T16:35:38Z

I think this is not far from a shape where it should be merged. I spent a little time trying to support browsers that don't support binary WebSockets, but I'm not sure it's feasible -- it really complicates things. Firefox 17 LTS is now out, superceding the Firefox 10 LTS, so the issue that there's a Firefox LTS that doesn't support this is all but moot now. In any case we should add some browser detection code and warnings if the browser is likely too old.

Other than that, here's my TODO list:

Fix the HTML/CSS to look less clunky. I'm really not an expert in these areas, so hopefully someone can step up there. We can shamelessly steal the CSS from IPython to have a consistent look.

I think we can then merge this before tacking the IPython integration. I'm sure they're excited about this -- but I'm not sure about the best way to go about integrating this. Will need to bring some IPython experts on board.

crbates · 2012-12-15T04:02:59Z

@mdboom As far as I can tell in it's current state it would be rather difficult to integrate this in any direct way to an ipython notebook due to it's reliance on the tornado websockets since in ipython notebooks the kernel running the webserver is different than the notebook kernel, but @ellisonbg is the real expert on the notebook so he would be the one to ask.

This is already incredibly useful though as a way to have an interactive figure in a browser for a remote ipython notebook session. It might be good to change the default port though because currently it can't be used alongside an ipython notebook since they both try to use localhost:8888 and they get confused.

I think any direct integration of matplotlib into an ipython notebook will require the interactivity to be done using js callbacks to the local ipython kernel but I may be wrong.

…lib#1426.

…using image resampling.

…ges.

…8988

WebAgg backend

The current MouseEvent docstring has no mention of this issue so I did not remove anything there. This code was originally added to account for difficulties in preventing the context menu in chrome 23 matplotlib#1426 (comment)

pelson reviewed Oct 23, 2012
View reviewed changes

mdboom added a commit to mdboom/matplotlib that referenced this pull request Oct 23, 2012

Address documentation and smaller issues raised by @pelson in matplot…

7af2b90

…lib#1426.

wkerzendorf mentioned this pull request Nov 14, 2012

Matplotlib html backend support ipython/ipython#2579

Closed

mdboom added a commit to mdboom/matplotlib that referenced this pull request Nov 29, 2012

Address documentation and smaller issues raised by @pelson in matplot…

ef72616

…lib#1426.

mdboom and others added 18 commits January 4, 2013 15:30

Initial work on WebAgg backend

dd32575

mplot3d should call canvas.draw_idle()

31479e6

Address documentation and smaller issues raised by @pelson in matplot…

2f9c992

…lib#1426.

Revert change to how width and height are handled -- the other was ca…

021d74d

…using image resampling.

Handle slightly older versions of Firefox.

914e525

Handle non-binary image streams

0c99763

Display URL if not set to open webbrowser

6e735d5

Remove some logging

d9e68a4

Fix toolbar button events on Firefox.

8193feb

Add message about stopping the server

5efca62

Warn when using a browser that doesn't support binary websocket messa…

bf11a13

…ges.

Add support for rubberband drawing.

8f414b4

Make the html backend prettier with ipython's css and switch port to …

788c8c9

…8988

remove empty ipython notebook

d977828

Add jquery ui hover hook for buttons

499a960

Move status message to a new line

c5479c2

PEP8 fixes. Rename web_static -> web_backend

ae884b0

Indentation

1789f6c

mdboom added a commit that referenced this pull request Jan 4, 2013

Merge pull request #1426 from mdboom/browser_backend

7ed17e3

WebAgg backend

mdboom merged commit 7ed17e3 into matplotlib:master Jan 4, 2013

mdboom deleted the browser_backend branch August 7, 2014 13:52

ianhi mentioned this pull request Aug 4, 2020

webAgg backend incorrectly reports middle clicks as right clicks #18171

Closed

ianhi mentioned this pull request Aug 4, 2020

allow webAgg to report middle click events #18172

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WebAgg backend #1426

WebAgg backend #1426

mdboom commented Oct 22, 2012

mdboom commented Oct 22, 2012

pelson Oct 23, 2012

mdboom Oct 23, 2012

pelson commented Oct 23, 2012

pelson Oct 23, 2012

mdboom Oct 23, 2012

pelson commented Oct 23, 2012

pelson Oct 23, 2012

mdboom Oct 23, 2012

mdboom Oct 23, 2012

pelson Oct 23, 2012

pelson commented Oct 23, 2012

mdboom commented Oct 23, 2012

jenshnielsen commented Oct 23, 2012

mdboom commented Oct 23, 2012

mdboom commented Oct 23, 2012

WeatherGod commented Oct 24, 2012

pelson commented Dec 10, 2012

mdboom commented Dec 10, 2012

crbates commented Dec 15, 2012



		class FigureCanvasWebAgg(backend_agg.FigureCanvasAgg):
		supports_blit = False

WebAgg backend #1426

WebAgg backend #1426

Conversation

mdboom commented Oct 22, 2012

mdboom commented Oct 22, 2012

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pelson commented Oct 23, 2012

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pelson commented Oct 23, 2012

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pelson commented Oct 23, 2012

mdboom commented Oct 23, 2012

jenshnielsen commented Oct 23, 2012

mdboom commented Oct 23, 2012

mdboom commented Oct 23, 2012

WeatherGod commented Oct 24, 2012

pelson commented Dec 10, 2012

mdboom commented Dec 10, 2012

crbates commented Dec 15, 2012