This pull request passes (merged 99b9ae77 into 3c8d448).

This was meant to be just an experiment to see what would be involved in switching over to SockJS, and it turns out it was super easy. After dropping in the JS and tornado code, it was only about an hour of fiddling to make the changes necessary for it to be basically functional.

This pull request passes (merged 68a2c5fa into 3c8d448).

I tried my own attack for unauthorized execution, and it seemed to be behaving exactly as expected (websocket connection closes with wrong cookie password), so no change seems necessary there.

I tested, and can confirm that execution works in IE9 (CSS is obviously wonky), as well as behind an Apache reverse proxy.

@minrk, do you want this one merge or was is just for experimenting ?
If the former could you rebase ?

I think we should do this, but it is certainly open to discussion, as it is a significant change.

This pull request passes (merged 1bcc1b6 into c678a87).

This pull request passes (merged e2f8f50 into c678a87).

Is IPython going to stop supporting websocket when this is merged? I know it sounds selfish, but let me say that I really hope you don't merge this if that is the case, because my Emacs client will not work.

@tkf - SockJS still uses websockets when they are available, but your client will definitely need to be updated since there is an extra handshake, etc.

@tkf - in fact, it looks to be just a trivial URL change.

Thanks for the link. I should have read the page more carefully. So I guess all I need to do is to connect to something like

ws://127.0.0.1:8888/kernels/UUID/shell/websocket

instead of

ws://127.0.0.1:8888/kernels/UUID/shell

BTW, in the page you mentioned it says:

You can't open more than one SockJS connection to one domain at the same time due to the browsers limit of consurrent connections

Is this limitation fine for IPython? It needs two websocket connections (shell and iopub), right?

Is this limitation fine for IPython? It needs two websocket connections (shell and iopub), right?

I saw that, but I don't fully understand it, as everything appears to work fine. It may warrant further investigation, as that limitation probably only comes up when websockets are unavailable (even though I have tested this as well, and it still works everywhere I have tried).

Added merged subclass, so that it can be used with only one SockJS connection (separate websocket connections still work the same).

This pull request passes (merged bbfe2c7 into c678a87).

So how was this working before when you tested it?

On Tue, Sep 4, 2012 at 4:22 PM, The Travis Bot [email protected]:

This pull request passeshttp://travis-ci.org/ipython/ipython/builds/2338061(merged
bbfe2c7 bbfe2c7f into c678a87c678a878
).

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2321#issuecomment-8282544.

Brian E. Granger
Cal Poly State University, San Luis Obispo
[email protected] and [email protected]

Before, it worked no differently to current master, just with SockJS handling fallback if websockets were unavailable. Now a single web socket connection will handle everything instead of two. I was able to artificially induce the failing case pointed out by @tkf with some custom config in Firefox (it does not seem possible with default config in FF or Chrome).

I haven't had a chance to look at the code. Does this always use 1
websocket, or just some of the time? Do you think there are disadvantages
to using just one of them? We have other channels to hook up (like the
stdin) and I don't want to make our lives more difficult later.

On Tue, Sep 4, 2012 at 11:33 PM, Min RK [email protected] wrote:

Before, it worked no differently to current master, just with SockJS
handling fallback if websockets were unavailable. Now a single web socket
connection will handle everything instead of two. I was able to
artificially induce the failing case pointed out by @tkfhttps://github.com/tkfwith some custom config in Firefox (it does not seem possible with default
config in FF or Chrome).

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2321#issuecomment-8288940.

Brian E. Granger
Cal Poly State University, San Luis Obispo
[email protected] and [email protected]

Kernel.js only uses one websocket connection. It's pretty trivial to switch on a key in the message, so I don't think there is any real difficulty extending multiple channels on a single connection. That said, I wouldn't do this if SockJS didn't require it, so if we stick to requiring pure websockets, a 1:1 channel system is cleaner.

The clear advantage of this: it works behind proxies, etc. in the many environments where websockets don't. It's simply a matter of whether we want to wait for systems to support websockets or not, and how long we expect that to take. Making the move to sockjs is extremely easy, so we can do it at any later point if we want to give up on waiting, etc.

I didn't set out to make a PR, but just reading about how it might work resulted in fully working code, so I thought it might be worth it.

On Wed, Sep 5, 2012 at 4:01 PM, Min RK [email protected] wrote:

The clear advantage of this: it works behind proxies, etc. in the _many_environments where websockets don't.

I think that's an advantage we shouldn't discount lightly: schools, an
environment where we really want ipython to work very well, often have
proxies. Since the added complexity doesn't seem that significant (a key
switch on messages, if I understand @minrk correctly), I'm slightly in
favor.

I am not as worried about older browsers, but the proxy issue is huge.
That will also affect many companies (including MSFT). I will try to have
a look at the code, but I think we should probably go in this direction.

On Wed, Sep 5, 2012 at 7:17 PM, Fernando Perez [email protected]:

On Wed, Sep 5, 2012 at 4:01 PM, Min RK [email protected] wrote:

The clear advantage of this: it works behind proxies, etc. in the
_many_environments where websockets don't.

I think that's an advantage we shouldn't discount lightly: schools, an
environment where we really want ipython to work very well, often have
proxies. Since the added complexity doesn't seem that significant (a key
switch on messages, if I understand @minrk correctly), I'm slightly in
favor.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2321#issuecomment-8320338.

Brian E. Granger
Cal Poly State University, San Luis Obispo
[email protected] and [email protected]

One more thing I can change:

For backwards compatibility, the individual IOPub / Shell channels still exist, and are usable as pure websockets (not tested, but should be true according to sockjs spec). However, they, too, are SockJS, so the raw websocket URLs will be <kernel-id>/shell/websocket instead of <kernel-id>/shell. If I refactored it a little bit, I could make the original urls available as plain websocket handlers again, so only the merged single handler uses SockJS.

I don't view the URL scheme of web service of the notebook as being
anywhere close to stable. I think it it definitely to early to worry about
backwards compatibility in this area so lets just use the best design we
can. As we add multiuser capabilities and live-notebook sharing, all of
these URLs are going to change anyways. If I understand things correctly,
we should probably just move everything to a single websocket/sockjs
connection and get rid of the two socket URLs, etc.

On Fri, Sep 7, 2012 at 11:16 AM, Min RK [email protected] wrote:

One more thing I can change:

For backwards compatibility, the individual IOPub / Shell channels still
exist, and are usable as pure websockets (not tested, but should be true
according to sockjs spec). However, they, too, are SockJS, so the raw
websocket URLs will be /shell/websocket instead of
/shell. If I refactored it a little bit, I could make the
original urls available as plain websocket handlers again, so only the
merged single handler uses SockJS.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2321#issuecomment-8373872.

Brian E. Granger
Cal Poly State University, San Luis Obispo
[email protected] and [email protected]

Hi, I'm interested in trying this, but am still a git noob. How do I merge this onto a ipython and what version? Thank you for this work. We need it badly to keep ipython alive for us!

Hi @Sesshomurai ,

Which version are you currently using ?
Merging is not always straitforward.

What you can do is try directly the state of this branch, that for example you can download it as a zip file autogenerated by github. I got the link by going to min's profile https://github.com/minrk/ipython go to branches, then sockjs then zip link in the headers.

If you want to do it with git, add a remote

git remote add minrk https://github.com/minrk/ipython.git

fetch it

git fetch minrk

now you can try to merge minrk/sockjs

git merge minrk/sockjs

But please be sure to understand what you do, or ask for clarification.

You could also get a look at https://gist.github.com/3342247
which allow you to simply refer to pull request by their number. which would be

git merge origin/pr/2321

don't hesitate to ask if you need help, and also https://help.github.com/ is a good place to start learning git.

Thanks so much for the response. I don't need to merge if its not necessary, I just want a complete ipython with this capability embedded. So I will try the autogenerated zip and see.

I also didn't understand this part "...then sockjs then zip link in the headers...." Do I need to layer in other code from elsewhere?

Longer version would have been:

Go to the 'branch' tab. Select the 'sockjs' branch, now the 'zip' button in
the github header bar allow you to download a snapshot of the sockjs
branch. This is where i got the link i embed in my previous answer.

Does it make more sens?
Le 15 sept. 2012 16:01, "Sesshomurai" [email protected] a écrit :

I also didn't understand this part "...then sockjs then zip link in the
headers...." Do I need to layer in other code from elsewhere?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2321#issuecomment-8584966.

I haven't noticed this issue before. Just FYI, we use SockJS in the Sage Cell server to communicate with an IPython kernel. I haven't had time to look at this implementation, but just in case it's useful, here are quick links to our code:

javascript:

https://github.com/sagemath/sagecell/blob/master/static/compute_server.js#L85

https://github.com/sagemath/sagecell/blob/master/static/compute_server.js#L989

server:
https://github.com/sagemath/sagecell/blob/master/handlers.py#L105

this does not merge cleanly anymore...

As this is pretty big, I would suggest cleaning it up and merging as it only improve things.
(when @minrk have time of course, good luck BTW)

rebased.

One issue that we may want to address, alluded to by @jasongrout, is the restriction that SockJS requires only one active connection per client. This will limit the ability to connect to multiple kernels from one webpage. The restriction only actually applies when the websocket transport is unavailable.

@minrk do you know if SockJS allows regular expression matching is the
handler URLs? This is something I think we will need.

On Mon, Jan 14, 2013 at 4:00 PM, Min RK [email protected] wrote:

rebased.

One issue that we may want to address, alluded to by @jasongrouthttps://github.com/jasongrout,
is the restriction that SockJS requires only one active connection per
client. This will limit the ability to connect to multiple kernels from one
webpage. The restriction only actually applies when the websocket transport
is unavailable.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2321#issuecomment-12246895.

Brian E. Granger
Cal Poly State University, San Luis Obispo
[email protected] and [email protected]

@ellisonbg I'm not sure what you mean. regex matching is already being used in the SockJS URLs.

I looked at the code and I see that there is not native regular expression
handling, but you have added it in.

On Mon, Jan 14, 2013 at 7:19 PM, Min RK [email protected] wrote:

@ellisonbg https://github.com/ellisonbg I'm not sure what you mean.
regex matching is already being used in the SockJS URLs.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2321#issuecomment-12251820.

Brian E. Granger
Cal Poly State University, San Luis Obispo
[email protected] and [email protected]

Ah, I hadn't remembered doing that.

rebased

I'm going to see if I can integrate the Sage MultiSockJS code, so that this will work with multiple kernels on a single page. I don't like how it changes the wire format from {JSON} to kernel_id/channel,{JSON},
but I don't really see a good way around that. @ellisonbg what do you think about that?

I really don't like the idea of taking things that belong in kernel related URLs (kernel_id, username, etc) and putting them into the message format. It breaks the main abstraction of the web, which is that resources are addressable using URLs. Some problems with this:

1. As we have more complex URLs, we have to add those things too to the message spec. If we have usernames in the URLs, those have to be moved to the message spec. These things just don't belong there.
2. It makes these channels much more vulnerable to certain kinds of attack. If someone tries to get a WebSocket connection to /kernels/kid/iopub for a kid that doesn't exist, the server returns a simple 404 - the protocol upgrade to WebSocket never occurs and the client doesn't have a chance to start sending WebSocket messages. This means that you have to know what a valid kid is to even get a WebSocket connection. This is not security, but it minimizes the attack surface. If/when we add usernames to our URLs /ellisonbg/kernels/kid/iopub, that attack surface decreases even more - someone has to know my username and my particular kid to get a WebSocket connection.

This also relates to the inability to do native URL regular expression in SockJS - you have to make the full WebSocket connection and then do your own URL parsing.

I really want to like SockJS, but I am not excited about the changes it forces us to make. But maybe we should IRC about this, as it is pretty important.

Based on IRC discussion, I won't do the MultiSockJS work. As proxy support for WebSockets improves (HA Proxy and node-http-proxy so far, nginx sort of / soon), the case for SockJS diminishes.

I will leave this here for now, but at current pace, I expect WebSocket support to improve quickly enough that we won't need to make the compromises that SockJS would force upon us.

This was just an experiment to see how easy integrating SockJS would be, anyhow.

Just curious: is the IRC discussion archived anywhere?

I too see SockJS as a temporary solution (in fact, the premise behind SockJS is that it is a temporary solution until websocket support matures). But it is an important hole for us to fill, as we are supporting legacy clients.

We are less worried about legacy clients - by now all browsers have very good WebSocket support. You could argue that some people still use older browsers, but it is safe to say that those users are not our target market. The main area where we have run into problems with WebSockets is in the proxy support. But this is improved quicklyi and there are already some really good solutions. The only issue right now is that it forces people to choose a supported proxy, rather than letting them choose their own.

On our end, we should record how many times sockjs uses a websocket connection vs. some other method.

The gist of the IRC discussion was:

drawbacks of SockJS:

• we can't use url-based scheme, because we have to serve everything via one SockJS connection
• we have to cram things into the message spec that we don't think belong there (kernel, channel)
• some security/performance issues associated with cramming all channels in one connection.

drawbacks of WebSockets:

• don't work on old browsers (don't much care - even current IE works)
• don't work in various proxy environments

The drawbacks of SockJS aren't going to change, and the drawbacks of WebSockets are only getting smaller. The decision to go with SockJS is really one of: do we need to work in a proxy environment before the necessary proxy environment adds support for WebSockets. Six months ago, I thought the answer was probably yes. Now, I think it's probably no.

Should we close this and open an issue to track the it. That will allow us to keep our open PRs on work that is actively moving foward.

Sure. Closing here and opening as #2822