Install the ZMQ prebuilt binaries and verify ZMQ works on startup #10551

rchiodo · 2020-03-13T00:39:57Z

For #10483

rchiodo · 2020-03-13T00:47:54Z

news/3 Code Health/10483.md

@@ -0,0 +1 @@
+Add ZMQ library to extension


Tested on windows and linux (ubuntu 18)

DonJayamanne · 2020-03-13T01:56:03Z

build/webpack/webpack.extension.config.js

+        new copyWebpackPlugin([{ from: './node_modules/zeromq/**/*.js' }]),
+        new copyWebpackPlugin([{ from: './node_modules/zeromq/**/*.node' }]),
+        new copyWebpackPlugin([{ from: './node_modules/zeromq/**/*.json' }]),
+        new copyWebpackPlugin([{ from: './node_modules/node-gyp-build/**/*' }])


Does this work? Will this be cross play?
I thought we'd be using the prebuilt binaries, instead of using npm install?

This is installing the prebuilt binaries into the node_modules directory. The '.node' files are the prebuilt binaries.

Then they end up in the vsix this way

Otherwise it wouldn't have worked on my linux machine

codecov-io · 2020-03-13T05:45:09Z

Codecov Report

Merging #10551 into master will increase coverage by 0.01%.
The diff coverage is 82.45%.

@@            Coverage Diff             @@
##           master   #10551      +/-   ##
==========================================
+ Coverage   60.76%   60.77%   +0.01%     
==========================================
  Files         578      579       +1     
  Lines       31355    31403      +48     
  Branches     4464     4469       +5     
==========================================
+ Hits        19052    19085      +33     
- Misses      11335    11346      +11     
- Partials      968      972       +4

Impacted Files	Coverage Δ
src/client/telemetry/index.ts	`87.38% <ø> (ø)`
.../datascience/interactive-common/interactiveBase.ts	`5.64% <25.00%> (+0.12%)`	⬆️
src/client/datascience/jupyter/jupyterExecution.ts	`51.68% <70.00%> (+2.31%)`	⬆️
...rc/client/datascience/errorHandler/errorHandler.ts	`90.62% <92.30%> (+0.62%)`	⬆️
src/client/common/utils/localize.ts	`95.19% <100.00%> (+0.05%)`	⬆️
src/client/datascience/commands/serverSelector.ts	`85.71% <100.00%> (+1.09%)`	⬆️
src/client/datascience/constants.ts	`99.71% <100.00%> (+<0.01%)`	⬆️
...science/jupyter/jupyterZMQBinariesNotFoundError.ts	`100.00% <100.00%> (ø)`
src/client/datascience/jupyter/serverSelector.ts	`91.17% <100.00%> (+0.40%)`	⬆️
src/datascience-ui/react-common/arePathsSame.ts	`75.00% <0.00%> (-12.50%)`	⬇️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ca776ab...1acba4f. Read the comment docs.

IanMatthewHuff · 2020-03-13T15:29:14Z

src/client/datascience/datascience.ts

+                traceInfo(`ZMQ connection to port ${port} verified.`);
+            } catch (e) {
+                traceError(`Exception while attempting zmq :`, e);
+                // If this happens, disable datascience altogether. We can't talk to


Last time we discussed this I thought the plan was that in this situation they would be allowed to remote connect using the old path of specifying the URI

Hmm, yes that's true. Didn't even think of making just remote connections work. Not even sure how we'd do that. That would likely be rather complex (for what might be less then 1% of users). I think we could do this for now and think about how to do just remote in a future update.

I think what I could do is put this same check in a different spot (where we start a server). Then instead of disabling, I'd put up an error dialog that says the same thing, and a button that lets you pick the remote URI

I like that idea as long as it's not too crazy much work. Just feels safer.

sonarqubecloud · 2020-03-13T18:32:08Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities (and 0 Security Hotspots to review)
0 Code Smells

No Coverage information
0.0% Duplication

IanMatthewHuff · 2020-03-13T18:45:42Z

src/client/common/utils/localize.ts

+
+    export const nativeDependencyFail = localize(
+        'DataScience.nativeDependencyFail',
+        '{0}. We cannot launch a jupyter server for you because your OS is not supported. Select an already running server if you wish to continue.'


At first the wording here looked odd. Since we are not launching a jupyter server. But I think from the user's perspective this wording makes sense, given that we are launching essentially a "fake" jupyter server on their behalf.

DonJayamanne · 2020-03-13T20:01:02Z

src/client/datascience/jupyter/jupyterExecution.ts

+        } catch (e) {
+            traceError(`Exception while attempting zmq :`, e);
+            sendTelemetryEvent(Telemetry.ZMQNotSupported);
+            this.zmqError = new JupyterZMQBinariesNotFoundError(e.toString());


Why not pass in e.message, if we're going to display this, then i'd prefer we passed a meaningfule message.
e.tostring() will contain unstructured data.

this.zmqError = new JupyterZMQBinariesNotFoundError(e); class JupyterZMQBinariesNotFoundError extends Error { constructor(originalException: Error) { super(originalException.message); this.stack = originalException.stack; }

The message is already meaningful. It says the platform binaries are missing.

DonJayamanne · 2020-03-13T20:03:53Z

src/client/datascience/jupyter/jupyterExecution.ts

    private async startOrConnect(
        options?: INotebookServerOptions,
        cancelToken?: CancellationToken
    ): Promise<IConnection> {
        // If our uri is undefined or if it's set to local launch we need to launch a server locally
        if (!options || !options.uri) {
+            // First verify we have ZMQ installed correctly (this might change when we don't 'launch' servers anymore)
+            await this.verifyZMQ();


Shouldn't this be done only if the user belongs to the ZMQ experiment?
Also, I would have expected this to go into a separate PR (hooking up experiment... ) and separate class (depending on which path to go down..).

Sorry already merged.

The work for the ZMQ experiment can be picked up when we add the new experiment.

* master: Move redux logging to a different file (#10605) Ensure local host only if connection not available (#10600) Fix a bunch of debugger issues (#10572) Bump acorn from 6.0.0 to 6.4.1 (#10570) Resize plot to fit within pdf page (#10547) Update dev build instructions to mention "python.insidersChannel" Install the ZMQ prebuilt binaries and verify ZMQ works on startup (#10551)

rchiodo added 2 commits March 12, 2020 13:42

Npm package

e6e2ffe

Add ZMQ to the extension

35cbd42

rchiodo requested review from IanMatthewHuff, DonJayamanne and DavidKutu March 13, 2020 00:39

rchiodo self-assigned this Mar 13, 2020

rchiodo commented Mar 13, 2020

View reviewed changes

news/3 Code Health/10483.md

@@ -0,0 +1 @@

Add ZMQ library to extension

Copy link

Author

rchiodo Mar 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested on windows and linux (ubuntu 18)

DonJayamanne requested changes Mar 13, 2020

View reviewed changes

IanMatthewHuff reviewed Mar 13, 2020

View reviewed changes

rchiodo added 2 commits March 13, 2020 08:59

Merge remote-tracking branch 'origin/master' into rchiodo/zmq_install

872642e

Fix functional failure

a4f64cb

DonJayamanne approved these changes Mar 13, 2020

View reviewed changes

rchiodo added 3 commits March 13, 2020 10:17

Make verification happen at server launch

67e4672

Fix native dependencies. Make sure error sticks around

fa28e16

Fix functional tests

773f500

rchiodo requested review from IanMatthewHuff and DonJayamanne March 13, 2020 18:16

Fix linter

1acba4f

IanMatthewHuff reviewed Mar 13, 2020

View reviewed changes

IanMatthewHuff approved these changes Mar 13, 2020

View reviewed changes

DonJayamanne requested changes Mar 13, 2020

View reviewed changes

rchiodo merged commit dbf4d70 into master Mar 13, 2020

rchiodo deleted the rchiodo/zmq_install branch March 13, 2020 20:12

lock bot locked as resolved and limited conversation to collaborators Mar 21, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Install the ZMQ prebuilt binaries and verify ZMQ works on startup #10551

Install the ZMQ prebuilt binaries and verify ZMQ works on startup #10551

rchiodo commented Mar 13, 2020

rchiodo Mar 13, 2020

DonJayamanne Mar 13, 2020 •

edited

Loading

rchiodo Mar 13, 2020

rchiodo Mar 13, 2020

rchiodo Mar 13, 2020

codecov-io commented Mar 13, 2020 •

edited

Loading

IanMatthewHuff Mar 13, 2020

rchiodo Mar 13, 2020

rchiodo Mar 13, 2020

IanMatthewHuff Mar 13, 2020

sonarqubecloud bot commented Mar 13, 2020

IanMatthewHuff Mar 13, 2020

DonJayamanne Mar 13, 2020

rchiodo Mar 13, 2020

DonJayamanne Mar 13, 2020

rchiodo Mar 13, 2020

rchiodo Mar 13, 2020

Install the ZMQ prebuilt binaries and verify ZMQ works on startup #10551

Install the ZMQ prebuilt binaries and verify ZMQ works on startup #10551

Conversation

rchiodo commented Mar 13, 2020

Choose a reason for hiding this comment

DonJayamanne Mar 13, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Mar 13, 2020 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Mar 13, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DonJayamanne Mar 13, 2020 •

edited

Loading

codecov-io commented Mar 13, 2020 •

edited

Loading