[WIP] Conda package #108

jonmmease · 2018-08-02T12:50:41Z

Work towards adding a conda build of orca to the CI build.

Discussion: #107
Standalone prototype: https://github.com/jonmmease/orca-feedstock

jonmmease · 2018-08-02T12:55:22Z

@etpinard it doesn't look like the Travis build ran (I only see AppVeyor). Is there anything else I need to do?

etpinard · 2018-08-02T13:39:10Z

it doesn't look like the Travis build ran (I only see AppVeyor)

Ha. It might only run on branches based off the repo origin, and not on branches based off forks.

I should get an invite giving you write rights to this repo. I think the easier way forward would be for you to pull down this repo, cherry-pick your commits unto a new branch and make a new PR. Sorry for the inconvenience.

etpinard · 2018-08-02T13:42:00Z

Actually, pushing another commit to this branch should now do the trick:

jonmmease · 2018-08-02T13:45:55Z

Cool, I'll give that a try first

jonmmease · 2018-08-02T13:46:49Z

Looks like they're all running now. Thanks!

Add Python logic to convert npm version string to setuptools standard version string (1.0.1rc1 in this case)

jonmmease · 2018-08-02T14:53:49Z

@etpinard the conda builds for Circle/Travis/AppVeyor are all working now. Each one successfully performs the conda build and lists the directory where the build artifact is output.

The docker-build-and-push job is failing on "Push Docker image to Quay", which is probably a good thing since I'm not intending up push a new docker image 🙂

All that's left now is to figure out how we want to upload the packages to anaconda cloud. I don't have a good sense of how this part should work. Should the CI jobs do the upload, or should we try to get the built packages out of the CI systems and upload them manually?

Also, someone with some experience with these CI tools should probably take a look at what I did to get things working 🙂

etpinard

the conda builds for Circle/Travis/AppVeyor are all working now.

Fantastic 🎉

The docker-build-and-push job is failing on "Push Docker image to Quay",

That thing should still succeed. We build and push to quay.io a new image for every commit at the moment. Perhaps, it's only intermittent. Perhaps, one needs permissions from quay.io to push from a fork branch?

All that's left now is to figure out how we want to upload the packages to anaconda cloud. I don't have a good sense of how this part should work.

Electron builds are currently uploaded manually to the release page (https://github.com/plotly/orca/releases), so I'd be ok with also manually uploading the conda builds to the conda cloud.

To do so though, we'll need to include the conda builds as CI artifacts so that we can download them:

For Circle: this is done through the store_artifacts field. Moving the conda build to a blank directory and listing it under store_artifacts should do the trick. See how's it's done for the Electron builds

orca/.circleci/config.yml

Lines 154 to 158 in 647239b

    
           - run: mkdir artifacts 
        
           - run: mv release/*.yml artifacts || true 
        
           - run: mv release/orca* artifacts 
        
           - store_artifacts: 
        
               path: artifacts

For Travis: the artifact paths field is here. I'm not sure if it support multiple paths; we might need to mkdir a directory.
For AppVeyor: adding another item under

orca/appveyor.yml

Lines 30 to 32 in 647239b

    
           artifacts: 
        
             - path: release.zip 
        
               name: release

should do the trick.

Moreover, we'll need to document the "push-to-conda-cloud" in our release instructions and mention conda installs in our installation instructions.

Thanks for all your hard work!

etpinard · 2018-08-02T15:05:33Z

.circleci/config.yml

@@ -21,6 +21,7 @@ workflows:
          requires:
            - test-node-v6
            - test-node-v8
+      - conda-build-and-upload


We probably only want to run this if test-node-v6 and test-node-v8 succeed. Adding

requires: - test-node-v6 - test-node-v8

should do the trick

... and we should probably rename this step conda-build as we won't be uploading the builds on CI.

etpinard · 2018-08-02T15:07:45Z

.circleci/config.yml

+          keys:
+            - v3-npm-
+
+      - run: wget http://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh


We could probably cache this, but conda-build-and-upload runs in under 3 minutes, so I don't see the need 👌

etpinard · 2018-08-02T15:11:25Z

package.json

@@ -1,6 +1,6 @@
 {
  "name": "orca",
-  "version": "1.0.0",
+  "version": "1.0.1-rc.1",


Did you base your branch off master or #103 ? If the former, we should stick to 1.0.0 here.

etpinard · 2018-08-02T15:12:35Z

recipe/meta.yaml

@@ -0,0 +1,16 @@
+{% set data = load_setup_py_data() %}


We should add a README.md file in recipe/ briefly explanation the conda-build process.

Environment variables aren't expanded here

e.g. conda-linux-64.zip -> linux-64/*.tar.bz2

jonmmease · 2018-08-02T20:11:59Z

@etpinard
Well that took more tries that I thought it would 🙂 The artifact zip files look good for CircleCI and AppVeyor. I don't have access to the Travis artifacts (at least I don't have an AWS account), could you check that one?

I think I also addressed all of your feedback. Let me know if the Travis artifacts need any work, and if there are any other changes you'd like to see before merging this in.

Thanks!

etpinard · 2018-08-02T21:57:10Z

I don't have access to the Travis artifacts

Actually, looks like the artifacts didn't get pushed to S3 after your commits, perhaps due to another permissions restriction.

Oh well, I fetched your branch and created https://github.com/plotly/orca/compare/conda-build and all CI builds there (including the docker-build-and-push job on Circle) are ✅

Here's the link to Mac conda build:

https://s3.amazonaws.com/image-exporter-travis-artifacts/plotly/orca/433/433.1/conda-osx-64.zip

if you like to try it out.

etpinard · 2018-08-02T21:59:57Z

recipe/README.md

+The resulting package will be named `plotly-orca-*.tar.bz2`, and the build
+command will display the full path of the output location. 
+
+How it works


Wow. I didn't expect this much detail. Thanks for your efforts!

haha, sure. Once I got started a bunch of details came to mind that I thought would be useful background for future contributors.

etpinard · 2018-08-02T22:01:11Z

README.md

@@ -14,6 +14,17 @@ Follow [@plotlygraphs](https://twitter.com/plotlygraphs) on Twitter for Orca ann

 ## Installation

+### Method 1: conda


Beautifully done. Thanks!

etpinard · 2018-08-02T22:01:24Z

CONTRIBUTING.md

@@ -114,6 +114,19 @@ automate this process somehow.
  + Grab the Mac build automatically pushed to [Amazon S3](https://s3.console.aws.amazon.com/s3/buckets/image-exporter-travis-artifacts/plotly/orca/?region=us-east-1&tab=overview) from Travis.
    **N.B.** Select the latest build (largest number *note:* the folders are not necessarily sequential) -> release.zip
 - Run `npm publish`
+- Publish conda packages:


Very clear. 📚

jonmmease · 2018-08-02T22:23:55Z

Great! And the mac artifact looks good (At least it's got the right stuff in it when I unzip if, I haven't done a test install of the new packages yet).

So perhaps all that's left is tracking down the plotly channel credentials? I can try out the installation on Mac, Linux, and Windows as soon as we publish everything to the plotly channel. But we should probably get some other independent users to give it a try as well.

etpinard · 2018-08-03T13:53:44Z

So perhaps all that's left is tracking down the plotly channel credentials?

Yep! Perhaps @chriddyp could help us out for this?

But we should probably get some other independent users to give it a try as well.

Good call 👌

This causes the bash process to become the orca process, which means that signals will be processed by the orca executable directly. See http://veithen.github.io/2014/11/16/sigterm-propagation.html This fixes a problem where it was not possible to kill an orca server from Python if it was launched through the wrapper script using subprocess.Popen

jonmmease · 2018-08-09T14:43:49Z

Replaced by #113

Add conda recipe and initial updates to travis (OS X) build

f450564

Initial AppVeyor build

f0f735e

Initial attempt at CircleCI build

1a0dfcd

jonmmease added 5 commits August 2, 2018 09:54

Fixed OS X miniconda URL for travis

cec5e63

Fixed OS X miniconda URL for circle and added extra logging

27a5a21

Use full path to conda executable on Travis and Circle

24f3a64

Bump version to 1.0.1-alpha.1

d9f13dc

Bump version to 1.0.1-rc.1

f728d18

Add Python logic to convert npm version string to setuptools standard version string (1.0.1rc1 in this case)

etpinard reviewed Aug 2, 2018

View reviewed changes

jonmmease added 14 commits August 2, 2018 14:02

Change cersion back to 1.0.0

5e5da81

Add conda build README

d9513d5

First attempt to add conda packages as build artifacts

a1a7ca3

Take 2 on conda build artifact paths.

a0338b1

Environment variables aren't expanded here

Take 2 on conda build artifact paths.

ab82b98

Environment variables aren't expanded here

Take 4 on conda build artifact paths.

c5c1160

Added conda package publication instructions to CONTRIBUTING.md

b978394

globs not allowed in artifact paths: Try parent directory

4cf83b2

zip win64 directory directly from conda-bld location.

b3f3234

zip win64 directory itself, not contents

5db9330

zip Linux/OS X build directories as artifacts too.

586af58

Guess I need to make the artifacts directory

5cd5be6

Get consistent directory structure in zipped package artifacts

d03da3e

e.g. conda-linux-64.zip -> linux-64/*.tar.bz2

Update CONTRIBUTING conda instruction with consistent zip file structure

474d02f

Added conda installation instructions to README

62f706f

etpinard reviewed Aug 2, 2018

View reviewed changes

jonmmease mentioned this pull request Aug 9, 2018

Conda package #113

Merged

jonmmease closed this Aug 9, 2018

	- run: mkdir artifacts
	- run: mv release/*.yml artifacts \|\| true
	- run: mv release/orca* artifacts
	- store_artifacts:
	path: artifacts

		@@ -14,6 +14,17 @@ Follow [@plotlygraphs](https://twitter.com/plotlygraphs) on Twitter for Orca ann

		## Installation

		### Method 1: conda

Uh oh!

[WIP] Conda package #108

[WIP] Conda package #108

Uh oh!

Conversation

jonmmease commented Aug 2, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonmmease commented Aug 2, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

etpinard commented Aug 2, 2018

Uh oh!

etpinard commented Aug 2, 2018

Uh oh!

jonmmease commented Aug 2, 2018

Uh oh!

jonmmease commented Aug 2, 2018

Uh oh!

jonmmease commented Aug 2, 2018

Uh oh!

etpinard left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonmmease commented Aug 2, 2018

Uh oh!

etpinard commented Aug 2, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonmmease commented Aug 2, 2018

Uh oh!

etpinard commented Aug 3, 2018

Uh oh!

jonmmease commented Aug 9, 2018

Uh oh!

Uh oh!

jonmmease commented Aug 2, 2018 •

edited

Loading

jonmmease commented Aug 2, 2018 •

edited

Loading