Thanks to visit codestin.com
Credit goes to github.com

Skip to content
This repository was archived by the owner on Sep 3, 2022. It is now read-only.

Commit f2b3242

Browse files
ojarjurojarjur
authored
Add an experimental tools directory, including a PoC Datalab manager. (#2081)
* Add an experimental tools directory, including a PoC Datalab manager. * Improve instructions for the MY_DOMAIN environment variable * Fix typos in the datalab-hosted-service README * Pin the version of typescript used when building the base image
1 parent 16f7c3c commit f2b3242

File tree

16 files changed

+1668
-2
lines changed

16 files changed

+1668
-2
lines changed

‎.gitignore‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ containers/base/pydatalab
2222
node_modules/
2323
bower_components/
2424
test/ui/broken
25-
experimental
25+
sources/web/datalab/static/experimental
2626
polymer_bundled
2727
polymer_unbundled
2828
sources/web/datalab/polymer/**/*.js

‎containers/base/Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -246,7 +246,7 @@ RUN if [ -d /datalab/lib/pydatalab/.git ]; then \
246246
git clone https://github.com/googledatalab/pydatalab.git /datalab/lib/pydatalab; \
247247
fi && \
248248
cd /datalab/lib/pydatalab && \
249-
/tools/node/bin/npm install -g typescript && \
249+
/tools/node/bin/npm install -g typescript@3.0.3 && \
250250
tsc --module amd --noImplicitAny --outdir datalab/notebook/static datalab/notebook/static/*.ts && \
251251
tsc --module amd --noImplicitAny --outdir google/datalab/notebook/static google/datalab/notebook/static/*.ts && \
252252
/tools/node/bin/npm uninstall -g typescript && \

‎tools/experimental/README.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Experimental Datalab Tools
2+
3+
This directory contains tools that explore ideas for improving Datalab
4+
but which are *not* yet recommended for end users.
5+
6+
In particular, none of the tools under this directory are supported
7+
by either the Datalab team or by the Google Cloud Platform Support
8+
channels.

‎tools/experimental/hosted-service/README.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
# Managed Datalab Service Example
2+
3+
This directory shows how you can run your own service for automatically
4+
allocating and managing Datalab instances.
5+
6+
This is done by defining a VM manager that creates and deletes the instances,
7+
and pairing that with an [inverting proxy](https://github.com/google/inverting-proxy)
8+
that is used to connect to those instances.
9+
10+
The resulting service allows users under a particular domain (e.g.
11+
`[email protected]`) to automatically get a Datalab instance that
12+
is assigned to them and runs in your GCP project.
13+
14+
Each Datalab instance runs with its own service account which can be found
15+
by clicking on the user icon in the Datalab UI.
16+
17+
## DISCLAIMER
18+
19+
This is experimental and unsupported.
20+
21+
Note, also, that this is not cheap. The pool of VMs, the VM manager, and
22+
the static file server are all running continuously. If you run this with
23+
the default settings then you should expect it to cost you at least several
24+
hundred dollars per month. If you increase the machine type then it will
25+
cost even more.
26+
27+
## Setup
28+
29+
To deploy this, first create a GCP project to run the service and enable
30+
the Compute Engine API in that project.
31+
32+
Then, set the `PROJECT_ID` environment variable to the ID of your project,
33+
and run the following commands to deploy the inverting proxy to your project:
34+
35+
```sh
36+
pushd ./
37+
cd /tmp
38+
git clone https://github.com/google/inverting-proxy
39+
cd inverting-proxy
40+
make deploy
41+
popd
42+
```
43+
44+
Next, deploy the `datalabstatic` App Engine service to your project.
45+
46+
This serves static content for Datalab instances directly from a single,
47+
read-only instance of Datalab, and makes the UI much more responsive by
48+
avoiding the need to proxy static resources:
49+
50+
Run the following commands to deploy it. The `app deploy` step can sometimes
51+
fail saying that the backend did not report as healthy in time. If that happens,
52+
just retry it:
53+
54+
```sh
55+
gcloud --project ${PROJECT_ID} app deploy staticfiles/*.yaml
56+
```
57+
58+
Now, you can deploy the VM manager, which also runs in an App Engine Flex app.
59+
60+
Before you do, set the value of the `MY_DOMAIN` environment variable to the
61+
domain portion of your user's email addresses. For example, if you are allowing
62+
users with `@example.com` email addresses, then you would run
63+
`export MY_DOMAIN=example.com`:
64+
65+
```sh
66+
sed -i -e "s/MY_DOMAIN/${MY_DOMAIN}/" vmmanager/vmmanager.yaml
67+
gcloud --project ${PROJECT_ID} app deploy vmmanager/vmmanager.yaml
68+
git checkout -- vmmanager/vmmanager.yaml
69+
```
70+
71+
Finally, configure the VM manager as the default backend for your proxy:
72+
73+
```sh
74+
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
75+
-d "{\"id\": \"vm-manager\", \"endUser\": \"allUsers\", \"pathPrefixes\": [\"\"], \"backendUser\": \"${PROJECT_ID}@appspot.gserviceaccount.com\"}" \
76+
https://api-dot-${PROJECT_ID}.appspot.com/api/backends
77+
```

‎tools/experimental/hosted-service/staticfiles/Dockerfile‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Copyright 2018 Google Inc. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# This relies on the new feature of running a Datalab instance in read-only
16+
# mode, which was submitted in #2075 but has not yet been released.
17+
#
18+
# Once it gets released, this image tag should be changed to be the image
19+
# for the latest release rather than being pinned to this specific commit.
20+
FROM gcr.io/cloud-datalab/datalab:commit-9dd43fe68e6cc2975ca0383caeb04caa4057ab91
21+
22+
ENV DATALAB_SETTINGS_OVERRIDES="{\"proxyWebSockets\": \"true\", \"readOnly\": \"true\"}"

‎tools/experimental/hosted-service/staticfiles/README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# Read-Only Datalab Static Files Server
2+
3+
This directory contains the definition of an App Engine Flex app that can
4+
be used to serve the static files for a managed Datalab service.
5+
6+
This app allows requests for static files to be served without going through
7+
the inverting proxy, which makes the Datalab UI much more responsive.

‎tools/experimental/hosted-service/staticfiles/datalabstatic.yaml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
runtime: custom
2+
env: flex
3+
service: datalabstatic
4+
5+
liveness_check:
6+
path: "/static/logo.png"
7+
check_interval_sec: 30
8+
timeout_sec: 4
9+
failure_threshold: 10
10+
success_threshold: 2
11+
12+
readiness_check:
13+
path: "/static/logo.png"
14+
check_interval_sec: 30
15+
timeout_sec: 4
16+
failure_threshold: 10
17+
success_threshold: 2
18+
app_start_timeout_sec: 60
19+
20+
automatic_scaling:
21+
min_num_instances: 1
22+
23+
resources:
24+
cpu: 4
25+
memory_gb: 16
26+
disk_size_gb: 100
27+
28+
handlers:
29+
- url: /static/.*
30+
script: this_is_required_but_ignored

‎tools/experimental/hosted-service/staticfiles/dispatch.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
dispatch:
2+
# Serve the favicon from the static server
3+
- url: "*/favicon.ico"
4+
service: datalabstatic
5+
6+
# We still want the user's chosen theme to be served by their backend
7+
- url: "*/static/style*"
8+
service: default
9+
10+
# Serve the remaining contents of the "/static/" path from the static server
11+
- url: "*/static/*"
12+
service: datalabstatic

‎tools/experimental/hosted-service/vmmanager/Dockerfile‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# Copyright 2018 Google Inc. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
FROM debian
15+
16+
ADD supervisord.conf /etc/supervisor/conf.d/supervisord.conf
17+
ADD ./ /opt/src/vm-manager
18+
19+
RUN apt-get update && apt-get upgrade -y && \
20+
apt-get install -y -qq --no-install-recommends \
21+
ca-certificates \
22+
curl \
23+
git \
24+
supervisor \
25+
unzip \
26+
wget && \
27+
mkdir -p /opt/bin && \
28+
mkdir -p /var/log/supervisor && \
29+
mkdir -p /var/log/app_engine/custom_logs && \
30+
wget -O /opt/go1.10.3.linux-amd64.tar.gz \
31+
https://storage.googleapis.com/golang/go1.10.3.linux-amd64.tar.gz && \
32+
tar -C /usr/local -xzf /opt/go1.10.3.linux-amd64.tar.gz && \
33+
export PATH=${PATH}:/usr/local/go/bin/:/opt/bin/ && \
34+
export GOPATH=/opt/ && \
35+
cd /opt/src/vm-manager && \
36+
go get github.com/golang/groupcache/lru && \
37+
go get golang.org/x/net/context && \
38+
go get golang.org/x/oauth2/google && \
39+
go get google.golang.org/api/compute/v1 && \
40+
go get google.golang.org/api/iam/v1 && \
41+
go build -o /opt/bin/vm-manager manager.go && \
42+
go get github.com/google/inverting-proxy/agent && \
43+
rm -rf /opt/go1.4.2.linux-amd64.tar.gz && \
44+
rm -rf /usr/local && \
45+
wget -nv https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.zip && \
46+
unzip -qq google-cloud-sdk.zip -d /opt/tools && \
47+
rm google-cloud-sdk.zip && \
48+
/opt/tools/google-cloud-sdk/install.sh --usage-reporting=false \
49+
--path-update=false --bash-completion=false \
50+
--disable-installation-options
51+
52+
ENV ZONE us-west1-a
53+
ENV MACHINE_TYPE n1-standard-1
54+
ENV NETWORK default
55+
ENV ALLOWED_DOMAIN gmail.com
56+
57+
EXPOSE 8080
58+
59+
CMD /usr/bin/supervisord

0 commit comments

Comments
 (0)