cdsl-research · tomoyk · May 31, 2022 · May 24, 2022 · May 24, 2022 · May 24, 2022
diff --git a/.github/workflows/image-build.yml b/.github/workflows/image-build.yml
@@ -0,0 +1,46 @@
+on:
+  push:
+    branches:
+      - master
+
+jobs:
+  build-and-push:
+    runs-on: ubuntu-20.04
+    permissions:
+      contents: read
+      packages: write
+    strategy:
+      matrix:
+        target:
+          [
+            "crawler",
+            "dashboard",
+            "executor",
+          ]
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v2
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v1
+
+      - name: Log in to the Container registry
+        uses: docker/login-action@v1
+        with:
+          registry: ghcr.io
+          username: ${{ github.repository_owner }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Set image tag for master
+        run: |
+          COMMITID=$(git rev-parse --short ${{ github.sha }} | cut -c1-7)
+          CUR_BRANCH=$(git symbolic-ref --short HEAD)
+          echo "DOCKER_TAG=${CUR_BRANCH}-${COMMITID}" >> $GITHUB_ENV
+
+      - name: Build and push Docker image
+        uses: docker/build-push-action@v2
+        with:
+          context: .
+          file: ${{ matrix.target }}/Dockerfile
+          push: true
+          tags: ghcr.io/cdsl-research/ecoman/${{ matrix.target }}:${{ env.DOCKER_TAG }}
diff --git a/.gitignore b/.gitignore
@@ -1,3 +1,133 @@
-hosts.yaml
-env/**
-__pycache__/**
+hosts.yml
+id_rsa*
+
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
diff --git a/Dockerfile b/Dockerfile
diff --git a/README.md b/README.md
@@ -1,75 +1,67 @@
-# ECoMan - Easy Computer Manager
+# ECoMan
 
-## How to run
+ECoMan is a tool for supporting virtual machine management on VMware ESXi.
+It provides administrators to view virtual machine lists with web user interface.
 
-Setup venv
+<img src="assets/demo.gif">
 
-```
-python3 -m venv env
-. env/bin/activate
-pip installl -r requirements.txt
-```
+## Feature
 
-Configure esxi-node credentials on hosts.yaml
+- View virtual machine lists over multiple ESXi Nodes
+- Create a virtual machine via Web UI
+- Set power status via Web UI
+- Emphasize several machines which have large hardware resources in Web UI
 
-```
-### Example
-mint:
-  addr: 'mint.a910.tak-cslab.org'
-  username: 'root'
-  password: 'xxxxxxxxxx'
-```
+## Requirements
 
-Try to connect ssh on terminal
+- Python 3.9 or later
+    - Paramiko
+    - FastAPI
+    - PyMongo
+- MongoDB 5.0 or later
 
-```
-ssh [email protected]
-```
+## Installation
 
-Starting app
+ECoMan has several ways for running as below.
 
-```
-python app.py
+- [Linux Machine (General)](docs/install_linux_machine.md)
+- [Kubernetes](docs/install_kubernetes.md)
 
-# uwsgi
-pip install uwsgi
-uwsgi --http :3301 --wsgi-file app.py --callable app
-```
+## Optional Parameters
 
-Acceess Web UI
+Hosts config file `hosts.yml` for ESXi Nodes:
 
 ```
-http://<your-hostname>:3301/
+hoge:  # ESXi Node Name
+  addr: 'hoge.a910.tak-cslab.org'  # ESXi Node Address
+  username: 'root'  # ESXi Username
+  identity_file_path: "ssh/id_rsa"  # SSH Private Key File
+  datastore_path: '/vmfs/volumes/datastore1/'  # VM stored path on ESXi
+  installer_iso_path: "/vmfs/volumes/datastore1/os-images/ubuntu2004.iso"  # Installer ISO stored path
+  network_port_group: "Management Network"  # VM network port group
 ```
 
-## Docker
-
-```
-docker run -it --rm -v $PWD:/config -e HOSTS_PATH=/config/hosts.yaml -p 2200:3301 tomoyk/ecoman
-```
-
-## Option: OAuth2-Proxy support
-
-This app support enable user to SSO(Single Sing-On) with OAuth2-Proxy. 
-
-Example config `oauth2-proxy.cfg` is put that [here](oauth2-proxy.cfg.example).
-
-New endpoint is `http://<your-hostname>:4180/`.
-
-## Screenshot
-
-VM List
-
-<img src="https://raw.githubusercontent.com/cdsl-research/ecoman/master/images/ecoman1.jpg" width="500">
-
-VM Detail
+Environment Variables:
 
-<img src="https://raw.githubusercontent.com/cdsl-research/ecoman/master/images/ecoman2.jpg" width="500">
+| Name                           | Default Value | Type    | Description | Component |
+| ---                            | ---           | ---     | ---         | ---       |
+| MONGO_USERNAME                 |               | String  | If this value is empty, ECoMan connects to MongoDB without authentication. | crawler, dashboard |
+| MONGO_PASSWORD                 |               | String  |             | crawler, dashboard |
+| MONGO_DBNAME                   | ecoman        | String  |             | crawler, dashboard |
+| MONGO_HOST                     | 127.0.0.1     | String  |             | crawler, dashboard |
+| CRAWLER_INTERVAL               | 60            | Integer |             | crawler   |
+| DASHBOARD_THRESHOLD_CPU        | 5             | Integer | CPU usage on VM lists is emphasized by this value. | dashboard |
+| DASHBOARD_THRESHOLD_RAM_MB     | 8192          | Integer | RAM usage on VM lists is emphasized by this value. | dashboard |
+| DASHBOARD_THRESHOLD_STORAGE_GB | 50            | Integer | Storage usage on VM lists is emphasized by this value. | dashboard |
+| EXECUTOR_ADDRESS               | 127.0.0.1     | String  |             | dashboard |
+| EXECUTOR_PORT                  | 8600          | Integer |             | dashboard, executor |
 
-VM Creation
+## Architecture
 
-<img src="https://raw.githubusercontent.com/cdsl-research/ecoman/master/images/ecoman3.jpg" width="500">
+ECoMan has following system archtiecture.
 
-## Demo Movie
+- `crawler` fetches VMs info via SSH and regist VMs info to MongoDB. This component works by polling.
+- `dashboard` provides Web UI to Administrators. When Administrator accesses `dashboard`, it fetches VMs info from MongoDB and returns to Administrator.
+- `executor` receives actions, "Create VM" or "Update VM's power", from `dashboard` and does the actions via SSH.
 
-[Demo Link](https://twitter.com/i/status/1277117890764828673)
+<img src="assets/architecture.png">