Minor editorial changes on README.md; changed the default value for ping_timeout to 360.

baochunli · baochunli · commit a96f7cd7e40a · 2021-08-13T13:21:43.000+08:00
diff --git a/README.md b/README.md
@@ -1,7 +1,9 @@
-## Plato: A New Framework for Federated Learning Research
+# Plato: A New Framework for Federated Learning Research
 
 Welcome to *Plato*, a new software framework to facilitate scalable federated learning research.
 
+## Installation
+
 ### Installing Plato with PyTorch
 
 To install *Plato*, first clone this repository to the desired directory.
@@ -73,6 +75,77 @@ It goes without saying that `/absolute/path/to/project/home/directory` should be
 
 **Tip:** When working in Visual Studio Code as the development environment, one of the project developer's colour theme favourites is called `Bluloco`, both of its light and dark variants are excellent and very thoughtfully designed. The `Pylance` extension is also strongly recommended, which represents Microsoft's modern language server for Python.
 
+### Installing YOLOv5 as a Python package
+
+If object detection using the YOLOv5 model and any of the COCO datasets is needed, it is necessary to install YOLOv5 as a Python package first:
+
+```shell
+cd packages/yolov5
+pip install .
+```
+
+### Installing Plato with MindSpore
+
+Plato is designed to support multiple deep learning frameworks, including PyTorch, TensorFlow, and MindSpore. For MindSpore support, Plato currently supports MindSpore 1.1.1 (1.2.1 and 1.3.0 are not supported, as [they do not support `Tensor` objects to be pickled](https://gitee.com/mindspore/mindspore/issues/I43RPP?from=project-issue) and sent over a network). Though we provided a `Dockerfile` for building a Docker container that supports MindSpore 1.1.1, in rare cases it may still be necessary to install Plato with MindSpore in a GPU server running Ubuntu Linux 18.04 (which MindSpore requires). Similar to a PyTorch installation, we need to first create a new environment with Python 3.7.5 (which MindSpore 1.1.1 requires), and then install the required packages:
+
+```shell
+conda create -n mindspore python=3.7.5
+pip install -r requirements.txt
+```
+
+We should now install MindSpore 1.1.1 with the command provided by the [official MindSpore website](https://mindspore.cn/install).
+
+MindSpore 1.1.1 may also need additional packages, which should installed if they do not exist:
+
+```shell
+sudo apt-get install libssl-dev
+sudo apt-get install build-essential
+```
+
+If CuDNN has not yet been installed, it needs to be installed with the following commands:
+
+```shell
+wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
+sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
+sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
+sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /"
+sudo apt-get update
+sudo apt-get install libcudnn8=8.0.5.39-1+cuda10.1
+```
+
+To check the current CuDNN version, the following commands are helpful:
+
+```shell
+function lib_installed() { /sbin/ldconfig -N -v $(sed 's/:/ /' <<< $LD_LIBRARY_PATH) 2>/dev/null | grep $1; }
+function check() { lib_installed $1 && echo "$1 is installed" || echo "ERROR: $1 is NOT installed"; }
+check libcudnn
+```
+
+To check if MindSpore is correctly installed on the GPU server, try to run the command:
+
+```shell
+python -c "import mindspore"
+```
+
+Finally, to use trainers and servers based on MindSpore, assign `true` to `use_mindspore` in the `trainer` section of the configuration file. If GPU is not available when MindSpore is used, assign `true` to `cpuonly` in the `trainer` section as well. These variables are unassigned by default, and *Plato* would use PyTorch as its default framework.
+
+## Running Plato
+
+### Running Plato using a configuration file
+
+To start a federated learning training workload, run [`run`](run) from the repository's root directory. For example:
+
+```shell
+./run --config=configs/MNIST/fedavg_lenet5.yml
+```
+
+* `--config` (`-c`): the path to the configuration file to be used. The default is `config.yml` in the project's home directory.
+* `--log` (`-l`): the level of logging information to be written to the console. Possible values are `critical`, `error`, `warn`, `info`, and `debug`, and the default is `info`.
+
+*Plato* uses the YAML format for its configuration files to manage the runtime configuration parameters. Example configuration files have been provided in the `configs` directory.
+
+*Plato* can opt to use `wandb` to produce and collect logs in the cloud. If this is needed, add `use_wandb: true` to the `trainer` section in your configuration file.
+
 ### Running Plato in a Docker container
 
 Most of the codebase in *Plato* is designed to be framework-agnostic, so that it is relatively straightfoward to use *Plato* with a variety of deep learning frameworks beyond PyTorch, which is the default framwork it is using. One example of such deep learning frameworks that *Plato* currently supports is [MindSpore 1.1.1](https://www.mindspore.cn). Due to the wide variety of tricks that need to be followed correctly for running *Plato* without Docker, it is strongly recommended to run Plato in a Docker container, on either a CPU-only or a GPU-enabled server.
@@ -119,30 +192,7 @@ The provided `Dockerfile` helps to build a Docker image running Ubuntu 20.04, wi
 
 If MindSpore support is needed, the provided `Dockerfile_MindSpore` contains two pre-configured environments for CPU and GPU environments, respectively, called `plato_cpu` or `plato_gpu`. They support [MindSpore 1.1.1](https://github.com/mindspore-ai/mindspore) and Python 3.7.5 (which is the Python version that MindSpore requires). Both Dockerfiles have GPU support enabled. Once an image is built and a Docker container is running, one can use Visual Studio Code to connect to it and start development within the container.
 
-### Installing YOLOv5 as a Python package
-
-If object detection using the YOLOv5 model and any of the COCO datasets is needed, it is necessary to install YOLOv5 as a Python package first:
-
-```shell
-cd packages/yolov5
-pip install .
-```
-### Running Plato
-
-To start a federated learning training workload, run [`run`](run) from the repository's root directory. For example:
-
-```shell
-./run --config=configs/MNIST/fedavg_lenet5.yml
-```
-
-* `--config` (`-c`): the path to the configuration file to be used. The default is `config.yml` in the project's home directory.
-* `--log` (`-l`): the level of logging information to be written to the console. Possible values are `critical`, `error`, `warn`, `info`, and `debug`, and the default is `info`.
-
-*Plato* uses the YAML format for its configuration files to manage the runtime configuration parameters. Example configuration files have been provided in the `configs` directory.
-
-*Plato* can opt to use `wandb` to produce and collect logs in the cloud. If this is needed, add `use_wandb: true` to the `trainer` section in your configuration file.
-
-### Potential Runtime Errors
+### Potential runtime errors
 
 If runtime exceptions occur that prevent a federated learning session from running to completion, the potential issues could be:
 
@@ -152,21 +202,21 @@ If runtime exceptions occur that prevent a federated learning session from runni
  
 * The time that a client waits for the server to respond before disconnecting is too short. This could happen when training with large neural network models. If you get an `AssertionError` saying that there are not enough launched clients for the server to select, this could be the reason. But make sure you first check if it is due to the *out of CUDA memory* error.
 
-  *Potential solutions:* Add `ping_timeout` in the `server` section in your configuration file. The default value for `ping_timeout` is 20 (seconds). You could specify a larger timeout value, such as 120.
+  *Potential solutions:* Add `ping_timeout` in the `server` section in your configuration file. The default value for `ping_timeout` is 360 (seconds). 
 
-  For example, to run a training session with the CIFAR-10 dataset and the ResNet-18 model, and if 10 clients are selected per round, `ping_timeout` needs to be 120. Consider an even larger number if you run with larger models and more clients.
+  For example, to run a training session on [Google Colaboratory or Compute Canada](https://github.com/TL-System/plato/blob/main/docs/Running.md) with the CIFAR-10 dataset and the ResNet-18 model, and if 10 clients are selected per round, `ping_timeout` needs to be 360 when clients' local datasets are non-iid by symmetric Dirichlet distribution with the concentration of 0.01. Consider an even larger number if you run with larger models and more clients.
 
 * Running processes have not been terminated from previous runs. 
 
   *Potential solutions:* Use the command `pkill python` to terminate them so that there will not be CUDA errors in the upcoming run.
 
-### Client Simulation Mode
+### Client simulation mode
 
 Plato supports a *client simulation mode*, in which the actual number of client processes launched equals the number of clients to be selected by the server per round, rather than the total number of clients. This supports a simulated federated learning environment, where the set of selected clients by the server will be simulated by the set of client processes actually running. For example, with a total of 10000 clients, if the server only needs to select 100 of them to train their models in each round, only 100 client processes will be launched in client simulation mode, and a client process may assume a different client ID in each round.
 
 To turn on the client simulation mode, add `simulation: true` to the `clients` section in the configuration file.
 
-### Plotting Runtime Results
+### Plotting runtime results
 
 If the configuration file contains a `results` section, the selected performance metrics, such as accuracy, will be saved in a `.csv` file in the `results/` directory. By default, the `results/` directory is under the path to the used configuration file, but it can be easily changed by modifying `Config.result_dir` in [`config.py`](config.py).
 
@@ -178,60 +228,17 @@ python plot.py --config=config.yml
 
 * `--config` (`-c`): the path to the configuration file to be used. The default is `config.yml` in the project's home directory.
 
-### Running Unit Tests
+### Running unit tests
 
 All unit tests are in the `tests/` directory. These tests are designed to be standalone and executed separately. For example, the command `python lr_schedule_tests.py` runs the unit tests for learning rate schedules.
 
-### Installing Plato with MindSpore
-
-Plato is designed to support multiple deep learning frameworks, including PyTorch, TensorFlow, and MindSpore. For MindSpore support, Plato currently supports MindSpore 1.1.1 (1.2.1 and 1.3.0 are not supported, as [they do not support `Tensor` objects to be pickled](https://gitee.com/mindspore/mindspore/issues/I43RPP?from=project-issue) and sent over a network). Though we provided a `Dockerfile` for building a Docker container that supports MindSpore 1.1.1, in rare cases it may still be necessary to install Plato with MindSpore in a GPU server running Ubuntu Linux 18.04 (which MindSpore requires). Similar to a PyTorch installation, we need to first create a new environment with Python 3.7.5 (which MindSpore 1.1.1 requires), and then install the required packages:
-
-```shell
-conda create -n mindspore python=3.7.5
-pip install -r requirements.txt
-```
-
-We should now install MindSpore 1.1.1 with the command provided by the [official MindSpore website](https://mindspore.cn/install).
-
-MindSpore 1.1.1 may also need additional packages, which should installed if they do not exist:
-
-```shell
-sudo apt-get install libssl-dev
-sudo apt-get install build-essential
-```
-
-If CuDNN has not yet been installed, it needs to be installed with the following commands:
-
-```shell
-wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
-sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
-sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
-sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /"
-sudo apt-get update
-sudo apt-get install libcudnn8=8.0.5.39-1+cuda10.1
-```
-
-To check the current CuDNN version, the following commands are helpful:
-
-```shell
-function lib_installed() { /sbin/ldconfig -N -v $(sed 's/:/ /' <<< $LD_LIBRARY_PATH) 2>/dev/null | grep $1; }
-function check() { lib_installed $1 && echo "$1 is installed" || echo "ERROR: $1 is NOT installed"; }
-check libcudnn
-```
-
-To check if MindSpore is correctly installed on the GPU server, try to run the command:
-
-```shell
-python -c "import mindspore"
-```
-
-Finally, to use trainers and servers based on MindSpore, assign `true` to `use_mindspore` in the `trainer` section of the configuration file. If GPU is not available when MindSpore is used, assign `true` to `cpuonly` in the `trainer` section as well. These variables are unassigned by default, and *Plato* would use PyTorch as its default framework.
+## Deploying Plato
 
-### Deploying Plato Servers in a Production Environment in the Cloud
+### Deploying Plato servers in a production environment in the cloud
 
 The Plato federated learning server is designed to use Socket.IO over HTTP and HTTPS, and can be easily deployed in a production server environment in the public cloud. See `/docs/Deploy.md` for more details on how the nginx web server can be used as a reverse proxy for such a deployment in production servers.
 
-### Uninstalling Plato
+## Uninstalling Plato
 
 Remove the `conda` environment used to run *Plato* first, and then remove the directory containing *Plato*'s git repository.
 
@@ -244,6 +251,6 @@ where `federated` (or `mindspore`) is the name of the `conda` environment that *
 
 For more specific documentation on how Plato can be run on GPU cluster environments such as Google Colaboratory or Compute Canada, refer to `docs/Running.md`.
 
-### Technical support
+## Technical Support
 
 Technical support questions should be directed to the maintainer of this software framework: Baochun Li (bli@ece.toronto.edu).
diff --git a/docs/Running.md b/docs/Running.md
@@ -228,9 +228,9 @@ If runtime exceptions occur that prevent a federated learning session from runni
  
 * The time that a client waits for the server to respond before disconnecting is too short. This could happen when training with large neural network models. If you get an `AssertionError` saying that there are not enough launched clients for the server to select, this could be the reason. But make sure you first check if it is due to the *out of CUDA memory* error.
 
-  *Potential solutions:* Add `ping_timeout` in the `server` section in your configuration file. The default value for `ping_timeout` is 20 (seconds). You could specify a larger timeout value, such as 120.
-  
-  For example, to run a training session with the CIFAR-10 dataset and the ResNet-18 model, and if 10 clients are selected per round, `ping_timeout` needs to be 120. Consider an even larger number if you run with larger models and more clients.
+  *Potential solutions:* Add `ping_timeout` in the `server` section in your configuration file. The default value for `ping_timeout` is 360 (seconds). 
+
+  For example, to run a training session with the CIFAR-10 dataset and the ResNet-18 model, and if 10 clients are selected per round, `ping_timeout` needs to be 360 when clients' local datasets are non-iid by symmetric Dirichlet distribution with the concentration of 0.01. Consider an even larger number if you run with larger models and more clients.
 
 * Running processes have not been terminated from previous runs. 
 
diff --git a/plato/servers/base.py b/plato/servers/base.py
@@ -108,7 +108,7 @@ def start(self, port=Config().server.port):
         ping_interval = Config().server.ping_interval if hasattr(
             Config().server, 'ping_interval') else 3600
         ping_timeout = Config().server.ping_timeout if hasattr(
-            Config().server, 'ping_timeout') else 20
+            Config().server, 'ping_timeout') else 360
         self.sio = socketio.AsyncServer(ping_interval=ping_interval,
                                         max_http_buffer_size=2**31,
                                         ping_timeout=ping_timeout)
