THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Google doc as below and welcome to comment and modify:
...
Machine: SKX6148, Network: 10GbE, Topology: VGG16, Local Batch Size: 64, KVStore Type: dist_sync. Parameter Server
| work num | server num | Per Node FPS(pic/s) | Scaling Efficiency |
|---|---|---|---|
| 8 | 8(worker and server share node) | 19.87 | 67.81% |
| 8 | 8 | 27.3 | 93.17% |
| 8 | 4 | 22.7 | 77.47% |
| 8 | 2 | 11.11 | 37.90% |
Command line: python tools/launch.py -n 8 -s <server_num> --launcher ssh -H hosts python example/image-classification/train_vgg16.py --kv-store dist_sync
Following is the result of MXNet multinode with mpi allreduce supported from our proof of concept (ready):
| Node Num | Per Node FPS(pic/s) | Scaling Efficiency |
|---|---|---|
| 8 | 27.76 | 94.74% |
Command line: mpirun -n 8 -ppn 1 -machinefile hosts python example/image-classification/train_vgg16.py --kv-store dist_sync_mpi
...
Following is allreduce benchmark (400M payload 8 worker num) for reference:
| Method | server num | worker num | Time(s) |
|---|---|---|---|
| ParameterServer (push+pull) | 1 | 8 | 6.5 |
| ParameterServer (push+pull) | 2 | 8 | 3.4 |
| ParameterServer (push+pull) | 4 | 8 | 2.0 |
| ParameterServer (push+pull) | 8 | 8 | 1.2 |
| MPI.Allreduce | N/A | 8 | 1.0 |
From the performance data, we can draw the following conclusions:
...
Moreover, we noticed that most of mainstream Deep Learning frameworks have mpi-allreduce-based distributed training mechnism:
| Framework | Distributed Communication Mechanism |
|---|---|
| Tensorflow | PS + mpi-allreduce(baidu allreduce) |
| MXNet | PS |
| Caffe | mpi-allreduce |
| Torch + PyTorch | mpi-allreduce |
| Chainer | mpi-allreduce(mpi4py) |
Goal
Besides existing distributed training mechanism parameter server in MXNet, we suggest to add mpi-allreduce as an alternative distributed training mechanism which can significantly enhance multi-node scaling efficiency for synchronous SGD distributed training with least cost.
...