...
There is no additional 3rd party dependency required.
Performance Benchmark
TCP/IP Network Benchmark
To demo MXNet+BytePS performance, we test two models: VGG16 (communication-intensive) and Resnet50 (computation-intensive). Both models are trained using fp32.
...
BytePS outperforms Horovod (NCCL) by 44% for Resnet50, and 100% for VGG16.
RDMA Network Benchmark
We also test the BERT-large model using fp16 on with RDMA network. The model is implemented using the gluon-nlp toolkit.
We use Tesla V100 32GB GPUs and set batch size equal to 64 per GPU. Each machine has 8 V100 GPUs with NVLink-enabled. Machines are inter-connected with 100 Gbps Infiniband network.
BytePS outperforms Horovod (carefully tuned) by 16% in this case, both with RDMA enabled.
Limitation
BytePS currently has the following limitations:
...