...
View file | ||||
---|---|---|---|---|
|
View file | ||||
---|---|---|---|---|
|
(a) Ring algorithm used by NCCL (b) Parameter server algorithm
...
Figure 2 explains the end-to-end performance results (see Figure x8) that show Parameter server is faster for networks that require many Push-Pulls over relatively small keys (e.g. ResNet-50, Inception-48 need over 157 Push-Pulls on keys not exceeding 2M floats in size), but NCCL ring Reduce is faster for networks that only need Push-Pull over few keys (e.g. VGG-16 and AlexNet only need fewer than 32 Push-Pulls on keys that exceed 10M in size).
...
When to switch between Single and Multiple tree
Figure 67. VGG-16 performance as function of MXNET_KVSTORE_BIGARRAY_BOUND using batch size 4 per GPU.
...
Vs. Parameter Server (in comm.h) | Vs. NCCL (in kvstore_nccl.h) | |
---|---|---|
Resnet-50 | 1.19 | 1.33 |
VGG-16 | 5.89 | 1.06 |
Inception-v3 | 1.15 | 1.34 |
AlexNet | 6.60 | 1.42 |
Figure 78. End-to-end training results on synthetic data showing speed-up vs. NCCL on fp32 and fp16.
...