...
Both methods are shown in Figure 34.
View file | ||||
---|---|---|---|---|
|
View file | ||||
---|---|---|---|---|
|
(a) Single tree algorithm (b) Multiple tree algorithm
Figure 34. Proposed Reduce and Broadcast algorithms.
...
(c) Reduce (d) Broadcast
Figure 45. Block diagram of proposed addition. Changes to old initialization (InitMergeBuffersAndComm), Reduce and Broadcast are illustrated.
...
When to switch between Single and Multiple tree
Figure 56. VGG-16 performance as function of MXNET_KVSTORE_BIGARRAY_BOUND using batch size 4 per GPU.
...
Vs. Parameter Server (in comm.h) | Vs. NCCL (in kvstore_nccl.h) | |
---|---|---|
Resnet-50 | 1.19 | 1.33 |
VGG-16 | 5.89 | 1.06 |
Inception-v3 | 1.15 | 1.34 |
AlexNet | 6.60 | 1.42 |
Figure 67. End-to-end training results on synthetic data showing speed-up vs. NCCL on fp32 and fp16.
...