Page History

...

      Following is the overview architecture of mpi-allreduce and parameter server:Image Removed

Image Added                                     

                                      AllReduce-based distributed training

...

Image Added

                                        PS         PS-based distributed training

No coincidence, in 2016-2017, both baidu Baidu and Uber propose their mpi-allreduce-based distributed training framework (tensorflow-allreduce, horovod) for tensorflow, considering the drawbacks we mentioned in tensorflow inherent parameter server based distributed training.

...