...
Following is the overview architecture of mpi-allreduce and parameter server:
AllReduce-based distributed training
...
PS PS-based distributed training
No coincidence, in 2016-2017, both baidu Baidu and Uber propose their mpi-allreduce-based distributed training framework (tensorflow-allreduce, horovod) for tensorflow, considering the drawbacks we mentioned in tensorflow inherent parameter server based distributed training.
...