Versions Compared

Old Version 11

changes.mady.by.user Carl Yang

Saved on

compared with

New Version 12

changes.mady.by.user Carl Yang

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The ring Reduce communication pattern used by NCCL (Figure 1a) and Parameter server Reduce (Figure 1b) currently used in MXNet are not optimal for small batch sizes on p3.16xlarge instances with 8 GPUs.

Note: To clarify, usage of "parameter server" refers to a single-machine communication rather than the distributed sense the term is usually used in. This may cause some confusion.

...