Table of Contents | ||
---|---|---|
|
Introduction
This page details benchmark results comparing MXNet 1.3.0 with MKLDNN vs without MKLDNN (integration proposal). The results clearly shows that MKL-DNN boosts inference throughput between 6x to 37x, latency reduced between 2x to 41x, while accuracy is equivalent up to an epsilon of 1e-8.
Inference Performance
This group of the performance test is gathered on AWS EC2 instance C5.18xLarge with 1 socket and 1 processor.
For the throughput, 2 sockets can provide about 2X speedup while latency will keep the constant.
Performance
...
on Intel CPU with Intel MKL-DNN backend in release 1.3
The c5.18xlarge instance offers a 2-socket Intel Xeon Platinum processor with 72 vCPUs.
...
Category | Model | Latency batchsize=1 (ms, small is better) | Throughput batchsize=128 (fps, big is better) | ||||
---|---|---|---|---|---|---|---|
w/o MKL-DNN | w/ MKL-DNN | speedup | w/o MKL-DNN | w/ MKL-DNN | speedup | ||
CNN/classification | ResNet-50 v1 | 97.19 | 13.04 | 7.45 | 10.29 | 163.52 | 15.90 |
ResNet-50 v2 | 98.69 | 13.02 | 7.58 | 9.94 | 154.17 | 15.51 | |
Inception v3 | 175.17 | 16.77 | 10.44 | 5.74 | 135.33 | 23.57 | |
Inception v4 | 330.93 | 31.40 | 10.54 | 3.04 | 69.60 | 22.87 | |
DenseNet | 111.66 | 18.90 | 5.91 | 8.52 | 149.88 | 17.60 | |
MobileNet | 38.56 | 4.42 | 8.73 | 24.87 | 512.25 | 20.60 | |
VGG16 | 406.50 | 20.07 | 20.25 | 2.91 | 70.84 | 24.31 | |
AlexNet | 64.60 | 3.80 | 17.00 | 26.58 | 965.20 | 36.32 | |
inception-resnet v2 | 181.10 | 49.40 | 3.67 | 5.48 | 82.97 | 15.14 | |
CNN/object detection | Faster R-CNN | 1175.74 | 118.62 | 9.91 | 0.85 | 8.57 | 10.08 |
SSD-VGG16 | 721.03 | 47.62 | 15.14 | 1.43(batchsize=224) | 28.90(batchsize=224) | 19.13 | |
SSD-MobileNet | 239.40 | 28.33 | 8.45 | 4.07(batchsize=256) | 69.97(batchsize=256) | 14.18 | |
RNN | GNMT | 683.43 | 94.00 | 7.27 | 1.46(batchsize=64) | 10.63(batchsize=64) | 6.83 |
GAN | DCGAN | 8.94 | 0.24 | 37.85 | 109.13 | 4249.36 | 38.94 |
Performance AMD CPU with Intel MKL-DNN backend in release 1.3
The m5a.24xlarge offers 96 vCPUs using the AMD EPYC processors (AVX2)
Category | Model | Throughput batchsize=32 (fps, bigger is better) | ||
---|---|---|---|---|
w/o MKL-DNN | w/ MKL-DNN | speedup | ||
CNN/classification | ResNet-50 v1 | 2.44 | 38.57 | x15.8 |
MobileNet | 5.03 | 194.7 | x38.7 |
Inference Accuracy
The c5.18xlarge instance offers a 2-socket Intel Xeon Platinum processor with 72 vCPUs.
...
Inference Accuracy Comparison | |||||||
---|---|---|---|---|---|---|---|
Alias | Network | CPU (without MKL-DNN) | CPU (with MKL-DNN) Backend | Delta | |||
top1 | top5 | top1 | top5 | top1 | top5 | ||
alexnet | AlexNet | 0.56312500 | 0.78992188 | 0.56312500 | 0.78992188 | 0.00000000 | 0.00000000 |
densenet121 | DenseNet-121 | 0.74203125 | 0.91929688 | 0.74203125 | 0.91929688 | 0.00000000 | 0.00000000 |
densenet161 | DenseNet-161 | 0.77195313 | 0.93390625 | 0.77195313 | 0.93390625 | 0.00000000 | 0.00000000 |
densenet169 | DenseNet-169 | 0.75710938 | 0.92828125 | 0.75710938 | 0.92828125 | 0.00000000 | 0.00000000 |
densenet201 | DenseNet-201 | 0.76906250 | 0.93093750 | 0.76906250 | 0.93093750 | 0.00000000 | 0.00000000 |
inceptionv3 | Inception V3 299x299 | 0.77609375 | 0.93664063 | 0.77609375 | 0.93664063 | 0.00000000 | 0.00000000 |
mobilenet0.25 | MobileNet 0.25 | 0.51039063 | 0.75687500 | 0.51039063 | 0.75687500 | 0.00000000 | 0.00000000 |
mobilenet0.5 | MobileNet 0.5 | 0.61851563 | 0.83789063 | 0.61851563 | 0.83789063 | 0.00000000 | 0.00000000 |
mobilenet0.75 | MobileNet 0.75 | 0.66546875 | 0.87070313 | 0.66546875 | 0.87070313 | 0.00000000 | 0.00000000 |
mobilenet1.0 | MobileNet 1.0 | 0.70093750 | 0.89109375 | 0.70093750 | 0.89109375 | 0.00000000 | 0.00000000 |
mobilenetv2_1.0 | MobileNetV2 1.0 | 0.69976563 | 0.89281250 | 0.69976563 | 0.89281250 | 0.00000000 | 0.00000000 |
mobilenetv2_0.75 | MobileNetV2 0.75 | 0.68210938 | 0.88007813 | 0.68210938 | 0.88007813 | 0.00000000 | 0.00000000 |
mobilenetv2_0.5 | MobileNetV2 0.5 | 0.64453125 | 0.84929688 | 0.64453125 | 0.84929688 | 0.00000000 | 0.00000000 |
mobilenetv2_0.25 | MobileNetV2 0.25 | 0.50890625 | 0.74546875 | 0.50890625 | 0.74546875 | 0.00000000 | 0.00000000 |
resnet18_v1 | ResNet-18 V1 | 0.70812500 | 0.89453125 | 0.70812500 | 0.89453125 | 0.00000000 | 0.00000000 |
resnet34_v1 | ResNet-34 V1 | 0.73960938 | 0.91609375 | 0.73960938 | 0.91609375 | 0.00000000 | 0.00000000 |
resnet50_v1 | ResNet-50 V1 | 0.76062500 | 0.93046875 | 0.76062500 | 0.93046875 | 0.00000000 | 0.00000000 |
resnet101_v1 | ResNet-101 V1 | 0.77937500 | 0.93617188 | 0.77937500 | 0.93617188 | 0.00000000 | 0.00000000 |
resnet152_v1 | ResNet-152 V1 | 0.78320313 | 0.93867188 | 0.78320313 | 0.93867188 | 0.00000000 | 0.00000000 |
resnet18_v2 | ResNet-18 V2 | 0.71046875 | 0.89671875 | 0.71046875 | 0.89671875 | 0.00000000 | 0.00000000 |
resnet34_v2 | ResNet-34 V2 | 0.74085938 | 0.91578125 | 0.74085938 | 0.91578125 | 0.00000000 | 0.00000000 |
resnet50_v2 | ResNet-50 V2 | 0.76750000 | 0.93187500 | 0.76750000 | 0.93187500 | 0.00000000 | 0.00000000 |
resnet101_v2 | ResNet-101 V2 | 0.78125000 | 0.94015625 | 0.78125000 | 0.94015625 | 0.00000000 | 0.00000000 |
resnet152_v2 | ResNet-152 V2 | 0.78554688 | 0.94140625 | 0.78554688 | 0.94140625 | 0.00000000 | 0.00000000 |
squeezenet1.0 | SqueezeNet 1.0 | 0.57273438 | 0.79554688 | 0.57273438 | 0.79554688 | 0.00000000 | 0.00000000 |
squeezenet1.1 | SqueezeNet 1.1 | 0.57023438 | 0.79601563 | 0.57023438 | 0.79601563 | 0.00000000 | 0.00000000 |
vgg11 | VGG-11 | 0.67062500 | 0.87531250 | 0.67062500 | 0.87531250 | 0.00000000 | 0.00000000 |
vgg13 | VGG-13 | 0.68132813 | 0.87984375 | 0.68132813 | 0.87984375 | 0.00000000 | 0.00000000 |
vgg16 | VGG-16 | 0.72062500 | 0.90585938 | 0.72062500 | 0.90585938 | 0.00000000 | 0.00000000 |
vgg19 | VGG-19 | 0.73468750 | 0.91000000 | 0.73468750 | 0.91000000 | 0.00000000 | 0.00000000 |
vgg11_bn | VGG-11 with batch normalization | 0.68953125 | 0.88882813 | 0.68953125 | 0.88882813 | 0.00000000 | 0.00000000 |
vgg13_bn | VGG-13 with batch normalization | 0.69835938 | 0.88953125 | 0.69835938 | 0.88953125 | 0.00000000 | 0.00000000 |
vgg16_bn | VGG-16 with batch normalization | 0.72226563 | 0.90390625 | 0.72226563 | 0.90390625 | 0.00000000 | 0.00000000 |
vgg19_bn | VGG-19 with batch normalization | 0.72992188 | 0.90992188 | 0.72992188 | 0.90992188 | 0.00000000 | 0.00000000 |
Performance boost on AMD CPU with Intel MKL-DNN backend in release 1.3
The m5a.24xlarge offers 96 vCPUs using the AMD EPYC processors (AVX2)
CMD for Reproducing Result
...