Introduction
This page details benchmark results comparing MXNet 1.3.0 with MKLDNN vs without MKLDNN (integration proposal). The results clearly shows that MKL-DNN boosts inference throughput between 6x to 37x, latency reduced between 2x to 41x, while accuracy is exactly equivalentequivalent up to an epsilon of 1e-8.
This group of the performance test is gathered on AWS EC2 instance C5.18xLarge with 1 socket and 1 processor.
For the throughput, 2 sockets can provide about 2X speedup while latency will keep the constant.
Performance
...
on Intel CPU with Intel MKL-DNN backend in release 1.3
The c5.18xlarge instance offers a 2-socket Intel Xeon Platinum processor with 72 vCPUs.
...
$ numactl --physcpubind=0-17 --membind=0 python …
Category | Model | Latency batchsize=1 (ms, small is better) | Throughput batchsize=128 (fps, |
---|
higher no mkldnn | release 1.3 + mkldnn | speedup | no mkldnn | w/o MKL-DNN | w/ MKL-DNN | speedup | w/o MKL-DNN | w/ MKL-DNN |
---|
release 1.3 + mkldnn | speedup |
---|
CNN/classification | ResNet-50 v1 | 97.19 |
1894513132051284189352112712792634665110001916669649459281947533120912179143073252738054153040081014692396433149068986259632 |
inception-resnet v2 | 181.10 |
111281636939126614 |
CNN/object detection | Faster R-CNN | 1175.74 |
951512361051123612748566273590(batchsize=224) | 19.13 |
SSD-MobileNet |
23910075 239 4577318 1003068199763(batchsize=64) | 6.83 |
GAN | DCGAN | 8.94 | 0. |
2241364059.74 | 37.20Performance AMD CPU with Intel MKL-DNN backend in release 1.3
The m5a.24xlarge offers 96 vCPUs using the AMD EPYC processors (AVX2)
Category | Model | Throughput batchsize=32 (fps, bigger is better) |
---|
w/o MKL-DNN | w/ MKL-DNN | speedup |
---|
CNN/classification | ResNet-50 v1 | 2.44 | 38.57 | x15.8
|
MobileNet | 5.03 | 194.7 | x38.7 |
The c5.18xlarge instance offers a 2-socket Intel Xeon Platinum processor with 72 vCPUs.
The model is from gluon model zoo by pre-trained parameters. The top1 and top5 accuracy are verified by MKL-DNN backend.
...
Inference Accuracy Comparison |
---|
Alias | Network | CPU (without MKL-DNN) | CPU (with MKL-DNN) Backend | Delta |
---|
top1 | top5 | top1 | top5 | top1 | top5 |
---|
alexnet | AlexNet | 0.56312500 | 0.78992188 | 0.56312500 | 0.78992188 | 0.00000000 | 0.00000000 |
densenet121 | DenseNet-121 | 0.74203125 | 0.91929688 | 0.74203125 | 0.91929688 | 0.00000000 | 0.00000000 |
densenet161 | DenseNet-161 | 0.77195313 | 0.93390625 | 0.77195313 | 0.93390625 | 0.00000000 | 0.00000000 |
densenet169 | DenseNet-169 | 0.75710938 | 0.92828125 | 0.75710938 | 0.92828125 | 0.00000000 | 0.00000000 |
densenet201 | DenseNet-201 | 0.76906250 | 0.93093750 | 0.76906250 | 0.93093750 | 0.00000000 | 0.00000000 |
inceptionv3 | Inception V3 299x299 | 0.77609375 | 0.93664063 | 0.77609375 | 0.93664063 | 0.00000000 | 0.00000000 |
mobilenet0.25 | MobileNet 0.25 | 0.51039063 | 0.75687500 | 0.51039063 | 0.75687500 | 0.00000000 | 0.00000000 |
mobilenet0.5 | MobileNet 0.5 | 0.61851563 | 0.83789063 | 0.61851563 | 0.83789063 | 0.00000000 | 0.00000000 |
mobilenet0.75 | MobileNet 0.75 | 0.66546875 | 0.87070313 | 0.66546875 | 0.87070313 | 0.00000000 | 0.00000000 |
mobilenet1.0 | MobileNet 1.0 | 0.70093750 | 0.89109375 | 0.70093750 | 0.89109375 | 0.00000000 | 0.00000000 |
mobilenetv2_1.0 | MobileNetV2 1.0 | 0.69976563 | 0.89281250 | 0.69976563 | 0.89281250 | 0.00000000 | 0.00000000 |
mobilenetv2_0.75 | MobileNetV2 0.75 | 0.68210938 | 0.88007813 | 0.68210938 | 0.88007813 | 0.00000000 | 0.00000000 |
mobilenetv2_0.5 | MobileNetV2 0.5 | 0.64453125 | 0.84929688 | 0.64453125 | 0.84929688 | 0.00000000 | 0.00000000 |
mobilenetv2_0.25 | MobileNetV2 0.25 | 0.50890625 | 0.74546875 | 0.50890625 | 0.74546875 | 0.00000000 | 0.00000000 |
resnet18_v1 | ResNet-18 V1 | 0.70812500 | 0.89453125 | 0.70812500 | 0.89453125 | 0.00000000 | 0.00000000 |
resnet34_v1 | ResNet-34 V1 | 0.73960938 | 0.91609375 | 0.73960938 | 0.91609375 | 0.00000000 | 0.00000000 |
resnet50_v1 | ResNet-50 V1 | 0.76062500 | 0.93046875 | 0.76062500 | 0.93046875 | 0.00000000 | 0.00000000 |
resnet101_v1 | ResNet-101 V1 | 0.77937500 | 0.93617188 | 0.77937500 | 0.93617188 | 0.00000000 | 0.00000000 |
resnet152_v1 | ResNet-152 V1 | 0.78320313 | 0.93867188 | 0.78320313 | 0.93867188 | 0.00000000 | 0.00000000 |
resnet18_v2 | ResNet-18 V2 | 0.71046875 | 0.89671875 | 0.71046875 | 0.89671875 | 0.00000000 | 0.00000000 |
resnet34_v2 | ResNet-34 V2 | 0.74085938 | 0.91578125 | 0.74085938 | 0.91578125 | 0.00000000 | 0.00000000 |
resnet50_v2 | ResNet-50 V2 | 0.76750000 | 0.93187500 | 0.76750000 | 0.93187500 | 0.00000000 | 0.00000000 |
resnet101_v2 | ResNet-101 V2 | 0.78125000 | 0.94015625 | 0.78125000 | 0.94015625 | 0.00000000 | 0.00000000 |
resnet152_v2 | ResNet-152 V2 | 0.78554688 | 0.94140625 | 0.78554688 | 0.94140625 | 0.00000000 | 0.00000000 |
squeezenet1.0 | SqueezeNet 1.0 | 0.57273438 | 0.79554688 | 0.57273438 | 0.79554688 | 0.00000000 | 0.00000000 |
squeezenet1.1 | SqueezeNet 1.1 | 0.57023438 | 0.79601563 | 0.57023438 | 0.79601563 | 0.00000000 | 0.00000000 |
vgg11 | VGG-11 | 0.67062500 | 0.87531250 | 0.67062500 | 0.87531250 | 0.00000000 | 0.00000000 |
vgg13 | VGG-13 | 0.68132813 | 0.87984375 | 0.68132813 | 0.87984375 | 0.00000000 | 0.00000000 |
vgg16 | VGG-16 | 0.72062500 | 0.90585938 | 0.72062500 | 0.90585938 | 0.00000000 | 0.00000000 |
vgg19 | VGG-19 | 0.73468750 | 0.91000000 | 0.73468750 | 0.91000000 | 0.00000000 | 0.00000000 |
vgg11_bn | VGG-11 with batch normalization | 0.68953125 | 0.88882813 | 0.68953125 | 0.88882813 | 0.00000000 | 0.00000000 |
vgg13_bn | VGG-13 with batch normalization | 0.69835938 | 0.88953125 | 0.69835938 | 0.88953125 | 0.00000000 | 0.00000000 |
vgg16_bn | VGG-16 with batch normalization | 0.72226563 | 0.90390625 | 0.72226563 | 0.90390625 | 0.00000000 | 0.00000000 |
vgg19_bn | VGG-19 with batch normalization | 0.72992188 | 0.90992188 | 0.72992188 | 0.90992188 | 0.00000000 | 0.00000000 |
CMD for Reproducing Result
Please access the script and model from the link below.
https://drive.google.com/open?id=17JenLnZKsmPoZIIyktINFfMjZtDY2Ehc
(Note: select the parent folder and click download in the drop-down menu)
You can refer to launch_benchmark_aws.sh for reproducing.