Introduction
This page details benchmark results comparing MXNet 1.3.0 with MKLDNN vs without MKLDNN (integration proposal). The results clearly shows that MKL-DNN boosts inference throughput between 6x to 37x, latency reduced between 2x to 41x, while accuracy is equivalent up to an epsilon of 1e-8.
This group of the performance test is gathered on AWS EC2 instance C5.18xLarge with 1 socket and 1 processor.
For the throughput, 2 sockets can provide about 2X speedup while latency will keep the constant.
Performance
...
on Intel CPU with Intel MKL-DNN backend in release 1.3
...
The c5.18xlarge instance offers a 2-socket Intel Xeon Platinum processor with 72 vCPUs.
$ export KMP_AFFINITY=granularity=fine,compact,1,0
$ export OMP_NUM_THREADS=18
$ numactl --physcpubind=0-17 --membind=0 python …
Category | Model | Latency batchsize=1 (ms, small is better) | Throughput batchsize=128 (fps, big is better) |
---|
no mkldnn | release 1.3 + mkldnn | speedup | no mkldnn | w/o MKL-DNN | w/ MKL-DNN | speedup | w/o MKL-DNN | w/ MKL-DNN |
---|
release 1.3 + mkldnn | speedup |
---|
CNN/classification | ResNet-50 v1 | 97.19 |
1894513132051284189352112712792634665110001916669649459281947533120912179143073252738054153040081014692396433149068986259632 |
inception-resnet v2 | 181.10 |
111281636939126614 |
CNN/object detection | Faster R-CNN | 1175.74 |
951512361051123612748566273590(batchsize=224) | 19.13 |
SSD-MobileNet |
23910075 239 4577318 1003068199763(batchsize=64) | 6.83 |
GAN | DCGAN | 8.94 | 0. |
2241364059.74 | 37.20Performance AMD CPU with Intel MKL-DNN backend in release 1.3
The m5a.24xlarge offers 96 vCPUs using the AMD EPYC processors (AVX2)
Category | Model | Throughput batchsize=32 (fps, bigger is better) |
---|
w/o MKL-DNN | w/ MKL-DNN | speedup |
---|
CNN/classification | ResNet-50 v1 | 2.44 | 38.57 | x15.8
|
MobileNet | 5.03 | 194.7 | x38.7 |
The c5.18xlarge instance offers a 2-socket Intel Xeon Platinum processor with 72 vCPUs.
The model is from gluon model zoo by pre-trained parameters. The top1 and top5 accuracy are verified by MKL-DNN backend.
As below table shown, the accuracy from MXNet 1.3 without and with MKL-DNN got the exact same results with 10e-8.
Note: The dataset used ImageNet1k valdata/ are generated by imagenet1k-val.sh
Inference Accuracy Comparison |
---|
Alias | Network | # Parameters | GPU (with cuDNN) Backend | CPU (without MKL-DNN) | CPU (with MKL-DNN) Backend | Delta |
---|
top1 | top5 top5 | top1 | top5 | top1top1 | top5top5 |
---|
alexnet | AlexNet | 61,100,840 | 0.56312556312500 | 0.789921978992188 | 0.56312556312500 | 0.78992188 | 0.56312500000000 | 0.789921900000000 |
densenet121 | DenseNet-121 | 8,062,504 | 0.74203125 | 0.919296991929688 | 0.74203125 | 0.91929688 | 0.742031300000000 | 0.919296900000000 |
densenet161 | DenseNet-16128,900,936 | 0.77195313 | 0.933906393390625 | 0.77195313 | 0.93390625 | 0.771953100000000 | 0.933906300000000 |
densenet169 | DenseNet-16914,307,880 | 0.75710938 | 0.928281392828125 | 0.75710938 | 0.92828125 | 0.757109400000000 | 0.928281300000000 |
densenet201 | DenseNet-20120,242,984 | 0.769062576906250 | 0.930937593093750 | 0.769062576906250 | 0.930937593093750 | 0.769062500000000 | 0.930937500000000 |
inceptionv3 | Inception V3 299x299 | 23,869,000 | 0.77609375 | 0.936640693664063 | 0.77609375 | 0.93664063 | 0.776093800000000 | 0.936640600000000 |
mobilenet0.25 | MobileNet 0.25 | 475,544 | 0.51039063 | 0.75687575687500 | 0.51039063 | 0.75687575687500 | 0.510390600000000 | 0.75687500000000 |
mobilenet0.5 | MobileNet 0.51,342,536 | 0.61851563 | 0.837890683789063 | 0.61851563 | 0.83789063 | 0.618515600000000 | 0.837890600000000 |
mobilenet0.75 | MobileNet 0.75 | 2,601,976 | 0.66546875 | 0.870703187070313 | 0.66546875 | 0.87070313 | 0.665468800000000 | 0.870703100000000 |
mobilenet1.0 | MobileNet 1.04,253,864 | 0.700937570093750 | 0.891093889109375 | 0.700937570093750 | 0.89109375 | 0.700937500000000 | 0.891093800000000 |
mobilenetv2_1.0 | MobileNetV2 1.0 | 3,539,136 | 0.69976563 | 0.892812589281250 | 0.69976563 | 0.892812589281250 | 0.699765600000000 | 0.892812500000000 |
mobilenetv2_0.75 | MobileNetV2 0.75 | 2,653,864 | 0.68210938 | 0.880078188007813 | 0.68210938 | 0.88007813 | 0.682109400000000 | 0.880078100000000 |
mobilenetv2_0.5 | MobileNetV2 0.51,983,104 | 0.64453125 | 0.849296984929688 | 0.64453125 | 0.84929688 | 0.644531300000000 | 0.849296900000000 |
mobilenetv2_0.25 | MobileNetV2 0.25 | 1,526,856 | 0.50890625 | 0.745468874546875 | 0.50890625 | 0.74546875 | 0.508906300000000 | 0.745468800000000 |
resnet18_v1 | ResNet-18 V1 | 11,699,112 | 0.70812570812500 | 0.894531389453125 | 0.70812570812500 | 0.89453125 | 0.70812500000000 | 0.894531300000000 |
resnet34_v1 | ResNet-34 V1 | 21,814,696 | 0.73960938 | 0.916093891609375 | 0.73960938 | 0.91609375 | 0.739609400000000 | 0.916093800000000 |
resnet50_v1 | ResNet-50 V125,629,032 | 0.76062576062500 | 0.930468893046875 | 0.76062576062500 | 0.93046875 | 0.76062500000000 | 0.930468800000000 |
resnet101_v1 | ResNet-101 V1 | 44,695,144 | 0.77937577937500 | 0.936171993617188 | 0.77937577937500 | 0.93617188 | 0.77937500000000 | 0.936171900000000 |
resnet152_v1 | ResNet-152 V1 | 60,404,072 | 0.78320313 | 0.938671993867188 | 0.78320313 | 0.93867188 | 0.783203100000000 | 0.938671900000000 |
resnet18_v2 | ResNet-18 V211,695,796 | 0.71046875 | 0.896718889671875 | 0.71046875 | 0.89671875 | 0.710468800000000 | 0.896718800000000 |
resnet34_v2 | ResNet-34 V2 | 21,811,380 | 0.74085938 | 0.915781391578125 | 0.74085938 | 0.91578125 | 0.740859400000000 | 0.915781300000000 |
resnet50_v2 | ResNet-50 V2 | 25,595,060 | 0.767576750000 | 0.93187593187500 | 0.767576750000 | 0.93187593187500 | 0.767500000000 | 0.93187500000000 |
resnet101_v2 | ResNet-101 V244,639,412 | 0.7812578125000 | 0.940156394015625 | 0.7812578125000 | 0.94015625 | 0.7812500000000 | 0.940156300000000 |
resnet152_v2 | ResNet-152 V2 | 60,329,140 | 0.78554688 | 0.941406394140625 | 0.78554688 | 0.94140625 | 0.785546900000000 | 0.941406300000000 |
squeezenet1.0 | SqueezeNet 1.01,248,424 | 0.57273438 | 0.795546979554688 | 0.57273438 | 0.79554688 | 0.572734400000000 | 0.795546900000000 |
squeezenet1.1 | SqueezeNet 1.1 | 1,235,496 | 0.57023438 | 0.796015679601563 | 0.57023438 | 0.79601563 | 0.570234400000000 | 0.796015600000000 |
vgg11 | VGG-11132,863,336 | 0.67062567062500 | 0.875312587531250 | 0.67062567062500 | 0.875312587531250 | 0.67062500000000 | 0.875312500000000 |
vgg13 | VGG-13133,047,848 | 0.68132813 | 0.879843887984375 | 0.68132813 | 0.87984375 | 0.681328100000000 | 0.879843800000000 |
vgg16 | VGG-16 | 138,357,544 | 0.72062572062500 | 0.905859490585938 | 0.72062572062500 | 0.90585938 | 0.72062500000000 | 0.905859400000000 |
vgg19 | VGG-19143,667,240 | 0.734687573468750 | 0.9191000000 | 0.734687573468750 | 0.9191000000 | 0.734687500000000 | 0.9100000000 |
vgg11_bn | VGG-11 with batch normalization | 132,874,344 | 0.68953125 | 0.888828188882813 | 0.68953125 | 0.88882813 | 0.689531300000000 | 0.888828100000000 |
vgg13_bn | VGG-13 with batch normalization | 133,059,624 | 0.69835938 | 0.889531388953125 | 0.69835938 | 0.88953125 | 0.698359400000000 | 0.889531300000000 |
vgg16_bn | VGG-16 with batch normalization138,374,440 | 0.72226563 | 0.903906390390625 | 0.72226563 | 0.90390625 | 0.722265600000000 | 0.903906300000000 |
vgg19_bn | VGG-19 with batch normalization | 143,689,256 | 0.72992188 | 0.909921990992188 | 0.72992188 | 0.90992188 | 0.729921900000000 | 0.9099219.00000000 |
CMD for Reproducing Result
Please access the script and model from the link below.
https://drive.google.com/open?id=17JenLnZKsmPoZIIyktINFfMjZtDY2Ehc
(Note: select the parent folder and click download in the drop-down menu)
You can refer to launch_benchmark_aws.sh for reproducing.