Introduction
This page details benchmark results comparing MXNet 1.3.0 with MKLDNN vs without MKLDNN (integration proposal). The results clearly shows that MKL-DNN boosts inference throughput between 6x to 37x, latency reduced between 2x to 41x, while accuracy is equivalent up to an epsilon of 1e-8.
This group of the performance test is gathered on AWS EC2 instance C5.18xLarge with 1 socket and 1 processor.
For the throughput, 2 sockets can provide about 2X speedup while latency will keep the constant.
Performance
...
on Intel CPU with Intel MKL-DNN backend in release 1.3
...
The c5.18xlarge instance offers a 2-socket Intel Xeon Platinum processor with 72 vCPUs.
$ export KMP_AFFINITY=granularity=fine,compact,1,0
$ export OMP_NUM_THREADS=18
$ numactl --physcpubind=0-17 --membind=0 python …
Category | Model | Latency batchsize=1 (ms, small is better) | Throughput batchsize=128 (fps, |
---|
higher no mkldnn | release 1.3 + mkldnn | speedup | no mkldnn | w/o MKL-DNN | w/ MKL-DNN | speedup | w/o MKL-DNN | w/ MKL-DNN |
---|
release 1.3 + mkldnn | speedup |
---|
CNN/classification | ResNet-50 v1 | 97.19 |
1894513132051284189352112712792634665110001916669649459281947533120912179143073252738054153040081014692396433149068986259632 |
inception-resnet v2 | 181.10 |
111281636939126614 |
CNN/object detection | Faster R-CNN | 1175.74 |
951512361051123612748566273590(batchsize=224) | 19.13 |
SSD-MobileNet |
23910075 239 4577318 1003068199763(batchsize=64) | 6.83 |
GAN | DCGAN | 8.94 | 0. |
2241364059.74 | 37.20 | ...
Performance AMD CPU with Intel MKL-DNN backend in release 1.3
The m5a.24xlarge offers 96 vCPUs using the AMD EPYC processors (AVX2)
Category | Model | Throughput batchsize=32 (fps, bigger is better) |
---|
w/o MKL-DNN | w/ MKL-DNN | speedup |
---|
CNN/classification | ResNet-50 v1 | 2.44 | 38.57 | x15.8
|
MobileNet | 5.03 | 194.7 | x38.7 |
The c5.18xlarge instance offers a 2-socket Intel Xeon Platinum processor with 72 vCPUs.
The model is from gluon model zoo by pre-trained parameters. The top1 and top5 accuracy are verified by MKL-DNN backend.
As below table shown, the accuracy from MXNet 1.3 without and with MKL-DNN got the exact same results with 10e-8.
Note: The dataset used ImageNet1k valdata/ are generated by imagenet1k-val.sh
Inference Accuracy Comparison |
---|
Alias | Network | # Parameters | CPU (without MKL-DNN) | CPU (with MKL-DNN) Backend | ) | Delta |
---|
top1 | top5 | top1 | top5 | top1 | top5 |
---|
alexnet | AlexNet61,100,840 | 0.56312500 | 0.78992188 | 0.56312556312500 | 0.78992188 | 0.56312500000000 | 0.789921900000000 |
densenet121 | DenseNet-121 | 0.74203125 | 0.919296888,062,504 | 0.74203125 | 0.91929688 | 0.742031300000000 | 0.919296900000000 |
densenet161 | DenseNet-161 | 0.77195313 | 0.9339062528,900,936 | 0.77195313 | 0.93390625 | 0.771953100000000 | 0.933906300000000 |
densenet169 | DenseNet-169 | 0.75710938 | 0.9282812514,307,880 | 0.75710938 | 0.92828125 | 0.757109400000000 | 0.928281300000000 |
densenet201 | DenseNet-201 | 0.76906250 | 0.9309375020,242,984 | 0.769062576906250 | 0.930937593093750 | 0.769062500000000 | 0.930937500000000 |
inceptionv3 | Inception V3 299x299 | 0.77609375 | 0.9366406323,869,000 | 0.77609375 | 0.93664063 | 0.776093800000000 | 0.936640600000000 |
mobilenet0.25 | MobileNet 0.25 | 0.51039063 | 0.75687500475,544 | 0.51039063 | 0.75687575687500 | 0.510390600000000 | 0.75687500000000 |
mobilenet0.5 | MobileNet 0.5 | 0.61851563 | 0.837890631,342,536 | 0.61851563 | 0.83789063 | 0.618515600000000 | 0.837890600000000 |
mobilenet0.75 | MobileNet 0.75 | 0.66546875 | 0.870703132,601,976 | 0.66546875 | 0.87070313 | 0.665468800000000 | 0.870703100000000 |
mobilenet1.0 | MobileNet 1.0 | 0.70093750 | 0.891093754,253,864 | 0.700937570093750 | 0.89109375 | 0.700937500000000 | 0.891093800000000 |
mobilenetv2_1.0 | MobileNetV2 1.0 | 0.69976563 | 0.892812503,539,136 | 0.69976563 | 0.892812589281250 | 0.699765600000000 | 0.892812500000000 |
mobilenetv2_0.75 | MobileNetV2 0.75 | 0.68210938 | 0.880078132,653,864 | 0.68210938 | 0.88007813 | 0.682109400000000 | 0.880078100000000 |
mobilenetv2_0.5 | MobileNetV2 0.5 | 0.64453125 | 0.849296881,983,104 | 0.64453125 | 0.84929688 | 0.644531300000000 | 0.849296900000000 |
mobilenetv2_0.25 | MobileNetV2 0.25 | 0.50890625 | 0.745468751,526,856 | 0.50890625 | 0.74546875 | 0.508906300000000 | 0.745468800000000 |
resnet18_v1 | ResNet-18 V1 | 0.70812500 | 0.8945312511,699,112 | 0.70812570812500 | 0.89453125 | 0.70812500000000 | 0.894531300000000 |
resnet34_v1 | ResNet-34 V1 | 0.73960938 | 0.9160937521,814,696 | 0.73960938 | 0.91609375 | 0.739609400000000 | 0.916093800000000 |
resnet50_v1 | ResNet-50 V1 | 0.76062500 | 0.9304687525,629,032 | 0.76062576062500 | 0.93046875 | 0.76062500000000 | 0.930468800000000 |
resnet101_v1 | ResNet-101 V1 | 0.77937500 | 0.9361718844,695,144 | 0.77937577937500 | 0.93617188 | 0.77937500000000 | 0.936171900000000 |
resnet152_v1 | ResNet-152 V1 | 0.78320313 | 0.9386718860,404,072 | 0.78320313 | 0.93867188 | 0.783203100000000 | 0.938671900000000 |
resnet18_v2 | ResNet-18 V2 | 0.71046875 | 0.8967187511,695,796 | 0.71046875 | 0.89671875 | 0.710468800000000 | 0.896718800000000 |
resnet34_v2 | ResNet-34 V2 | 0.74085938 | 0.9157812521,811,380 | 0.74085938 | 0.91578125 | 0.740859400000000 | 0.915781300000000 |
resnet50_v2 | ResNet-50 V2 | 0.76750000 | 0.9318750025,595,060 | 0.767576750000 | 0.93187593187500 | 0.767500000000 | 0.93187500000000 |
resnet101_v2 | ResNet-101 V2 | 0.78125000 | 0.9401562544,639,412 | 0.7812578125000 | 0.94015625 | 0.7812500000000 | 0.940156300000000 |
resnet152_v2 | ResNet-152 V2 | 0.78554688 | 0.9414062560,329,140 | 0.78554688 | 0.94140625 | 0.785546900000000 | 0.941406300000000 |
squeezenet1.0 | SqueezeNet 1.01,248,424 | 0.57273438 | 0.79554688 | 0.57273438 | 0.79554688 | 0.572734400000000 | 0.795546900000000 |
squeezenet1.1 | SqueezeNet 1.11,235,496 | 0.57023438 | 0.79601563 | 0.57023438 | 0.79601563 | 0.570234400000000 | 0.796015600000000 |
vgg11 | VGG-11 | 0.67062500 | 0.87531250132,863,336 | 0.67062567062500 | 0.875312587531250 | 0.67062500000000 | 0.875312500000000 |
vgg13 | VGG-13 | 0.68132813 | 0.87984375133,047,848 | 0.68132813 | 0.87984375 | 0.681328100000000 | 0.879843800000000 |
vgg16 | VGG-16 | 0.72062500 | 0.90585938138,357,544 | 0.72062572062500 | 0.90585938 | 0.72062500000000 | 0.905859400000000 |
vgg19 | VGG-19 | 0.73468750 | 0.91000000143,667,240 | 0.734687573468750 | 0.9191000000 | 0.734687500000000 | 0.9100000000 |
vgg11_bn | VGG-11 with batch normalization | 0.68953125 | 0.88882813132,874,344 | 0.68953125 | 0.88882813 | 0.689531300000000 | 0.888828100000000 |
vgg13_bn | VGG-13 with batch normalization | 0.69835938 | 0.88953125133,059,624 | 0.69835938 | 0.88953125 | 0.698359400000000 | 0.889531300000000 |
vgg16_bn | VGG-16 with batch normalization | 0.72226563 | 0.90390625138,374,440 | 0.72226563 | 0.90390625 | 0.722265600000000 | 0.903906300000000 |
vgg19_bn | VGG-19 with batch normalization | 0.72992188 | 0.90992188143,689,256 | 0.72992188 | 0.90992188 | 0.729921900000000 | 0.9099219.00000000 |
CMD for Reproducing Result
Please access the script and model from the link below.
https://drive.google.com/open?id=17JenLnZKsmPoZIIyktINFfMjZtDY2Ehc
(Note: select the parent folder and click download in the drop-down menu)
You can refer to launch_benchmark_aws.sh for reproducing.