Table of Contents | ||
---|---|---|
|
Introduction
This page details benchmark results comparing MXNet 1.3.0 with MKLDNN vs without MKLDNN (integration proposal). The results clearly shows that MKL-DNN boosts inference throughput between 6x to 37x, latency reduced between 2x to 41x, while accuracy is equivalent up to an epsilon of 1e-8.
Inference Performance
This group of the performance test is gathered on AWS EC2 instances in instance C5.18xLarge with 1 socket and 1 processor.
...
For the throughput, 2 sockets can provide about 2X speedup while latency will keep the constant.
Performance on Intel CPU with Intel MKL-DNN backend in release 1.3
w/o MKL-DNN, pip install mxnet==1.3.0
...
The c5.18xlarge instance offers a 2-socket Intel Xeon Platinum processor with 72 vCPUs.
$ export KMP_AFFINITY=granularity=fine,compact,1,0
$ export OMP_NUM_THREADS=18
$ numactl --physcpubind=0-17 --membind=0 python …
Category | Model | Latency batchsize=1 (ms, small is better) | Throughput batchsize=128 (fps, big is better) | ||||
---|---|---|---|---|---|---|---|
w/o MKL-DNN | w/ MKL-DNN | speedup | w/o MKL-DNN | w/ MKL-DNN | speedup | ||
CNN/classification | ResNet-50 v1 | 97.19 | 13.04 | 7.45 | 10.29 | 163.52 | 15.90 |
ResNet-50 v2 | 98.69 | 13.02 | 7.58 | 9.94 | 154.17 | 15.51 | |
Inception v3 | 175.17 | 16.77 | 10.44 | 5.74 | 135.33 | 23.57 | |
Inception v4 | 330.93 | 31.40 | 10.54 | 3.04 | 69.60 | 22.87 | |
DenseNet | 111.66 | 18.90 | 5.91 | 8.52 | 149.88 | 17.60 | |
MobileNet | 38.56 | 4.42 | 8.73 | 24.87 | 512.25 | 20.60 | |
VGG16 | 406.50 | 20.07 | 20.25 | 2.91 | 70.84 | 24.31 | |
AlexNet | 64.60 | 3.80 | 17.00 | 26.58 | 965.20 | 36.32 | |
inception-resnet v2 | 181.10 | 49.40 | 3.67 | 5.48 | 82.97 | 15.14 | |
CNN/object detection | Faster R-CNN | 1175.74 | 118.62 | 9.91 | 0.85 | 8.57 | 10.08 |
SSD-VGG16 | 721.03 | 47.62 | 15.14 | 1.43(batchsize=224) | 28.90(batchsize=224) | 19.13 | |
SSD-MobileNet | 239.40 | 28.33 | 8.45 | 4.07(batchsize=256) | 69.97(batchsize=256) | 14.18 | |
RNN | GNMT | 683.43 | 94.00 | 7.27 | 1.46(batchsize=64) | 10.63(batchsize=64) | 6.83 |
GAN | DCGAN |
Performance gain from operator fusion by subgraph
- R1.3 w/ MKL-DNN, pip install mxnet-mkl==1.3.0
- master w/ subgraph, CI https://github.com/apache/incubator-mxnet/commit/213ab09e7a2924da436c0d0526d62fefeeea6aa7
build: make USE_OPENCV=1 USE_MKLDNN=1 USE_BLAS=mkl USE_INTEL_PATH=/opt/intel/ -j
runtime env: export MXNET_SUBGRAPH_BACKEND=MKLDNN
8.94 | 0.24 | 37.85 | 109.13 | 4249.36 | 38.94 |
Performance AMD CPU with Intel MKL-DNN backend in release 1.3
The m5a.24xlarge offers 96 vCPUs using the AMD EPYC processors (AVX2)
Category
Model
Category | Model | Throughput batchsize=32 (fps, bigger is better) | ||
---|---|---|---|---|
w/o MKL-DNN | w/ MKL-DNN |
speedup | |
---|---|
CNN/classification | ResNet-50 v1 |
2.44 | 38.57 | x15.8 |
MobileNet |
CNN/object detection
RNN
Inference Accuracy
5.03 | 194.7 | x38.7 |
Inference Accuracy
The c5.18xlarge instance offers a 2-socket Intel Xeon Platinum processor with 72 vCPUs.
The model is from gluon model zoo by pre-trained parameters. The top1 and top5 accuracy are verified by MKL-DNN backend.
As below table shown, the accuracy from MXNet 1.3 without and with MKL-DNN got the exact same results with 10e-8.
Note: The dataset used ImageNet1k valdata/ are generated by imagenet1k-val.sh
Inference Accuracy Comparison | |||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Alias | Network | # Parameters | Official Results | CPU (without MKL-DNN) | CPU (with MKL-DNN) Backend | Top-1 Accuracy | Top-5 Accuracy | Delta | |||||||
top1 | top5 | top1 | top5 | top1 | top5 | ||||||||||
alexnet | AlexNet61,100,840 | 0.56312500 | 0.78992188 | 0.549256312500 | 0.780378992188 | 0.00000000 | 0.00000000 | ||||||||
densenet121 | DenseNet-121 | 0.74203125 | 0.919296888,062,504 | 0.749774203125 | 0.922591929688 | 0.00000000 | 0.00000000 | ||||||||
densenet161 | DenseNet-161 | 0.77195313 | 0.9339062528,900,936 | 0.77777195313 | 0.93893390625 | 0.00000000 | 0.00000000 | ||||||||
densenet169 | DenseNet-169 | 0.75710938 | 0.9282812514,307,880 | 0.761775710938 | 0.931792828125 | 0.00000000 | 0.00000000 | ||||||||
densenet201 | DenseNet-201 | 0.76906250 | 0.9309375020,242,984 | 0.773276906250 | 0.936293093750 | 0.00000000 | 0.00000000 | ||||||||
inceptionv3 | Inception V3 299x299 | 0.77609375 | 0.9366406323,869,000 | 0.775577609375 | 0.936493664063 | 0.00000000 | 0.00000000 | ||||||||
mobilenet0.25 | MobileNet 0.25 | 0.51039063 | 0.75687500475,544 | 0.518551039063 | 0.760875687500 | 0.00000000 | 0.00000000 | ||||||||
mobilenet0.5 | MobileNet 0.5 | 0.61851563 | 0.837890631,342,536 | 0.630761851563 | 0.847583789063 | 0.00000000 | 0.00000000 | ||||||||
mobilenet0.75 | MobileNet 0.75 | 0.66546875 | 0.870703132,601,976 | 0.673866546875 | 0.878287070313 | 0.00000000 | 0.00000000 | ||||||||
mobilenet1.0 | MobileNet 1.0 | 0.70093750 | 0.891093754,253,864 | 0.710570093750 | 0.900689109375 | 0.00000000 | 0.00000000 | ||||||||
mobilenetv2_1.0 | MobileNetV2 1.0 | 0.69976563 | 0.892812503,539,136 | 0.719269976563 | 0.905689281250 | 0.00000000 | 0.00000000 | ||||||||
mobilenetv2_0.75 | MobileNetV2 0.75 | 0.68210938 | 0.880078132,653,864 | 0.696168210938 | 0.889588007813 | 0.00000000 | 0.00000000 | ||||||||
mobilenetv2_0.5 | MobileNetV2 0.5 | 0.64453125 | 0.849296881,983,104 | 0.644964453125 | 0.854784929688 | 0.00000000 | 0.00000000 | ||||||||
mobilenetv2_0.25 | MobileNetV2 0.25 | 0.50890625 | 0.745468751,526,856 | 0.507450890625 | 0.745674546875 | 0.00000000 | 0.00000000 | ||||||||
resnet18_v1 | ResNet-18 V1 | 0.70812500 | 0.8945312511,699,112 | 0.709370812500 | 0.899289453125 | 0.00000000 | 0.00000000 | ||||||||
resnet34_v1 | ResNet-34 V1 | 0.73960938 | 0.9160937521,814,696 | 0.743773960938 | 0.918791609375 | 0.00000000 | 0.00000000 | ||||||||
resnet50_v1 | ResNet-50 V1 | 0.76062500 | 0.9304687525,629,032 | 0.764776062500 | 0.931393046875 | 0.00000000 | 0.00000000 | ||||||||
resnet101_v1 | ResNet-101 V1 | 0.77937500 | 0.9361718844,695,144 | 0.783477937500 | 0.940193617188 | 0.00000000 | 0.00000000 | ||||||||
resnet152_v1 | ResNet-152 V1 | 0.78320313 | 0.9386718860,404,072 | 0.7978320313 | 0.943893867188 | 0.00000000 | 0.00000000 | ||||||||
resnet18_v2 | ResNet-18 V2 | 0.71046875 | 0.8967187511,695,796 | 0.7171046875 | 0.899289671875 | 0.00000000 | 0.00000000 | ||||||||
resnet34_v2 | ResNet-34 V2 | 0.74085938 | 0.9157812521,811,380 | 0.74474085938 | 0.920891578125 | 0.00000000 | 0.00000000 | ||||||||
resnet50_v2 | ResNet-50 V2 | 0.76750000 | 0.9318750025,595,060 | 0.771176750000 | 0.934393187500 | 0.00000000 | 0.00000000 | ||||||||
resnet101_v2 | ResNet-101 V2 | 0.78125000 | 0.9401562544,639,412 | 0.785378125000 | 0.941794015625 | 0.00000000 | 0.00000000 | ||||||||
resnet152_v2 | ResNet-152 V2 | 0.78554688 | 0.9414062560,329,140 | 0.792178554688 | 0.943194140625 | 0.00000000 | 0.00000000 | ||||||||
squeezenet1.0 | SqueezeNet 1.01,248,424 | 0.57273438 | 0.79554688 | 0.561157273438 | 0.790979554688 | 0.00000000 | 0.00000000 | ||||||||
squeezenet1.1 | SqueezeNet 1.11,235,496 | 0.57023438 | 0.79601563 | 0.549657023438 | 0.781779601563 | 0.00000000 | 0.00000000 | ||||||||
vgg11 | VGG-11 | 0.67062500 | 0.87531250132,863,336 | 0.666267062500 | 0.873487531250 | 0.00000000 | 0.00000000 | ||||||||
vgg13 | VGG-13 | 0.68132813 | 0.87984375133,047,848 | 0.677468132813 | 0.881187984375 | 0.00000000 | 0.00000000 | ||||||||
vgg16 | VGG-16 | 0.72062500 | 0.90585938138,357,544 | 0.732372062500 | 0.913290585938 | 0.00000000 | 0.00000000 | ||||||||
vgg19 | VGG-19 | 0.73468750 | 0.91000000143,667,240 | 0.741173468750 | 0.913591000000 | 0.00000000 | 0.00000000 | ||||||||
vgg11_bn | VGG-11 with batch normalization | 0.68953125 | 0.88882813132,874,344 | 0.685968953125 | 0.887288882813 | 0.00000000 | 0.00000000 | ||||||||
vgg13_bn | VGG-13 with batch normalization | 0.69835938 | 0.88953125133,059,624 | 0.688469835938 | 0.888288953125 | 0.00000000 | 0.00000000 | ||||||||
vgg16_bn | VGG-16 with batch normalization | 0.72226563 | 0.90390625138,374,440 | 0.73172226563 | 0.917690390625 | 0.00000000 | 0.00000000 | ||||||||
vgg19_bn | VGG-19 with batch normalization | 0.72992188 | 0.90992188 | 0.72992188 | 0.90992188143,689,256 | 0.743300000000 | 0.9185 | 00000000 |
CMD for Reproducing Result
Please access the script and model from the link below.
https://drive.google.com/open?id=17JenLnZKsmPoZIIyktINFfMjZtDY2Ehc
(Note: select the parent folder and click download in the drop-down menu)
You can refer to launch_benchmark_aws.sh for reproducing.