THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
| Table of Contents | ||
|---|---|---|
|
Introduction
This page details benchmark results comparing MXNet 1.3.0 with MKLDNN vs without MKLDNN (integration proposal). The results clearly shows that MKL-DNN boosts inference throughput between 6x to 37x, latency reduced between 2x to 41x, while accuracy is equivalent up to an epsilon of 1e-8.
Inference Performance
This group of the performance test is gathered on AWS EC2 instances
...
instance C5.18xLarge with 1 socket and 1 processor.
For the throughput, 2 sockets can provide about 2X speedup while latency will keep the constant.
Performance on Intel CPU with Intel MKL-DNN backend in release 1.3
w/o MKL-DNN, pip install mxnet==1.3.0
...
The c5.18xlarge instance offers a 2-socket Intel Xeon Platinum processor with 72 vCPUs.
$ export KMP_AFFINITY=granularity=fine,compact,1,0
$ export OMP_NUM_THREADS=18
$ numactl --physcpubind=0-17 --membind=0 python …
| Category | Model | Latency batchsize=1 (ms, small is better) | Throughput batchsize=128 (fps, big is better) | ||||
|---|---|---|---|---|---|---|---|
| w/o MKL-DNN | w/ MKL-DNN | speedup | w/o MKL-DNN | w/ MKL-DNN | speedup | ||
| CNN/classification | ResNet-50 v1 | 97.19 | 13.04 | 7.45 | 10.29 | 163.52 | 15.90 |
| ResNet-50 v2 | 98.69 | 13.02 | 7.58 | 9.94 | 154.17 | 15.51 | |
| Inception v3 | 175.17 | 16.77 | 10.44 | 5.74 | 135.33 | 23.57 | |
| Inception v4 | 330.93 | 31.40 | 10.54 | 3.04 | 69.60 | 22.87 | |
| DenseNet | 111.66 | 18.90 | 5.91 | 8.52 | 149.88 | 17.60 | |
| MobileNet | 38.56 | 4.42 | 8.73 | 24.87 | 512.25 | 20.60 | |
| VGG16 | 406.50 | 20.07 | 20.25 | 2.91 | 70.84 | 24.31 | |
| AlexNet | 64.60 | 3.80 | 17.00 | 26.58 | 965.20 | 36.32 | |
| inception-resnet v2 | 181.10 | 49.40 | 3.67 | 5.48 | 82.97 | 15.14 | |
| CNN/object detection | Faster R-CNN | 1175.74 | 118.62 | 9.91 | 0.85 | 8.57 | 10.08 |
| SSD-VGG16 | 721.03 | 47.62 | 15.14 | 1.43（batchsize=224) | 28.90(batchsize=224) | 19.13 | |
| SSD-MobileNet | 239.40 | 28.33 | 8.45 | 4.07(batchsize=256) | 69.97(batchsize=256) | 14.18 | |
| RNN | GNMT | 683.43 | 94.00 | 7.27 | 1.46(batchsize=64) | 10.63(batchsize=64) | 6.83 |
| GAN | DCGAN | ||||||
Performance gain from operator fusion by subgraph
| 8.94 | 0.24 | 37.85 | 109.13 | 4249.36 | 38.94 |
Performance AMD CPU with Intel MKL-DNN backend in release 1.3
The m5a.24xlarge offers 96 vCPUs using the AMD EPYC processors (AVX2)
Category
Model
| Category | Model | Throughput batchsize=32 (fps, bigger is better) | ||
|---|---|---|---|---|
| w/o MKL-DNN | ||||
| w/ MKL-DNN |
|---|
| speedup | |
|---|---|
| CNN/classification | ResNet-50 v1 |
| 2.44 | 38.57 | x15.8 |
| MobileNet |
CNN/object detection
RNN
Inference Accuracy
| 5.03 | 194.7 | x38.7 |
Inference Accuracy
The c5.18xlarge instance offers a 2-socket Intel Xeon Platinum processor with 72 vCPUs.
The model is from gluon model zoo by pre-trained parameters. The top1 and top5 accuracy are verified by MKL-DNN backend.
As below table shown, the accuracy from MXNet 1.3 without and with MKL-DNN got the exact same results with 10e-8.
Note: The dataset used ImageNet1k valdata/ are generated by imagenet1k-val.sh
| Inference Accuracy | Inference AccuracyComparison | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Alias | Network | # Parameters | Official Results | CPU (without MKL-DNN) | CPU (with MKL-DNN) Backend | Top-1 Accuracy | Top-5 Accuracy | Delta | |||||||
| top1 | top5 | top1 | top5 | top1 | top5 | ||||||||||
| alexnet | AlexNet61,100,840 | 0.56312500 | 0.78992188 | 0.549256312500 | 0.780378992188 | 0.00000000 | 0.00000000 | ||||||||
| densenet121 | DenseNet-121 | 0.74203125 | 0.919296888,062,504 | 0.749774203125 | 0.922591929688 | 0.00000000 | 0.00000000 | ||||||||
| densenet161 | DenseNet-161 | 0.77195313 | 0.9339062528,900,936 | 0.77777195313 | 0.93893390625 | 0.00000000 | 0.00000000 | ||||||||
| densenet169 | DenseNet-169 | 0.75710938 | 0.9282812514,307,880 | 0.761775710938 | 0.931792828125 | 0.00000000 | 0.00000000 | ||||||||
| densenet201 | DenseNet-201 | 0.76906250 | 0.9309375020,242,984 | 0.773276906250 | 0.936293093750 | 0.00000000 | 0.00000000 | ||||||||
| inceptionv3 | Inception V3 299x299 | 0.77609375 | 0.9366406323,869,000 | 0.775577609375 | 0.936493664063 | 0.00000000 | 0.00000000 | ||||||||
| mobilenet0.25 | MobileNet 0.25 | 0.51039063 | 0.75687500475,544 | 0.518551039063 | 0.760875687500 | 0.00000000 | 0.00000000 | ||||||||
| mobilenet0.5 | MobileNet 0.5 | 0.61851563 | 0.837890631,342,536 | 0.630761851563 | 0.847583789063 | 0.00000000 | 0.00000000 | ||||||||
| mobilenet0.75 | MobileNet 0.75 | 0.66546875 | 0.870703132,601,976 | 0.673866546875 | 0.878287070313 | 0.00000000 | 0.00000000 | ||||||||
| mobilenet1.0 | MobileNet 1.0 | 0.70093750 | 0.891093754,253,864 | 0.710570093750 | 0.900689109375 | 0.00000000 | 0.00000000 | ||||||||
| mobilenetv2_1.0 | MobileNetV2 1.0 | 0.69976563 | 0.892812503,539,136 | 0.719269976563 | 0.905689281250 | 0.00000000 | 0.00000000 | ||||||||
| mobilenetv2_0.75 | MobileNetV2 0.75 | 0.68210938 | 0.880078132,653,864 | 0.696168210938 | 0.889588007813 | 0.00000000 | 0.00000000 | ||||||||
| mobilenetv2_0.5 | MobileNetV2 0.5 | 0.64453125 | 0.849296881,983,104 | 0.644964453125 | 0.854784929688 | 0.00000000 | 0.00000000 | ||||||||
| mobilenetv2_0.25 | MobileNetV2 0.25 | 0.50890625 | 0.745468751,526,856 | 0.507450890625 | 0.745674546875 | 0.00000000 | 0.00000000 | ||||||||
| resnet18_v1 | ResNet-18 V1 | 0.70812500 | 0.8945312511,699,112 | 0.709370812500 | 0.899289453125 | 0.00000000 | 0.00000000 | ||||||||
| resnet34_v1 | ResNet-34 V1 | 0.73960938 | 0.9160937521,814,696 | 0.743773960938 | 0.918791609375 | 0.00000000 | 0.00000000 | ||||||||
| resnet50_v1 | ResNet-50 V1 | 0.76062500 | 0.9304687525,629,032 | 0.764776062500 | 0.931393046875 | 0.00000000 | 0.00000000 | ||||||||
| resnet101_v1 | ResNet-101 V1 | 0.77937500 | 0.9361718844,695,144 | 0.783477937500 | 0.940193617188 | 0.00000000 | 0.00000000 | ||||||||
| resnet152_v1 | ResNet-152 V1 | 0.78320313 | 0.9386718860,404,072 | 0.7978320313 | 0.943893867188 | 0.00000000 | 0.00000000 | ||||||||
| resnet18_v2 | ResNet-18 V2 | 0.71046875 | 0.8967187511,695,796 | 0.7171046875 | 0.899289671875 | 0.00000000 | 0.00000000 | ||||||||
| resnet34_v2 | ResNet-34 V2 | 0.74085938 | 0.9157812521,811,380 | 0.74474085938 | 0.920891578125 | 0.00000000 | 0.00000000 | ||||||||
| resnet50_v2 | ResNet-50 V2 | 0.76750000 | 0.9318750025,595,060 | 0.771176750000 | 0.934393187500 | 0.00000000 | 0.00000000 | ||||||||
| resnet101_v2 | ResNet-101 V2 | 0.78125000 | 0.9401562544,639,412 | 0.785378125000 | 0.941794015625 | 0.00000000 | 0.00000000 | ||||||||
| resnet152_v2 | ResNet-152 V2 | 0.78554688 | 0.9414062560,329,140 | 0.792178554688 | 0.943194140625 | 0.00000000 | 0.00000000 | ||||||||
| squeezenet1.0 | SqueezeNet 1.01,248,424 | 0.57273438 | 0.79554688 | 0.561157273438 | 0.790979554688 | 0.00000000 | 0.00000000 | ||||||||
| squeezenet1.1 | SqueezeNet 1.11,235,496 | 0.57023438 | 0.79601563 | 0.549657023438 | 0.781779601563 | 0.00000000 | 0.00000000 | ||||||||
| vgg11 | VGG-11 | 0.67062500 | 0.87531250132,863,336 | 0.666267062500 | 0.873487531250 | 0.00000000 | 0.00000000 | ||||||||
| vgg13 | VGG-13 | 0.68132813 | 0.87984375133,047,848 | 0.677468132813 | 0.881187984375 | 0.00000000 | 0.00000000 | ||||||||
| vgg16 | VGG-16 | 0.72062500 | 0.90585938138,357,544 | 0.732372062500 | 0.913290585938 | 0.00000000 | 0.00000000 | ||||||||
| vgg19 | VGG-19 | 0.73468750 | 0.91000000143,667,240 | 0.741173468750 | 0.913591000000 | 0.00000000 | 0.00000000 | ||||||||
| vgg11_bn | VGG-11 with batch normalization | 0.68953125 | 0.88882813132,874,344 | 0.685968953125 | 0.887288882813 | 0.00000000 | 0.00000000 | ||||||||
| vgg13_bn | VGG-13 with batch normalization | 0.69835938 | 0.88953125133,059,624 | 0.688469835938 | 0.888288953125 | 0.00000000 | 0.00000000 | ||||||||
| vgg16_bn | VGG-16 with batch normalization | 0.72226563 | 0.90390625138,374,440 | 0.73172226563 | 0.917690390625 | 0.00000000 | 0.00000000 | ||||||||
| vgg19_bn | VGG-19 with batch normalization | 0.72992188 | 0.90992188 | 0.72992188 | 0.90992188143,689,256 | 0.743300000000 | 0.9185 | .00000000 | |||||||
CMD for Reproducing Result
Please access the script and model from the link below.
https://drive.google.com/open?id=17JenLnZKsmPoZIIyktINFfMjZtDY2Ehc
(Note: select the parent folder and click download in the drop-down menu)
You can refer to launch_benchmark_aws.sh for reproducing.