Inference Performance

This group of the performance test is gathered on AWS EC2 instances in 1 socket.

Performance boost with Intel MKL-DNN backend in release 1.3

- w/o MKL-DNN, pip install mxnet==1.3.0

- w/ MKL-DNN, pip install mxnet-mkl==1.3.0

Category	Model	Latency batchsize=1 (ms, small is better)			Throughput batchsize=128 (fps, big is better)
Category	Model	w/o MKL-DNN	w/ MKL-DNN	speedup	w/o MKL-DNN	w/ MKL-DNN	speedup
CNN/classification	ResNet-50 v1
	ResNet-50 v2
	Inception v3
	Inception v4
	DenseNet
	MobileNet
	VGG16
	AlexNet
	inception-resnet v2
CNN/object detection	Faster R-CNN
	SSD-VGG16
	SSD-MobileNet
RNN	GNMT
GAN	DCGAN

Performance gain from operator fusion by subgraph

- R1.3 w/ MKL-DNN, pip install mxnet-mkl==1.3.0
- master w/ subgraph, CI https://github.com/apache/incubator-mxnet/commit/213ab09e7a2924da436c0d0526d62fefeeea6aa7
  build: make USE_OPENCV=1 USE_MKLDNN=1 USE_BLAS=mkl USE_INTEL_PATH=/opt/intel/ -j
  runtime env: export MXNET_SUBGRAPH_BACKEND=MKLDNN

Category	Model	Latency batchsize=1 (ms, small is better)			Throughput batchsize=128 (fps, big is better)
Category	Model	R1.3 w/ MKL-DNN	master w/ subgraph	speedup	R1.3 w/ MKL-DNN	master w/ subgraph	speedup
CNN/classification	ResNet-50 v1
	ResNet-50 v2
	Inception v3
	Inception v4
	DenseNet
	MobileNet
	VGG16
	AlexNet
	inception-resnet v2
CNN/object detection	Faster R-CNN
	SSD-VGG16
	SSD-MobileNet
RNN	GNMT
GAN	DCGAN

Inference Accuracy

Inference Accuracy Comparison
Alias	Network	# Parameters	Official Results		MKL-DNN Backend
Alias	Network	# Parameters	Top-1 Accuracy	Top-5 Accuracy
alexnet	AlexNet	61,100,840	0.5492	0.7803
densenet121	DenseNet-121	8,062,504	0.7497	0.9225
densenet161	DenseNet-161	28,900,936	0.777	0.938
densenet169	DenseNet-169	14,307,880	0.7617	0.9317
densenet201	DenseNet-201	20,242,984	0.7732	0.9362
inceptionv3	Inception V3 299x299	23,869,000	0.7755	0.9364
mobilenet0.25	MobileNet 0.25	475,544	0.5185	0.7608
mobilenet0.5	MobileNet 0.5	1,342,536	0.6307	0.8475
mobilenet0.75	MobileNet 0.75	2,601,976	0.6738	0.8782
mobilenet1.0	MobileNet 1.0	4,253,864	0.7105	0.9006
mobilenetv2_1.0	MobileNetV2 1.0	3,539,136	0.7192	0.9056
mobilenetv2_0.75	MobileNetV2 0.75	2,653,864	0.6961	0.8895
mobilenetv2_0.5	MobileNetV2 0.5	1,983,104	0.6449	0.8547
mobilenetv2_0.25	MobileNetV2 0.25	1,526,856	0.5074	0.7456
resnet18_v1	ResNet-18 V1	11,699,112	0.7093	0.8992
resnet34_v1	ResNet-34 V1	21,814,696	0.7437	0.9187
resnet50_v1	ResNet-50 V1	25,629,032	0.7647	0.9313
resnet101_v1	ResNet-101 V1	44,695,144	0.7834	0.9401
resnet152_v1	ResNet-152 V1	60,404,072	0.79	0.9438
resnet18_v2	ResNet-18 V2	11,695,796	0.71	0.8992
resnet34_v2	ResNet-34 V2	21,811,380	0.744	0.9208
resnet50_v2	ResNet-50 V2	25,595,060	0.7711	0.9343
resnet101_v2	ResNet-101 V2	44,639,412	0.7853	0.9417
resnet152_v2	ResNet-152 V2	60,329,140	0.7921	0.9431
squeezenet1.0	SqueezeNet 1.0	1,248,424	0.5611	0.7909
squeezenet1.1	SqueezeNet 1.1	1,235,496	0.5496	0.7817
vgg11	VGG-11	132,863,336	0.6662	0.8734
vgg13	VGG-13	133,047,848	0.6774	0.8811
vgg16	VGG-16	138,357,544	0.7323	0.9132
vgg19	VGG-19	143,667,240	0.7411	0.9135
vgg11_bn	VGG-11 with batch normalization	132,874,344	0.6859	0.8872
vgg13_bn	VGG-13 with batch normalization	133,059,624	0.6884	0.8882
vgg16_bn	VGG-16 with batch normalization	138,374,440	0.731	0.9176
vgg19_bn	VGG-19 with batch normalization	143,689,256	0.7433	0.9185

Page tree

MXNet with Intel MKL-DNN - Performance Benchmarking (WIP)

Inference Performance

Performance boost with Intel MKL-DNN backend in release 1.3

Performance gain from operator fusion by subgraph

Inference Accuracy