...

• On average, after the MKL-DNN change, the inference speed of MXNet + MKLDNN outperforms MXNet + OpenBLAS by a factor of 32, outperforms MXNet + MKLML by 82% and outperforms MXNet + MKLML with the experimental flag by 8%. The experiments were run for the image classifcation example, for different networks and different batch sizes.
• Improved sparse SGD, sparse AdaGrad and sparse Adam optimizer speed on GPU by 30x (#9561, #10312, #10293, #10062).
• Improved 'sparse.retain' performance on CPU by 2.5x (#9722)
• Replaced std::swap_ranges with memcpy (#10351)
• Implemented DepthwiseConv2dBackwardFilterKernel which is over 5x faster (#10098)
• Implemented CPU LSTM Inference (#9977)
• Added Layer Normalization in C++ (#10029)
• Optimized Performance for rtc (#10018)
• Improved CPU performance of  ROIpooling operator by using OpenMP (#9958)
• Accelerated the calculation of F1 (#9833)

API Changes

• Block.save_params now match parameters according to model structure instead of names to avoid prefix mismatching problems during saving and loading (#10511).
• Added an optional argument `ctx` to `mx.random.seed`. Seeding with `ctx` option produces random number sequence independent of device id. (#10367).
• Added copy flag for astype (#10347).
• Added context parameter to Scala Infer API - ImageClassifier and ObjectDetector (#10252).
• Added axes support for dropout in gluon (#10032).
• Added default `ctx` to cpu for `gluon.Block.load_params` (#10160).
• Added support for variable sequence length in gluon.RecurrentCell (#9934).
• Added convenience fluent method for squeeze op (#9734).
• Made array.reshape compatible with numpy (#9790).
• Added axis support and gradient for L2norm (#9740).

...

• Added embedding learning example for Gluon (#9165).
•  Added tutorial on how to use data augmenters (#10055).
• Added tutorial for Data Augmentation with Masks (#10178).
• Added LSTNet example (#9512).
• Added MobileNetV2 example (#9614).
• Added tutorial for Gluon Datasets and DataLoaders (#10251).
• Added Language model with Google's billion words dataset (#10025).
• Added example for custom operator using RTC (#9870).
• Improved image classification examples (#9799, #9633).
• Added reshape predictor function to c_predict_api (#9984).
• Added guide for implementing sparse ops (#10081).
• Added naming tutorial for gluon blocks and parameters (#10511).