...

• On average, after the MKL-DNN change, the inference speed of MXNet + MKLDNN outperforms MXNet + OpenBLAS by a factor of 32, outperforms MXNet + MKLML by 82% and outperforms MXNet + MKLML with the experimental flag by 8%. The experiments were run for the image classifcation example, for different networks and different batch sizes.
• Improved sparse SGD, sparse AdaGrad and sparse Adam optimizer speed on GPU by 30x (#9561, #10312, #10293, #10062).
• Improved 'sparse.retain' performance on CPU by 2.5x (#9722)
• Replaced std::swap_ranges with memcpy (#10351)
• Implemented DepthwiseConv2dBackwardFilterKernel which is over 5x faster (#10098)
• Implemented CPU LSTM Inference (#9977)
• Added Layer Normalization in C++ (#10029)
• Optimized Performance for rtc (#10018)
• Improved CPU performance of  ROIpooling operator by using OpenMP (#9958)
• Accelerated the calculation of F1 (#9833)

API Changes

• Added an optional argument `ctx` to `mx.random.seed`. Seed with `ctx` option produces random number sequence independent of device id. (#10367).
• Added copy flag for astype (#10347).
• Added context parameter to Scala Infer API - ImageClassifier and ObjectDetector (#10252).
• Added axes support for dropout in gluon (#10032).
• Added default `ctx` to cpu for `gluon.Block.load_params` (#10160).
• Added support for variable sequence length in gluon.RecurrentCell (#9934).
• Added convenience fluent method for squeeze op (#9734).
• Made array.reshape compatible with numpy (#9790).
• Added axis support and gradient for L2norm (#9740).

...