Improved sparse SGD, sparse AdaGrad and sparse Adam optimizer speed on GPU by 30x (#9561, #10312, #10293, #10062).
Improved 'sparse.retain' performance on CPU by 2.5x (#9722)
Replaced std::swap_ranges with memcpy (#10351)
Implemented DepthwiseConv2dBackwardFilterKernel which is over 5x faster (#10098)
Implemented CPU LSTM Inference (#9977)
Added Layer Normalization in C++ (#10029)
Optimized Performance for rtc (#10018)
Improved CPU performance of ROIpooling operator by using OpenMP (#9958)
Accelerated the calculation of F1 (#9833)

API Changes

Changed Namespace changes for mxnet scala from ml.dmlc.mxnet to org.apache.mxnet (#10284).
Added copy flag for astype (#10347).
Added context parameter to Scala Infer API - ImageClassifier and ObjectDetector (#10252).
Added axes support for dropout in gluon (#10032).
Added default `ctx` to cpu for `gluon.Block.load_params` (#10160).
Added support for variable sequence length in gluon.RecurrentCell (#9934).
Added convenience fluent method for squeeze op (#9734).
Made array.reshape compatible with numpy (#9790).
Added axis support and gradient for L2norm (#9740)

...

Other Features

Added constant parameter for gluon (#9893).
Added contrib.rand.zipfian (#9747).
Added Gluon PreLU, ELU, SELU, Swish activation layers for Gluon (#9662)
Added Squeeze Op (#9700).
Added multi-proposal operator (CPU version) and fixed bug in multi-proposal operator (GPU version) (#9939).
Added in Large-Batch SGD with a warmup, and a LARS startegy. (#8918)
Added Language Modelling datasets and Sampler (#9514).
Added instance norm and reflection padding to Gluon (#7938).
Added micro-averaging strategy for F1 metric (#9777).
Added Softsign Activation Function (#9851).
Added eye operator, for default storage type (#9770).
Added TVM bridge support to JIT NDArray Function by TVM (#9880).
Added float16 support for correlation operator and L2Normalization operator (#10125, #10078).
Added random shuffle implementation for NDArray (#10048).
Added load from buffer functions for CPP package (#10261).

...