Added comprehensive support for sparse matrices. See help on mx.sym.sparse and mx.nd.sparse for more info
Added limited cpu support for two sparse formats in Symbol and NDArray - CSRNDArray and RowSparseNDArray
Added a sparse dot product operator and many element-wise sparse operators
Added a data iterator for sparse data input - LibSVMIter
Added three optimizers for sparse gradient updates: Ftrl, SGD and Adam
Added push and row_sparse_pull with RowSparseNDArray in distributed kvstore

New Features - Gluon

...

Added enhancements to the `Gluon` package, a high-level interface designed to be easy to use while keeping most of the flexibility of low level API. Gluon supports both imperative and symbolic programming, making it easy to train complex models imperatively with minimal impact on performance. Neural networks (and other machine learning models) can be defined and trained with `gluon.nn` and `gluon.rnn` packages.
`gluon.Trainer` now allows reading and setting learning rate with `trainer.learning_rate` property.Added
Added `ConvLSTM` to `gluon.contrib`
New Features - Autograd
Added enhancements to `autograd` package, which enables automatic differentiation of NDArray operations. Enter code in a `autograd.record()` block to capture the exact path by which each variable was generated. Use `*.backward()` on any variable to backpropagate.
Added `mx.autograd.grad` and experimental second order gradient support (most operators don't support second order gradient yet)
...
Added new loss functions - `SigmoidBinaryCrossEntropyLoss`, `CTCLoss`, `HuberLoss`, `HingeLoss`, `SquaredHingeLoss`, `LogisticLoss`, `TripletLoss`
...
.
...
Autograd now supports cross-device graphs.
...
Use `x.copyto(mx.gpu(i))`
...
and `x.copyto(mx.cpu())`
...
to do computation on multiple devices.

Added limited support for fancy indexing, which allows you to very quickly access and modify complicated subsets of an array's values. x[idx_arr0, idx_arr1, ..., idx_arrn] is now supported. Full support coming soon in next release. Checkout master to get a preview.
Added new loss functions - SigmoidBinaryCrossEntropyLoss, CTCLoss, HuberLoss, HingeLoss, SquaredHingeLoss, LogisticLoss, TripletLoss
Random number generators in mx.nd.random.* and mx.sym.random.* now supports support both CPU and GPU
NDArray and Symbol now supports "fluent" methods. You can now use x.exp() etc instead of mx.nd.exp(x) or mx.sym.exp(x)
Added mx.rtc.CudaModule for writing and running CUDA kernels from python
Added multi_precision option to optimizer for easier float16 training

...