New Features - Sparse Tensor Support

• Added comprehensive support for sparse matrices. See help on `mx.sym.sparse` and `mx.nd.sparse` for more info.

New Features - Autograd and Gluon

1. New loss functions added - `SigmoidBinaryCrossEntropyLoss`, `CTCLoss`, `HuberLoss`, `HingeLoss`, `SquaredHingeLoss`, `LogisticLoss`, `TripletLoss`

2. `gluon.Trainer` now allows reading and setting learning rate with `trainer.learning_rate` property.

3. Added `mx.autograd.grad` and experimental second order gradient support (though most operators don't support second order gradient yet)

4. Added `ConvLSTM` etc to `gluon.contrib`

5. Autograd now supports cross-device graphs. Use `x.copyto(mx.gpu(i))` and `x.copyto(mx.cpu())` to do computation on multiple devices.

Other New Features

1. (Almost) full support for fancy indexing (Waiting for Jun’s PR)

2. Random number generators in `mx.nd.random.*` and `mx.sym.random.*` now supports both CPU and GPU

3. `NDArray` and `Symbol` now supports "fluent" methods. You can now use `x.exp()` etc instead of `mx.nd.exp(x)` or `mx.sym.exp(x)`

4. Added `mx.rtc.CudaModule` for writing and running CUDA kernels from python

5. Added `multi_precision` option to optimizer for easier float16 training

Performance

• Full support for NVidia Volta GPU Architecture and Cuda 9. Training is up to 3.5x faster than Pascal when using float16.

API Changes

1. Operators like `mx.sym.linalg_*` and `mx.sym.random_*` are now moved to `mx.sym.linalg.*` and `mx.sym.random.*`. The old names are still available but deprecated.

2. `sample_*` and `random_*` are now merged as `random.*`, which supports both scalar and  `NDArray` distribution parameters.

Bug-fixes

1. Fixed a bug that causes `argsort` operator to fail on large tensors.

2. Fixed numerical stability issues when summing large tensors.