...

1. New loss functions added - `SigmoidBinaryCrossEntropyLoss`, `CTCLoss`, `HuberLoss`, `HingeLoss`, `SquaredHingeLoss`, `LogisticLoss`, `TripletLoss`

2. `gluon.Trainer` now allows reading and setting learning rate with `trainer.learning_rate` property.

3. Added `mx.autograd.grad` and experimental second order gradient support (though most operators don't support second order gradient yet)

4. Added `ConvLSTM` etc to `gluon.contrib`

5. Autograd now supports cross-device graphs. Use `x.copyto(mx.gpu(i))` and `x.copyto(mx.cpu())` to do computation on multiple devices.

Other New Features

1. (Almost) full support for fancy indexing (Waiting for Jun’s PR)

2. Random number generators in `mx.nd.random.*` and `mx.sym.random.*` now supports both CPU and GPU

3. `NDArray` and `Symbol` now supports "fluent" methods. You can now use `x.exp()` etc instead of `mx.nd.exp(x)` or `mx.sym.exp(x)`

4. Added `mx.rtc.CudaModule` for writing and running CUDA kernels from python

5. Added `multi_precision` option to optimizer for easier float16 training

...