...

• This is the first step towards optimizing dynamic neural networks by adding symbolic and imperative control flow operators. Proposal 
• New operators introduced: foreach(#11531), while_loop(#11566), cond(#11760).

New Features - MXNet Model Backwards Compatibility Checker

PR (#11626)

MKL-DNN

Breaking Changes

Big-fixes

Fix Flaky Tests

Performance Improvements

API Changes

Other features

...

Test CreateMKLDNNMem/CommitOutput MKLDNN helper functions.

Flaky Tests effort

Add flakiness checker (#11572)

Bug-fixes

Fix gperftools/jemalloc and lapack warning bug. (#11110)

Fix mkldnn performance regression + improve test logging (#11262)

Fix row_sparse_param.save() (#11266)

Fix trainer init_kvstore (#11266)

Fix axis Bug in MKLDNN Softmax (#11335)

Fix 'AttributeError: '_thread._local' object has no attribute 'value'' on distributed processing applications (#11332)

Fix recordfile dataset with multi worker (#11370)

Manually check node existence in CachedOp (#11545)

Javadoc fix (#11239)

Fix bugs in MKLDNN operators to handle the kAddTo request (#11129)

Fix InferStorage for sparse fallback in FullyConnected (#11498)

Fix batchnorm problem with sparse matrices when fix_gamma=True (#11656)

Fix rnn layer save (#11776)

Fix BucketSentenceIter bug related to #11430 (#11580)

Fix for _backward_softsign activation (#11827)

Fix a bug in CachedOp. (#11675)

Fix quantization divide by zero errors (#11833)

Refactor R optimizers to fix memory leak (#11374)

Avoid use of troublesome cudnnFind() results when grad_req='add' (#11338)

Fix shared memory with gluon dataloader, add option pin_memory (#11908)

Fix quantized graph pass bug (#11937)

Fix MXPredReshape in the c_predict_api (#11493)

Performance Improvements

Added static allocation for HybridBloc gluon (#11320)

Fix RecordIO augmentation speed (#11474)

Improve sparse pull performance for gluon trainer (#11429)

Improve performance of broadcast ops backward pass (#11252)

Accelerate the performance of topk for CPU side (#12085)

Fix the topk regression issue (#12197)

API Changes

Allow Scala users to specify data/label names for NDArrayIter (#11256)

Allow user to define unknown token symbol to rnn encode_sentences() (#10461)

Added count_include_pad argument for Avg Pooling (#11021)

Add standard ResNet data augmentation for ImageRecordIter (#11027)

Add seed_aug parameter for ImageRecordIter to fix random seed for default augmentation (#11247)

Add support for accepting MXNet NDArrays in ColorNormalizeAug (#11606)

Enhancement of take operator (#11326)

Add temperature parameter in Softmax operator (#11466)

Add support for 1D inputs in leaky relu (#11850)

Add verify_ssl option to gluon.utils.download (#11546)

Other features

Fix Flaky Tests

Added ccache reporting to CI (#11322)

Restructure dockcross dockerfiles to fix caching (#11302)

Added tests for MKLDNN backward operators  (#11232)

Add elemwise_add/sub between rsp and rsp on GPU (#11179)

Add clip_global_norm(row_sparse_grad) (#11266)

Add subgraph storage type inference to CachedOp  (#11306)

Enable support for dense weight and sparse grad Adagrad updates (#11355)

Added Histogram Operator (#10931)

Added Matthew's Correlation Coefficient to metrics (#10524)

Added support for add_n(dense, csr, dense) = dense on CPU & GPU (#11330)

Added support for add_n(any combination longer than 4 with at least one dense storage) = dense on CPU & GPU (#11330)

L1 Normalization (#11229)

Add support for int64 data type in CSVIter (#11446)

Add test for new int64 type in CSVIter (#11499)

Add sample ratio for ROI Align (#11145)

Shape and Size Operator (#10889)

Add HybidSequentialRNNCell, which can be nested in HybridBlock (#10514)

Support for a bunch of unary functions for csr matrices (#11559)

Added stable norm 2 Reducer (#11573)

Added Synchronized Batch Normalization (#11502)

Added NDArrayCollector to dispose intermediate allocated NDArrays automatically (#11751)

Added the diag() operator (#11643)

Added broadcast_like operator (#11820)

Allow Partial shape infer for Slice (#11406)

Added support to profile kvstore server during distributed training  (#11215)

Make gluon rnn layers hybrid blocks (#11482)

Add function for GPU Memory Query to C API (#12083)

Generalized reshape_like operator to be more flexible (#11928)

Add support for selu activation function (#12059)

Add support for accepting NDArray as input to Module predict API (#12166)

Add DataDesc type for the Scala Package (#11844)

Usability Improvements

Added docs for mx.initializer.Constant (#10637)

Added build from souce instructions on windows (#11276)

Added a tutorial explaining how to use the profiler (#11274)

Added 2 tutorials on Learning Rate Schedules (#11296)

Tutorial for mixed precision training with float16 (#10391)

Create CPP test for concat MKLDNN operator (#11371)

Update large word language model example (#11405)

MNIST Examples for Scala new API (#11250)

Updated installation info to have latest packages and more clarity (#11503)

GAN MNIST Examples for Scala new API (#11547)

Added Learning Rate Finder tutorial (#11304)

Fix Installation instructions for R bindings on Linux systems. (#11590)

Integration Test for Scala (#11596)

Documentation enhancement for optimizers (#11657)

Update rcnn example (#11373)

Gluon ModelZoo, Gluon examples for Perl APIs (#11642)

Fix R installation in CI (#11761,#11755, #11768, #11805, #11954, #11976)

CNN Examples for Scala new API (#11292)

Custom Operator Example for Scala (#11401)

Added detailed doc about global pool layers in Gluon (#11832)

Updated MultiTask example to use new infer api (#11605)

Add logistic regression tutorial (#11651)

Add Support for integer type in ImageIter (#11864)

Added depth_to_space and space_to_depth operators (#11587)

Increase operator support for ONNX to MXNet importer (#11856)

Add linux and macos MKLDNN Building Instruction (#11049)

Add download utility for Scala APIs (#11866)

Improving documentation and error messages for Async distributed training with Gluon (#11910)

Add NeuralStyle Example for Scala (#11621)

Knows Issues

How to build MXNet

...