Page History

Many neural network models in the fields, such as natural language processing and graph analysis, are dynamic. To truly express the dynamic neural networks, we need to use flow control. From the perspective of the usability, Python control flow is preferred to implement dynamic models. However, lack of symbolic flow control operators makes it difficult for MXNet to handle these dynamic models in some cases.

Let's take RNN models for example (many RNN models are dynamic). Currently, MXNet provides many solutions to handle some of the RNN workloads. For example, in many cases where If a model is defined for a known sequence length of an RNN, we can unroll an RNN cell accordingly to construct a static graph to process the sequences. If we use a standard cell, such as LSTM and GRU, we can also call a pre-defined kernel, such as CuDNN RNN, to process the entire sequences in a single operator. For sequences with variable lengths, MXNet pads data to a specified length and uses a bucketing mechanism to save computation. For more complex and more dynamic workloads, we can use Gluon to implement the models and rely on Python flow control to apply RNN cells on each element of a sequence.

Even though these solutions can handle RNN models in many cases, they have some limitations:

...