Model Backwards Compatibility Check

1. Problem

Currently, there is no mechanism to verify consistency and integrity of models trained on older MXNet versions. Any unwarranted change in an underlying model saving/loading API could potentially break the backwards compatibility across MXNet versions. 

2. Goal

Model Backward compatibility aims to check whether models trained on earlier versions of MXNet are loading fine on the latest version or the latest release candidate. It also aims to do a sanity check for consistency in the inference on these trained models.

3. Approach

Here's a proposed approach to do this : 

1. Train simple models on earlier versions of MXNet, save the model and model parameters and upload them in an S3 bucket. 
2. During the training phase perform a simple inference on randomly generated data and save the randomly generated data as well as the inference output along with the model files on S3. 
3. The inference script running on the latest master branch of MXNet repository, would pull the model files + data and would try to load the models back into memory. The tests would fail if the models fail to load into memory or they give a different inference output. The different inference output could indicate or flag a potential change in an underlying operator. 
4. Use the same seed values to ensure we have the same environment for both training and inference files. 
5. These tests could be a part of nightly build process and would help in flagging out the above mentioned issues.