Page History

...

Currently, there is no mechanism to verify consistency and integrity of models trained on older MXNet versions. Any unwarranted change in an underlying model saving/loading API could potentially break the backwards compatibility across MXNet versions. 

2. Goal

Model Backward compatibility Model Backwards Compatibility  aims to check whether models trained on earlier versions of MXNet are loading fine on the latest version or the latest release candidate. It also aims to do a sanity check for consistency in the inference on these trained models. every time maintainers need to @ a committer to add labels. This bot will help automate/simplify this issue labeling process.

...

3.

...

• Part I - Email Bot
  Create weekly email to dev@mxnet.incubator.apache.org:
  (Instead of sending emails directly to dev@, another option is to create another email alia and ask people who are interested in weekly reports to join. )
• Count of newly opened issues and closed issues in last 7 days
• Average and worst response time for all new issues
• List of non-responded new issues with links
• List of non-responded issues outside SLA
• Part II - Label Bot
  Create a bot to add labels for incubator-mxnet issues
• Create weekly email to internal team members:
• Count of newly opened issues and closed issues in last 7 days
• List of non-labelled issues
• List of non-responded issues
• Pie chart with top 10 labels for all issues
• Pie chart with top 10 labels for newly opened issues in last 7 days. （Add "unlabelled" as a segment）
• A line/bar graph with week over week statistics of the number of issues closed and the number of issues opened
• Generate a spreadsheet with detailed information of non-labelled issues. Every team member should have access to view and fill in labels to it.
• Read filled-in labels and add labels to corresponding issue.
• Part III - Determine labels automatically from GitHub issues:
  • Identify the corresponding programming language to it (ex: Python, C/C++, Scala)
  • Multi-label classification

3. Approach

• Part I - Email Bot

  Image Removed

• An amazon cloudwatch event will trigger lambda function in a certain frequency(ex: 9am every Monday). Once the lambda function is executed, the issue report will be generated and sent to the mailing list. Figure1 shows the bot design and Figure2 shows demo email content.

...

Part II - Label Bot

...

Approach

Image Added

Here's a proposed approach to do this : 

1. Create simple models on earlier versions of MXNet, initialize them with randomly generated weights, perform a forward pass on them. Save the model and model parameters and upload them in an S3 bucket. 
2. As a continuation of previous step, perform a simple inference on randomly generated input and save the randomly generated input as well as the inference output along with the model files on S3. 
3. The inference script running on the latest master branch of MXNet repository, would pull the model files + data and would try to load the models back into memory. The tests would fail if the models fail to load into memory or they give a different inference output. The different inference output could indicate or flag a potential change in an underlying operator. 
4. Use the same seed values to ensure we have the same environment for both training and inference files. 
5. These tests could be a part of nightly tests and would help in flagging out the above mentioned issues.
6. Primarily the model backwards compatibility checker would cover the following APIs to save/load models : 
   1. Declarative Models load_checkpoint() from Model API
   2. Gluon Models load_parameters/save_parameters API from Gluon Package
   3. Gluon Models load_params/save_params API from Gluon Package
   4. TheHybridized models import/export API from Gluon Package

4. Current work

A first cut of model backwards compatibility using the above approach has been implemented here : https://github.com/apache/incubator-mxnet/pull/11626. We would want to make it more robust and would like to get feedback on the ways in which we can improve this further.

Image Removed

Figure 3 Label Bot Design

Sample Issue Report

...

• Part III Determine labels automatically
  Each instance can be assigned with multiple categories, so these types of problems are known as multi-label classification problem, where we have a set of target labels. Multi-label classification problems are very common in the real world, for example, audio categorization, image categorization, bioinformatics..etc. Our project mainly focus on text categorizations because labels are learned from issue title and issue description.

Steps to achieve it

...

• Bag-of-word model uses all words in a document as the features, and thus the dimension of the feature space is equal to the number of different words in all of the documents.
• Binary, in which the feature weight is either one - if the corresponding word is present in the document - or zero otherwise.
• TF-IDF scheme gives the word w in the document d the weight
  TF-IDF Weight(w, d) = TermFreq(w,d) * log(N/DocFreq(w))
• Word2Vec, a two-layer neural net that processes text.
• Doc2Vec, an extension of Word2Vec that learns to correlate labels and words.

Step 4: Feature Extraction
Map original high-dimensional data onto a lower-demensional space. Remove non-informative terms (irrelevant words) from documents.Improve classification effectiveness and reduce computational complexity.
Feature selection methods:

...

Step 5: Multi-Label Classification
Use two different approaches for multi-label classification. Problem transformation methods try to transform the multi-label classification into single-label or multi-class classification problems. Algorithm adaptation methods adapt multi-label algorithms so they can be applied directly to the problem. Pick top 10 labels to do classification at the beginning.

• Problem Transformation
• Binary Relevance
  This is the simplest technique, which basically treats each label as a separate single class classification problem.
• Classifier Chains
  The first classifier is trained just on the input data and then each next classifier is trained on the input space and all the previous classifiers in the chain.
• Label Powerset
  Transform the problem into a multi-class problem with one multi-class classifier is trained on all unique label combinations found in the training data.
• Algorithm adaptation
  Manual: rule-based
  Automatic:
• Vector space model based
• Prototype-based
• K-nearest neighbor
• Decision-tree
• Neural Networks
• Support Vector Machines
• Probabilistic or generative model based
• Naive Bayes classifier

4. Technical Challenges

• Restrict permissions of this bot to avoid unexpected operations.
• Training data is limited.

5. Reference

...