Label Prediction by the @mxnet-label-bot

Description:

Currently, within the incubator-mxnet repo there are over 800+ issues and new ones being generated every day. The goal is to ease this process and handle developer's issues in an appropriate manner. With the use of labelling, experts in their respective areas can provide the help that users face. We employ the label bot today to help ease the issue/pull request process. Given the data which the repository provides of issues and pull requests which have been previously labelled, an interesting use case of this data opens up. Based upon the data of this repository, we are able to provide insights and predictions of labels on new issues and pull requests. This mechanism will provide a better experience for those who have raised an issue to get a faster response, and it allows for existing and new contributors to better filter for their areas of expertise who are wanting to help out welcoming new developers.

Proposal:

This prediction service offered by the label bot can be useful to the community and in for its implementation, the label bot can either auto label certain issue or pull requests or provide a recommendation by commenting on the appropriating threads (issues and pull requests). On the prediction, we 

Data Analysis:

Note: Training data here is limited (~13,000 issues both closed and opened), after the data cleaning process we expect this value to be greatly further reduced.

Metrics:

Multi-label Classification:
Accurate prediction of at least one label in an issue: ~87%
Accuracy in predicting all labels in an issue (i.e. an exact match of all labels to an issue): ~20%

These were labels chosen for prediction initially by the model. Only the issues which are specific to these labels are what is being tested on.

Results in accurately predicting a label – Meaning the model predicted a label and that was one of the actual labels in the repo:

Classification Accuracy:

https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score

Full classification report with precision, recall, and f1 score:

Data Insights:

There may be some potential overfitting happening with the 'Performance' label in particular – will require further examination.