THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
MXNet Edge team.
MXNet C++ Package
Introduction
For using MXNet for inference in a native production setting where speed and reliability is critical, a solid C++ Package offering is necessary.
A C++ package allows to integrate MXNet into native applications. Also avoids runtime errors which are common with dynamically typed languages such as Python. There are also cases where the extra loss in performance by using Python can't be afforded. In addition, using MXNet from C++ allows for more complex setups where multithreading might be needed or reduced data copying.
Background
Currently, there is a C++ Package in MXNet under the folder `cpp-package/`. This package wraps the MXNet C API and re-implements some classes on top of it, such as NDArray or Symbol. Additionally some classes or enums need duplication, in attempt to hide the implementation details of MxNet. Right now the example C++ applications are intermingled in the MXNet source folder and is nontrivial to have an external application compiled against MXNet.
For example, the main part of the library - the operators exposure need and special python script to be executed to run through the compiled MxNet binary file and the library sources to generate C++ API code. This causes significant maintenance effort to synchronize the classes and usages throughout the layers. It does not allow semantic and explicit versioning and is likely to break when implementation details will change. Though the attempt was made to hide the implementation details of the library and as such provide a proper public API of the library by duplicating classes, the package is still not standalone and decoupled from the MXNet source, as it requires additional includes that are "private" to the MXNet implementation such as dmlc and nnvm.
Our primary objective would be to provide a standalone package with a CMake build file, so that users can build a C++ application against an MXNet binary library. This includes a semantic, fixed, documented API with binary compatibility throughout minor version changes. Documentation would provide examples that have features marked from which versions of the library they are supported.
Another important goal is to separate inference and training API's, since customers want only effective, as lightweight as possible inference on their production edge devices. Training should also be considered, although with a lower priority.
From a user perspective, ideally they would get a package compiled for their distribution / OS and a set of header files to program and link against the MXNet library.
Goals
The proposed work in the C++ package aims to achieve the following improvements, areas of focus for users of the C++ package:
Usability
- Type safety
- Informative errors
- Documentation and versioning of the API
Exposed functions should be documented with doxygen or similar and added to the public documentation on the website - Inference / training separation of the API
With inference prioritization for now. - Examples
Example of using the C++ API, there already exist several in the cpp-package. - Standalone Package
An example standalone MXNet package with build file that produces a native application that does inference should be the main artefact of this activity.
C++ Development experience
- API / ABI compatiblity.
- Portable binary library with standalone headers
- Build configuration
Work areas
Remove intermediate C API layer for C++
We propose hooking directly into the C++ API and not going through the C API. This helps for type safety and maintainability. Also can be beneficial to propagate errors from MXNet to the C++ API via normal C++ exceptions without additional machinery, overhead and conversion to error codes in the C API boundary.
Make implementation private
We propose improving key core classes in MXNet such as NDArray, Symbols or Operators. Exposing their API via a "private pointer implementation (PIMPL)" which has the effect of providing a stable API to users and hiding the implementations to separate private classes.
This would improve significantly the recompilation time while development. It also provides binary compatibility, so the library can be updated by dependant applications without their need to be recompiled. While the latter is not a strong requirement for our users right now due to the experimental nature of our framework, it's commonplace in the software industry for commercial and open source libraries, where binary compatibility is required or nice to have.
Another important advantage is that it would make possible to separate the implementations based on different frameworks (such as computing with MKL) and platform dependent implementations, which would lead to a better architecture with less coupling. Currently most of the implementations reside in common header files, separated with preprocessor defines making it hard to develop and to maintain.
Read more at: "Pointer to implementation" or "pImpl" is a C++ programming technique
Semantic versioned API with no additional dependencies
Ideally no additional headers such as nnvm or dmlc would need to be added. We observed that libdmlc needs to be linked as well. And the build system has to be configured to handle cases such as when compiling with omp in the 3rd party folder with creates an additional binary artefact that is needed for linking.
Future work and considerations
With the library being binary compatible with minor version changes language bindings can be moved to use C++ API only and the there will be no need for maintaining a separate C interface. The bindings could be separated from the main project and have it's own versioning.
Additional care must be taken with other MXNet dependencies if there would be a need to expose their API's as part of the public interface (OpenCV, OpenBLAS, MKL etc.) or if they will be distributed as separate libraries to link against. This needs to be investigated further and additional steps to take in these cases must be defined and documented.