...
class | home-banner |
---|
...
With this project, we want to build Java APIs that a new set of APIs which are Java friendly, compatible with Java 7+, are easy to use for Inference inference, and lowers the entry barrier is of consuming MXNet for production use-cases.
...
As a user, I’d like to have a Java Inference API that allows me to use deep learning models from my existing Java application.
As a user, I'd like for the new Java Inference API to be thread safe.
As a user, I’d like for the new Java inference API to be idiomatic and easy to use so that I can quickly learn to deploy models.
...
As a user already familiar with MXNet, I’d like for the new API to be similar to existing implementations so that it’s easy for me to use.
Proposed Approach
As a user, I'd like to have examples and tutorials available to help learn how to use the new Java Inference API.
Proposed Approach
The proposed implementation for the new Java API is to create a Java friendly wrapper around the existing Scala API. The Scala API is already fully implemented and is undergoing significant improvements (most notably simplifying the memory management of off-heap memory). By utilizing the existing Scala API, the development effort require for the new Java API is greatly decreased. Additionally, the Java API would automatically (or with minimal work) benefit from new features and code improvements allowing for development efforts to remain focused. This is a very similar approach to how Apache Spark developed their Java API.
Since both Java and Scala are JVM languages, it is already possible for the Scala bindings to be called from Java code by loading the jar into the classpath. Due to differences in the languages, this process is currently very painful for users to implement. To improve upon this experience, a Java wrapper would Most notably, the difficulty comes from the liberal use of default values in the Scala code being unsupported by Java and converting between Java/Scala collections. To improve upon this experience, a Java wrapper would be created which will call the Scala bindings. The wrapper would be designed so that it abstracts away the complexities of the Java/Scala interaction and is by automating the conversions, simplifying the method calls, and making the API more idiomatic for the Java inferencing use case.
- Advantages
- Fastest time to market requiring the least amount of engineering effort.
- Interaction with the native code Interaction with the native code is already done.
- The Scala API is already designed and decided. Implementing a wrapper limits design decisions which needs to be made and keeps the APIs consistent.
- Allows for development continue to be focused on a single JVM implementation which can be utilized by other JVM languages.
- The implementation, adding new features, maintenance would be greatly simplified.
- Implementation is not one way. In the future we maintain the ability to walk this decision back and go with another implementation.
- Disadvantages
- Interaction with the Scala code could be complicated due to differences in the languages. Known issues are forming Scala collections and leveraging default values. Changes to the Scala API or the introduction of a builder function seems like the most obvious solutions here.
- Some overhead in converting collections should be expected.
- The JAR files will be larger than they would be without Scala in the middle. Theoretically, this could be an issue for some memory constrained edge devices.
Known Difficulties
Converting Java collections into Scala collections - Scala and Java use different collections. Generally, these can be converted through the scala.collection.JavaConverters library. Ideally, this will be done automatically on behalf of the user. The Java methods should take Java collections, do the necessary conversion, then call the corresponding Scala method.
Java doesn’t support methods with default arguments - The current Scala implementation makes liberal use of default arguments. For class instantiation, a simple builder pattern will work. Class methods with default values will likely need to be overloaded.
Limited by existing Scala Inference API - The current Scala Inference API is lacking support for some models such as RNNs. Since this API will be utilized by the new Java Inference API, it will be necessary to improve and expand the Scala Inference API. This work can be done in parallel and should undergo it’s own design process.
Performance
Performance should be very similar to Scala. Since both are JVM languages doing inference will be calling the same byte code from Java as it is in Scala. The only known issue which will cause a performance difference is converting the Java collections into Scala collections. Preliminary testing with simple models shows negligible to nonexistent impact to performance. Java performance should be measured via Benchmark AI in a manner similar to how it's measured in Scala. More details on Scala benchmarks are available here.
Preliminary comparisons are a WIP.
Distribution
The new Java inference API can be distributed alongside the existing Scala API. Currently, the Scala API is distributed via a jar file using a Maven repository. There is ongoing work to automate this process and ideally this work will include the new Java API as well. The design for the Automated Scala Release is available here.
Class Diagram
...
Planned Release Milestones
Milestone 1: Initial release with support for all existing Scala Inference APIs. Includes integration into the existing CI, working examples, tutorials, documentation, benchmarking, and integrations into Maven distribution pipeline.
Milestone 2: General improvements to Inference API, improved better support specific use cases, and add sparse support (required for RNNs).
Milestone 3+: ?? (Ideas include: auto grad, exposing module api, control flow support)
Known Difficulties
Converting Java collections into Scala collections - Scala and Java use different collections. Generally, these can be converted through the scala.collection.JavaConverters library. Ideally, this will be done automatically on behalf of the user. The Java methods should take Java collections, do the necessary conversion, then call the corresponding Scala method.
Java doesn’t support methods with default arguments - The current Scala implementation makes liberal use of default arguments. For class instantiation, a simple builder pattern will work. Class methods with default values will likely need to be overloaded.
Limited by existing Scala Inference API - The current Scala Inference API is lacking support for some models such as RNNs. Since this API will be utilized by the new Java Inference API, it will be necessary to improve and expand the Scala Inference API. This work can be done in parallel and should undergo it’s own design process. On the plus side this will serve as a forcing function to improve the Scala API.
Performance
Performance should be very similar to Scala. Since both are JVM languages doing inference will be calling the same byte code from Java as it is in Scala. The only known issue which will cause a performance difference is converting the Java collections into Scala collections. Preliminary testing with simple models shows negligible to nonexistent impact to performance. Java performance should be measured via Benchmark Scripts in a manner similar to how it's measured in Scala. More details on Scala benchmarks are available here.
Preliminary comparison results are a WIP and will be added soon.
Distribution
The new Java inference API can be distributed alongside the existing Scala API. Currently, the Scala API is distributed via a jar file using a Maven repository. There is ongoing work to automate this process and ideally this work will include the new Java API as well. The design for the Automated Scala Release is available here. Releases for the Java Inference API will be aligned with the MXNet release schedule and follow the same versioning.
Improving Scala Inference API
The existing Scala Inference API will need to be expanded and improved. These changes will need to undergo their own design process and can easily be incorporated into the new Java API. Although these improvements are not a requirement to begin working on the the Java API, ideally it will be done in parallel so that the Java API will be more useful upon release.
Known improvements to could made to the Scala API include:
- Support for RNNs
- Adding domain specific use cases
- Improving interface of existing APIs (for example, it should be possible to do batch inference using just an NDArray)
Existing Scala Infer API Class Diagram
draw.io Diagram | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Sequence Diagram
draw.io Diagram | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Java Inference API Design for Predictor Class
The Java Inference API will be a wrapper around the high level Scala Inference interface. Here is an example of what the Java wrapper will look like for the Scala inference Predictor class.
Code Block | ||||
---|---|---|---|---|
| ||||
/**
* Implementation of prediction routines.
*
* @param modelPathPrefix Path prefix from where to load the model artifacts.
* |
Sequence Diagram
draw.io Diagram | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Java Inference API Design
The Java Inference API will be a wrapper around the high level Scala Inference interface. Here is an example of what the Java wrapper will look like for the Scala inference Predictor class.
Code Block | ||||
---|---|---|---|---|
| ||||
/** * Implementation of prediction routines. * * @param modelPathPrefix Path prefix from where to load the model artifacts. * These include the symbol, parameters, and synset.txt * Example: file://model-dir/resnet-152 (containing * These resnet-152-symbol.json, resnet-152-0000.paramsinclude the symbol, parameters, and synset.txt). * @param inputDescriptors Descriptors defining the input node names, shape, * Example: file://model-dir/resnet-152 (containing * layout and type parameters * resnet-152-symbol.json, resnet-152-0000.params, and synset.txt). * @param inputDescriptors Descriptors <p>Note: Ifdefining the input Descriptorsnode is missing batchSizenames, shape, * ('N' in layout), a batchSize of 1 is assumedlayout forand thetype model.parameters * @param contexts Device contexts on which you want to run inference; defaults to CPU <p>Note: *If @paramthe epochinput Descriptors is missing batchSize * Model epoch to load; defaults to 0 */ Predictor(String modelPathPrefix, Array<DataDesc> inputDescriptors, ('N' in layout), a batchSize of 1 is Array[Context] Contexts, int epoch) | ||||
Code Block | ||||
| ||||
/**assumed for the model. * Takes input as IndexedSeq one dimensional arrays and creates the NDArray needed for inference * The array will be reshaped based on the input descriptors. * * @param input: @param contexts Device contexts on which you want to run inference; defaults to CPU * @param epoch Model epoch to load; defaults to 0 */ Predictor(String modelPathPrefix, List<DataDesc> inputDescriptors, An Array of a one-dimensional array. List<Context> Contexts, int epoch) /** * Predict using NDArray as input * This method is useful when the input is a batch of data * AnNote: ArrayUser is neededresponsible whenfor themanaging modelallocation/deallocation hasof more than one input.input/output NDArrays. * * @return@param inputBatch List of NDArrays * @return Indexed sequence array of outputs */ Array <Array <Float>> predict(Array <Array <Float>> input) | ||||
Code Block | ||||
| ||||
/** * PredictOutput usingof NDArraypredictions as inputNDArrays * This method is useful when the input is a batch of data * Note: User is responsible for managing allocation/deallocation of input/output NDArrays. * * @param inputBatch Array of NDArrays * @return Output of predictions as NDArrays */ Array <NDArray> predictWithNDArray(Array <NDArray> inputBatch) |
Example Uses
/
List <NDArray> predictWithNDArray(List <NDArray> inputBatch) |
Java Inference API usage
A primary goal of the Java Inference API is to provide a simple means for Java users to load and do inference on an existing model. Ideally, this will typically be as simple as defining the context (cpu vs gpu) to be used, defining what the input will look like, and setting up the model that will be used. After setting up the model like this, it should be simple to do input on the model.
Code Block | ||||
---|---|---|---|---|
| ||||
/*
* Psudeocode for how ObjectDetector Class can be used to do SSD detection
* A full working SSD example will be included in the release.
*/
// Set the context to be used
List<Context> context = new ArrayList<Context>();
context.add(Context.cpu());
// Define the shape and data type of the input
Shape inputShape = new Shape(new int[] {1, 3, 512, 512});
List<DataDesc> inputDescriptors = new ArrayList<DataDesc>();
inputDescriptors.add(new DataDesc("data", inputShape, DType.Float32(), "NCHW"));
// Instantiate the object detector with the model, input descriptors, context, and epoch
JavaObjectDetector objDetector = new JavaObjectDetector(modelPathPrefix, inputDescriptors, context, 0);
// Load an image and run inference on it
BufferedImage img = JavaImageClassifier.loadImageFromFile(inputImagePath);
objDetector.imageObjectDetect(img, 3); | ||||
Code Block | ||||
| ||||
List<float> doSinglePrediction(){
MXNetInferenceModel model = new MXNetInferenceModel();
model.load("/local/path/to/model");
List<float> singlePrediction = Arrays.asList(3.14159, .5772, 2.71828);
model.predict(singlePrediction);
return model.getPredictions().get(0);
}
List<List<float>> doBatchPrediction(){
MXNetInferenceModel model = new MXNetInferenceModel();
model.load("/local/path/to/model");
List<List<float>> batchPrediction = Arrays.asList(
Arrays.asList(3.14159, .5772, 2.71828),
Arrays.asList(.5772, 2.71828, 3.14159),
Arrays.asList(2.71828, 3.14159, .5772));
model.predict(batchPrediction);
return model.getPredictions();
} |
Open Questions
How to deal with Option[T] field in Java when calling from Scala?
...