Page History

...

With this project, we want to build Java APIs that a new set of APIs which are Java friendly, compatible with Java 8+, are easy to use for Inference inference, and lowers the entry barrier is of consuming MXNet for production use-cases.

...

The proposed implementation for the new Java API is to create a Java friendly wrapper around the existing Scala API. The Scala API is already fully implemented and is undergoing significant improvements (most notably simplifying the memory management of off-heap memory). By utilizing the existing Scala API, the development effort require for the new Java API is greatly decreased. Additionally, the Java API would automatically (or with minimal work) benefit from new features and code improvements allowing for development efforts to remain focused. This is a very similar approach to how Apache Spark developed their Java API.

Since both Java and Scala are JVM languages, it is already possible for the Scala bindings to be called from Java code by loading the jar into the classpath. Due to differences in the languages, this process is currently very painful for users to implement. To improve upon this experience, a Java wrapper would be created which will call the Scala bindings. The wrapper would be designed so that it abstracts away the complexities of the Java/Scala interaction and is more idiomatic for the Java inferencing use case.

...

Performance should be very similar to Scala. Since both are JVM languages doing inference will be calling the same byte code from Java as it is in Scala. The only known issue which will cause a performance difference is converting the Java collections into Scala collections. Preliminary testing with simple models shows negligible to nonexistent impact to performance. Java performance should be measured via Benchmark Scripts in a manner similar to how it's measured in Scala. More details on Scala benchmarks are available here.

Preliminary comparisons comparison results are a WIP and will be added soon.

Distribution

The new Java inference API can be distributed alongside the existing Scala API. Currently, the Scala API is distributed via a jar file using a Maven repository. There is ongoing work to automate this process and ideally this work will include the new Java API as well. The design for the Automated Scala Release is available here.

Improving Scala Inference API

The existing Scala Inference API will need to be expanded and improved. These changes will need to undergo their own design process and can easily be incorporated into the new Java API. Although these improvements are not a requirement to begin working on the the Java API, ideally it will be done in parallel so that the Java API will be more useful upon release.

Known improvements to could made to the Scala API include:

• Support for RNNs
• Adding domain specific use cases
• Improving interface of existing APIs (for example, it should be possible to do batch inference using just an NDArray)

Class Diagram

...

/**
 * Implementation of prediction routines.
 *
 * @param modelPathPrefix     Path prefix from where to load the model artifacts.
 *                            These include the symbol, parameters, and synset.txt
 *                            Example: file://model-dir/resnet-152 (containing
 *                            resnet-152-symbol.json, resnet-152-0000.params, and synset.txt).
 * @param inputDescriptors    Descriptors defining the input node names, shape,
 *                            layout and type parameters
 *                            <p>Note: If the input Descriptors is missing batchSize
 *                            ('N' in layout), a batchSize of 1 is assumed for the model.
 * @param contexts            Device contexts on which you want to run inference; defaults to CPU
 * @param epoch               Model epoch to load; defaults to 0

 */
Predictor(String modelPathPrefix, List<DataDesc> inputDescriptors,
                List<Context> Contexts, int epoch)

/**
 * Takes input as IndexedSeq one dimensional arrays and creates the NDArray needed for inference
 * The array will be reshaped based on the input descriptors.
 *
 * @param input:            A List of a one-dimensional array.
                            A List is needed when the model has more than one input.
 * @return                  Indexed sequence array of outputs
 */
List <List <Float>> predict(List <List <Float>> input)

/**
 * Predict using NDArray as input
 * This method is useful when the input is a batch of data
 * Note: User is responsible for managing allocation/deallocation of input/output NDArrays.
 *
 * @param inputBatch        List of NDArrays
 * @return                  Output of predictions as NDArrays
 */
List <NDArray> predictWithNDArray(List <NDArray> inputBatch)

...

// Rewriting the example. Will add on Monday  

Open Questions

How to deal with Option[T] field in Java when calling from Scala?

...