THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Code Block | ||
---|---|---|
| ||
List<Context> context = new ArrayList<>(); context.add(Context.cpu()); String modelPathPrefix = "my_model"; # Load the model along with the transformations. Predictor predictor = new Predictor(modelPathPrefix, context, load_transforms=True); # Inference List<NDArray> result = predictor.predictWithNDArray(inputNDArray); |
Performance Consideration
During inference, initial benchmarks shows a noticeable performance gain with End to end models i.e., a model with data transformations and neural network all fused as a single model graph.
- ResNet-18 model pre-trained with ImageNet. https://s3.us-east-2.amazonaws.com/mxnet-public/end_to_end_models
- Pre-processing - Resize(224, 224), ToTensor, Normalize(mean=(
0.485
,
0.456
,
0.406
), std=(0.229
,
0.224
,
0.225))
- We take average of 500 runs
- Single Request Inference - Input Data - Synthetic (random.uniform(0, 255, shape=(1, 300, 300, 3))
- Batch Inference - Input Data - Synthetic (random.uniform(0, 255, shape=(25, 300, 300, 3))
- Below time gives - Average Prediction Time Per Sample
A | B | C | Non End to End Models (ms) | End to End Models (ms) | Boost % |
---|---|---|---|---|---|
CPU (C5.2X) | Single Request Inference | Python (Module API) | 17 | 14 | 17.65% |
Java Inference APIs | 17.09 | 14.16 | 17.14% | ||
Scala Inference APIs | 17.93 | 13.19 | 26.44% | ||
Batch Inference (Batch size = 25) | Python (Module API) | 15.18 | 12.57 | 17.19% | |
Java Inference APIs | 18.54 | 13 | 29.88% | ||
Scala Inference APIs | 17 | 13.26 | 22.00% | ||
GPU (P3.16X) | Single Request Inference | Python (Module API) | 5.78 | 3.14 | 45.67% |
Java Inference APIs | 8.95 | 4.26 | 52.40% | ||
Scala Inference APIs | 9.14 | 4.42 | 51.64% | ||
Batch Inference (Batch size = 25) | Python (Module API) | 2.61 | 1.31 | 49.81% | |
Java Inference APIs | 8.03 | 5.53 | 31.13% | ||
Scala Inference APIs | 7.86 | 5.52 | 29.77% |
Backward compatibility
- All APIs changes are backward compatible.
- Old MXNet model should still load without breakage with new MXNet version.
- New MXNet model will not work on old MXNet versions. If a user tries to load new MXNet model with older MXNet version, they get errors such as - "unknown field 'inputs', 'outputs' in the model" because, MXNet's model JSON parser schema in old MXNet do not understand new fields introduced as part of this work.
...