Versions Compared

Old Version 4

changes.mady.by.user Sandeep Krishnamurthy

Saved on

compared with

New Version 5

changes.mady.by.user Sandeep Krishnamurthy

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Problem

Taking a MXNet model from training and deploying it in production poses several challenges to the users. Most important problems raised by users are:

...

Code Block
{
    "nodes": [..........]
    "arg_nodes": [0, 1, 2, 4, 5],
    "node_row_ptr": [0, 1, 2, 3, 4, 5, 6, 7, 8],
    "heads": [[7, 0, 0]],
    "attrs": {"mxnet_version": ["int", 10301]
              "inputs": {"data":[1,3,224,224]},
              "outputs" : {"softmax_label":[1,10]
              }
}

...

Possible use cases

Inputs and outputs are dictionary where key => Name of the node and value => shape. Though single input, single output models are more common, a model can be of following variations: 

  1. Single input, single output
  2. Single input, multiple outputs
  3. Multiple input with same or different transformation, single output
  4. Multiple input with same or different transformation, multiple output

Export API - Gluon

  1. Since this is an update to the most commonly used API - Export in Gluon, we will first introduce the change with a new Export API in gluon.contrib.

Before

Code Block
languagepy
net.export(path="./my_model", epoch=0)

After

User Experience

Before

Step 1 - Train and export the model from Gluon

Code Block
languagepy
# Trained network
net = mx.gluon.model_zoo.vision.resnet18_v1(pretrained=True, ctx=mx.cpu())

# Data transformations applicable during inference
inference_input_transforms = gluon.nn.HybridSequential()
inference_input_transforms.add(transforms.Resize((224, 224)))
inference_input_transforms.add(transforms.ToTensor())
inference_input_transforms.add(transforms.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)))


# Export the model. Cannot export data transformation and input/output signature
net.export(path="./my_model", epoch=0)

Step 2 - Import the model for inference with Module OR Gluon OR JAVA

Import the model in Gluon

Code Block
languagepy
# Load the model. Model does not contain transformation and input/output signature 
net = SymbolBlock.imports(symbol_file="my_model-symbol.json",
                          input_names=["data"],
                          param_file="my_model-0000.params",
                          ctx=mx.cpu())

Import the model in Module API

Code Block
languagepy
sym, arg_params, aux_params = mx.model.load_checkpoint('my_model', 0)
mod = mx.mod.Module(symbol=sym, context=ctx, label_names=None)
mod.bind(for_training=False, data_shapes=[('data', (1,3,224,224))], 
         label_shapes=mod._label_shapes)
mod.set_params(arg_params, aux_params, allow_missing=True)

mod.forward(...)

After

Step 1 - Train and export the model from Gluon

Code Block
languagepy
# Trained network
net = mx.gluon.model_zoo.vision.resnet18_v1(pretrained=True, ctx=mx.cpu())

# Data transformations applicable during inference
inference_input_transforms = gluon.nn.HybridSequential()
inference_input_transforms.add(transforms.Resize((224, 224)))
inference_input_transforms.add(transforms.ToTensor())
inference_input_transforms.add(transforms.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)))


# Export the model
Code Block
languagepy
## net => Hybrid Sequential Network
## input_transforms => Hybrid Sequential Transformation Pipeline
gluon.contrib.utils.export(net, path="./my_model", 
                           epoch=0, 
                           signature={constants.INPUT_DESC:[("data", (1,3,224,224))],
                                      constants.OUTPUT_DESC:[("softmax_label", (1,10))]},
                           input_transforms={"data":inference_input_transforms},
                           output_transforms=None)

Import API - Gluon

...

Step 2 - Import the model for inference with Module OR Gluon OR JAVA

Import the Model in Gluon

Code Block
languagepy
SymbolBlock.importsgluon.contrib.utils.import(symbol_file="my_model-symbol.json",
                    input_names=["data"]       param_file="my_model-0000.params",
                    param_file="my_model-0000.params",
       load_transforms = True,
                           ctx = 'cpu')

OR

Import the Model in Module API

(Supported to create a module for inference only)After

Code Block
languagepy
gluonmod = mx.contrib.utilsModule.import(
                symbol_file = "my_model-symbol.json",
                           param_file = "my_model-0000.params",
                load_transforms = True,
         load_transforms = True,
     ctx = 'cpu',
                    ctx = 'cpu')

Import API - Module

Before

batch_size = 1)
mod.forward(...)

OR

Import the Model in Java Predictor API

Code Block
languagepyjava
List<Context> context = new ArrayList<>();
context.add(Context.cpu()); 
String modelPathPrefix = "my_model";
Predictor predictor = new Predictor(modelPathPrefix, context, load_transforms=True);

List<NDArray> result = predictor.predictWithNDArray(inputNDArray);


Import API - Gluon

Before


Code Block
languagepy
SymbolBlock.imports(symbol_file="my_model-symbol.json",
                    input_names=["data"],
           sym, arg_params, aux_params = mx.model.load_checkpoint('my_model', 0)
mod = mx.mod.Module(symbol=sym, context=ctx, label_names=None)
mod.bind(for_training=False, data_shapes=[('data', (1,3,224,224))], 
         labelparam_shapes=mod._label_shapes)
mod.set_params(arg_params, aux_params, allow_missing=True)

mod.forward(...)file="my_model-0000.params",
                    ctx='cpu')


After


Code Block
languagepy
gluon.contrib.utils.import(symbol_file="my_model-symbol.json",
                           param_file="my_model-0000.params",
                           load_transforms = True,
                           ctx = 'cpu')


Import API - Module

Before


After
Supported to create a module for inference only.

...