...
Link to dev list discussion
Feature Shepherd
Lin Yuan (https://github.com/apeforest)
...
A deep learning framework like MXNet supports 100s of operators (~250). Benchmarking and profiling a standard neural network and use-case, like ResNet-50 based image classification, is not fully sufficient and does not guarantee to maintain the health and performance of all the supported operators under different settings (Hardware, Accelerator, Data etc...). We need to have an easy to use library/framework to benchmark utility to run benchmarks and profile each operator individually. Having an operator level benchmarks will help us in - fine grained understanding of performance of operators under different settings (Hardware, Accelerator, Data etc...), automated CI/CD performance tests, plan performance optimization tasks and more. In this document, we present a library an utility for MXNet operator benchmark and profiling.
...
A deep learning framework like MXNet supports 100s of operators (~250). Some operators are used as a layers layer in the neural network (ex: Conv2D), some operators work in combination to form a layer in the neural network (ex: dot, sum => Dense), and many more just used independently outside a neural network (ex: tensor creation/shape change/indexing, logical operators) mostly for data processing and tensor manipulation.
An operator is highly heterogeneous w.r.t supported precision (fp32, fp64, Int64 etc.), accelerators (mkldnn, cuda, cudnn, mxnet native only), different behaviors based on data (ex: broadcast sum behavior on a large square (1024, 1024) tensor is different than on a skewed tensor (10, 10000) and more). Below, we see few areas why we believe operator benchmarks are useful:
- Users use a lot more operators that are not part of a standard network like ResNet. Example: Tensor manipulation operators like mean, max, topk, argmax, sort etc.
- A standard Network Architecture like ResNet-50 is made up of many operators Ex: Convolution2D, Softmax, Dense, Pooling etc... Observing only the end to end performance can hide individual operator regressions for long time.
- We need to know on different hardware infrastructure (Ex: CPU with MKLDNN, GPU with NVIDIA CUDA and cuDNN) how different operators performsperform. With these details, we can plan the optimization work at operator level, which could exponentially boost up the end to end performance.
- Operator behavior varies based on different data load:
- For example, MXNet's reduction operations works work seamlessly with balanced tensor like (1024, 1024), however, performance behavior changes when the input tensor is skewed (1024, 10). Similar observations can be made when comparing Int32 v/s Int64 indexing of Tensor.
- See this issue - #14725 which talks about performance regression in FC layer backward pass with CUDA 10 based on input tensor shape - https://github.com/apache/incubator-mxnet/issues/14725#issuecomment-486016229
- You want to have nightly performance tests across all operators in a deep learning framework to catch regressions early.
- We can integrate this framework with a CI/CD system to run per operator performance tests for PRs. Ex: When a PR modifies the kernel of TransposeConv2D, we can run benchmarks of TransposeConv2D operator to verify performance.
Hence, in this utility, we will build the functionality to allow users and developers of deep learning frameworks to easily run benchmarks for individual operators across varying settings.
Requirements
- Useful insights can be derived to plan for operator performance improvements. Example - Argmax is much slower compared to max operator on a GPU. This is an area that we can work on to improve the performance of Argmax operator. https://github.com/apache/incubator-mxnet/issues/11337
- Benchmarking operator performance in MXNet comparing with other Deep Learning frameworks such as PyTorch. (Not in current scope)
Hence, in this utility, we will build the functionality to allow users and developers of deep learning frameworks to easily run benchmarks for individual operators across varying settings.
Requirements
- Benchmarks for Apache MXNet operators.
- Individual operator benchmarks to capture - time for operator execution (speed), memory usage.
- Fine grained individual operator
- Benchmarks for Apache MXNet operators.
- Common combination (Fused) of operators. Conv + Relu, Conv + BatchNorm.
- Individual operator benchmarks to capture - time for operator execution (speed), memory usage.
- Fine grained individual operator benchmarks to capture - time for forward pass, time for backward pass and both.
- Ability to run operator benchmarks with default inputs or , randomly generated inputs or customize with user specific inputs.
- Ability to run operator benchmarks on CPU/GPU with different flavors of MXNet (mxnet-mkl, mxnet-cu90mkl etc.)
- Benchmarks for operators with varying inputs to uncover any performance issues due to skewed input data. Ex: Measuring operator performance on small input tensors, large input tensors along with average normally used tensor sizes.
- Ability to run one, group or all operator benchmarks.
- Ability to extract results in multiple usable format - Python Dictionary, JSON, CSV, MD
- Statistics:
- Mean, Median, P50, P90, P99
- Reproducible tests
- Common combination (Fused) of operators. Conv + Relu, Conv + BatchNorm. (Not in current scope)
Design Tenets
- Defaults => Common use cases should be extremely easy, customized complex use cases should be possible.
- Example: I should be able to run Add operator benchmarks without specifying any inputs and library should provide benchmarks on valid default inputs. At the same time, as a power user, I should be able to provide my own inputs such as Tensor Shapes and context to run the benchmarks.
- Minimum Learning Curve => Keep APIs same or close to native NDArray / Gluon Operators being benchmarked.
- Example: If I am doing benchmarks on nd.add(lhs, rhs) operator, interface in the benchmark utility should be similar with zero learning curve.
- Modular and Reusable
- For a programmer or an automated system
- Example: Developer using the library or integration with CI/CD
Proposed Approach
...
- Provide a generic utility for executing an operator benchmarks and performance tests.
- This is responsible to creating input tensors of required shape on a given dtype, context.
- Execute the provided operator - forward or forward + backward.
- This generic utility will be integrated with MXNet profiler.
- Captures the profile output from MXNet profiler - time, memory.
- Return a dictionary of results.
- Input for the performance tests will be a key/value config.
Below is an example of performance runs for operators. It uses a base utility `run_performance_test`
...
.
Code Block | ||
---|---|---|
| ||
class MXNetOperatorBenchmarkBase(ABC): """Abstract Base class for all MXNet operator benchmarks. """ MXNet operator performance benchmarks. NOTE: 1. You can pass list of input dictionary to run benchmarks for an operator with different input configuration. 2. Results are dictionary of time, memory for the benchmark runs. """ # Run performance def __init__(selftest for Add operator results = run_performance_test(F=mx.nd.add, ctx=mx.cpu(), warmup=10, runs=1050, default_parametersinputs={}, custom_parameters=None):[{"lhs": (1024, 1024), self.ctx = ctx self.runs = runs self.warmup = warmup self.results = {} "rhs": (1024, 1024), self.inputs = prepare_input_parameters(caller=self.__class__.__name__, "initializer": nd.normal, default_parameters=default_parameters, "run_backward": True, custom_parameters=custom_parameters) @abstractmethod def run_benchmark(self): pass def print_benchmark_results(self): if not len(self.results):"dtype": "float32"}]) # Run performance test for Conv2D operator results += run_performance_test(F=nn.gluon.Conv2D, ctx=mx.cpu(), warmup=10, runs=50, inputs = [{"data": (32, 3, 256, 256), print("No benchmark results found. Run the benchmark before printing results!") return "data_initializer": nd.normal, for key, val in self.results.items(): print("{} - {:.6f} seconds".format(key, val)) "channels": 64, def get_benchmark_results(self): return self.results class Add(MXNetOperatorBenchmarkBase): """Helps to Benchmark Tensor Add operation. By default benchmark both forward and backward element_wise tensor addition "kernel_size": (3, 3), of 1024*1024 tensor of precision - 'float32'. """ def __init__(self, ctx=mx.cpu(), warmup=10, runs=50, inputs=None): # Set the default Inputs default_parameters = {"lhs "strides": (10241, 10241), "rhspadding": (10240, 10240), "initializer": nd.normal "dilation": (1, 1), "run_backwardlayout": True"NCHW", "dtypeactivation": "float32"} None, super().__init__(ctx=ctx, warmup=warmup, runs=runs, default_parameters=default_parameters, "run_backward": True, custom_parameters=inputs) self.lhs = get_mx_ndarray(ctx=self.ctx, in_tensor=self.inputs["lhs"], "dtype": "float32"]} |
How does the backend profiling utility code looks like?
Below we take an example of profiling Add operator.
Code Block | ||
---|---|---|
| ||
# Configurations warmup = 25 runs = 50 run_backward = True # Operator to benchmark F = mx.nd.add # Prepare data for the operator lhs dtype=self.inputs["dtype"], = mx.nd.ones(shape=(1024, 1024)) rhs = mx.nd.ones(shape=(1024, 1024)) lhs.attach_grad() rhs.attach_grad() mx.nd.waitall() # Warmup print("Warming up....") for _ in range(warmup): with mx.autograd.record(): res initializer=self.inputs["initializer"],= mx.nd.add(lhs, rhs) res.backward() mx.nd.waitall() print("Done warming up....") # Run Performance Runs print("Running performance runs....") profiler.set_config(profile_all=True, aggregate_stats=True) # Start Profiler profiler.set_state('run') for _ in range(runs): attach_grad=self.inputs["run_backward"])with mx.autograd.record(): self.rhsres = get_mx_ndarray(ctx=self.ctx, in_tensor=self.inputs["rhs"],mx.nd.add(lhs1, rhs1) res.backward() mx.nd.waitall() # Stop dtype=self.inputs["dtype"], Profiler profiler.set_state('stop') # Fetch Results from Profiler # We will add a new API in Profiler - profiler.get_summary(reset=True) # profiler.get_summary() => Return a JSON string representing the output as shown below. # initializer=self.inputs["initializer"], attach_grad=self.inputs["run_backward"]) def run_benchmark(self): # Warm up, ignore execution time value _, _ = nd_forward_backward_and_time(F=nd.add, runs=self.warmup, lhs=self.lhs, rhs=self.rhs) # Run Benchmarks exe_time, _ = nd_forward_backward_and_time(F=nd.add, runs=self.runs, lhs=self.lhs, rhs=self.rhs) self.results["MX_Add_Forward_Backward_Time"] = exe_time / self.runs |
Addition of new Module
...
=> Resets all the counter in the current profiler.
print("Done Running performance runs....")
print(profiler.dumps(reset=True))
|
Pros
- No need to write 1 class per operator to set up a performance test. Whenever a new operator is created, developer needs to add a `run_performance_test(..)` line with a list of inputs to run performance tests. A generic utility will handle the execution.
- Less code, easy to maintain.
- More control for users - default inputs, random inputs, specific user defined inputs.
- Deterministic and better suited for performance benchmarks, reproducibility and CI integration.
- More accurate benchmark results - Time and Memory because we use MXNet profiler.
- With Python interface:
- Easy to maintain and develop.
- Reflects the performance as seen by the users. (Majority users using Python interface)
- Fastest way to get performance tests in place. We do not have any tests in place as of today.
Cons
- Different operator will have different input names. For example, see above, add operator requires tensors with name lhs, rhs. However, Conv2D operator requires a tensor with data. The base performance executor utility will need to understand it and create tensors appropriately i.e., If it is one single executor, generalization across operator performance may make logic complex to manage.
- Not easily extensible:
- Hard to integrated with property based testing libraries like Hypothesis, to randomly generate test cases with different tensor shapes.
Addition of new Module
We propose to add this utility as a new module (opperf) under incubator-mxnet/benchmark as "incubator-mxnet/benchmark/opperf". Note that, this does not generate any user facing APIs, this is a utility under incubator-mxnet/benchmark folder for general use by community.
Addition of new API
We propose to add a new API to MXNet Profiler for easily fetching operator profile for processing programmatically.
1) mxnet.profiler.get_summary(reset=False)
Current Behavior:
Users can either use `mxnet.profiler.dump()` to output the profiler as a JSON file. Or, use `mxnet.profiler.dumps(reset=False)` API to print the summary on console.
Suggested Addition:
In order to enable easy programmatic usage of MXNet profiler output, we propose to introduce a new API to return the summary as JSON string. This enables users to run profiler, get summary output, perform analysis programmatically.
Code Block | ||
---|---|---|
| ||
mxnet.profiler.get_summary(reset=False)
"""Gets current profiler summary as a JSON string. If reset is True, resets all the aggregate statistics collected up to this point i.e., it clears all the profiler counters.
Parameters:
-----------
reset: boolean, If True, resets all profiler statistics collected up to this point.
""" |
Output:
We can visualize the output of this API as a JSON representation of the output from `mxnet.profiler.dumps(reset=False)` API as shown below.
However, please note that, below, Memory profile output is not the total bytes allocated. Current output from dumps is providing number of memory allocation calls made.
In the new suggested API, we will be adding additional Summary - Memory => Total Bytes Allocated (Per Device).
API / User Experience
We can define 2 types of users of the library and describe API interface for each of these users.
- General User, Automated Nightly tests
- Run benchmarks on all the operators or on specific categories of operators. Use default inputs provided by the library.
- Power User, PR validation tests
- Run benchmark with customized Inputs
Use Case 1 - Run benchmarks for all the operators
A driver to run all the MXNet operators (NDArray and Gluon) benchmarks with default inputs and saves the final result as JSON in the provided file.
Code Block |
---|
python incubator-mxnet/benchmark/opperf/run_all_mxnet_operator_benchmarks.py --output-format json --output-file mxnet_operator_benchmark_results.json |
Other Driver Script CLI Options:
- output-format :
json
ormd
for markdown file output orcsv
. - ctx : By default,
cpu
on CPU machine,gpu(0)
on GPU machine. You can override and set the global context for all operator benchmarks. Example:--ctx gpu(2)
. - dtype : By default,
float32
. You can override and set the global dtype for all operator benchmarks. Example:--dtype float64
.
Output for the above benchmark run, on a CPU machine, would look something like below:
Code Block |
---|
{
"MX_Multiply_Forward_Backward_Time": 0.025911798477172853,
"MX_Gluon_Imperative_RNN_Forward_Backward_Time": 0.011011338233947754,
"MX_Gluon_Imperative_MaxPool2D_Forward_Backward_Time": 0.1580966854095459,
"MX_Gluon_Imperative_Conv1D_Forward_Backward_Time": 0.03413449287414551,
"MX_Ones_Forward_Time": 0.002405076026916504,
"MX_Modulo_Forward_Backward_Time": 0.049943366050720216,
"MX_Subtract_Forward_Backward_Time": 0.01635995864868164,
"MX_ArgMin_Forward_Backward_Time": 0.01545732021331787,
"MX_Logical_Xor_Forward_Backward_Time": 0.018084139823913575,
"MX_Zeros_Like_Forward_Time": 0.0027973604202270507,
"MX_Inplace_Multiply_Forward_Time": 0.005555639266967774,
"MX_ArgSort_Forward_Time": 0.13972537994384765,
"MX_Arange_Forward_Time": 0.00010946273803710938,
........
........
} |
Use Case 2 - Power user - Run benchmarks for specific operator
As a power user, let us assume, you want to run benchmarks on Add operator with on a float64
tensor instead of a default float32
.
NOTE: Similarly, you could also specify the input tensors to use for benchmarking.
Use Case 2.1 - Customize Inputs for Operators
Code Block |
---|
results = run_performance_test(F=mx.nd.add, ctx=mx.cpu(), warmup=10, runs=50, inputs=[{"lhs": (1024, 1024),
"rhs": (1024, 1024),
"initializer": nd.normal,
"run_backward": True,
"dtype": "float64"}]) |
Output for the above benchmark run, on a CPU machine, would look something like below:
Code Block |
---|
MX_Add_Forward_Backward_Time - 0.025401 seconds |
Use Case 3 - Nightly CI Tests
- We will maintain a JSON file of expected performance for each operator under "incubator-mxnet/benchmark/opperf".
- These expected results are captured on different configuration such as - FP32/64/16, MKL, No MKL, CUDA10, instances (c5.16x, p3.8x).
- Runs all the operator performance runs and gets the results JSON.
- Compares with the expected results +/- % threshold.
Development Plan / Milestones
Phase 1
- ~150 most commonly used operators will be tested on CPU(with and without MKL), GPU, FP32, FP64. See Appendix 1 for list of operators.
- Operators will be tested with NDArray and Gluon interface only i.e., symbol interface is not used for testing owing to plans of deprecation.
- Python interface is used along with MXNet profiler.
- Time and Memory usage are measured to start with.
- Statistics - Mean of the metric.
Phase 2
- Cover remaining operators left out from Phase 1.
- Add more statistics - p50, p90, p99, min, max.
Phase 3
- Explore and have CPP performance tests for most commonly used operators. This will give the true measurements compared to using Python Interface.
- Integrate with property based testing libraries like Hypothesis, to randomly generate test cases with different tensor shapes and inputs.
Current Status
See this repo for more details - https://github.com/sandeep-krishnamurthy/dl-operator-benchmark
- 134 operators are supported:
- All Gluon Layers - Activation, Loss, Normalization, Basic like Dense, Convolutions, Recurrent (RNN, LSTM, GRU)
- NDArray operators like creation, random sampling, arithmetic, logical, comparison etc...
- Able to run individual operator benchmarks or use high level drivers to run all tests.
- Able to generate results as JSON.
- Timing metric - forward only, forward+backward operation.
Alternate Solutions
Alternate Solution 1 - Use Python Classes for each Operator instead of Config
Approach
- This benchmark utility will be built on top of MXNet's ND and Gluon interface.
- For each operator in ND and Gluon Block, there will be a corresponding Benchmarking operator in the library with a list of default inputs, functionality to process results. See below example for Add operator benchmarks.
- High-level drivers are provided to run operator benchmarks in bulk. Example: run_all_mxnet_operator_benchmarks(), run_all_arithmetic_operations_benchmarks() etc.
- Results can be generated as a python dictionary/JSON/CSV for upstream system (Ex: CI, Automated Performance Monitoring System) consumption.
Code Block | ||
---|---|---|
| ||
class Add(MXNetOperatorBenchmarkBase):
"""Helps to Benchmark Tensor Add operation.
By default benchmark both forward and backward element_wise tensor addition
of 1024*1024 tensor of precision - 'float32'.
"""
def __init__(self, ctx=mx.cpu(), warmup=10, runs=50, inputs=None):
# Set the default Inputs
default_parameters = {"lhs": (1024, 1024),
"rhs": (1024, 1024),
"initializer": nd.normal,
"run_backward": True,
"dtype": "float32"}
super().__init__(ctx=ctx, warmup=warmup, runs=runs, default_parameters=default_parameters,
custom_parameters=inputs)
self.lhs = get_mx_ndarray(ctx=self.ctx, in_tensor=self.inputs["lhs"],
dtype=self.inputs["dtype"],
initializer=self.inputs["initializer"],
attach_grad=self.inputs["run_backward"])
self.rhs = get_mx_ndarray(ctx=self.ctx, in_tensor=self.inputs["rhs"],
dtype=self.inputs["dtype"],
initializer=self.inputs["initializer"],
attach_grad=self.inputs["run_backward"])
def run_benchmark(self):
# Warm up, ignore execution time value
_, _ = nd_forward_backward_and_time(F=nd.add, runs=self.warmup, lhs=self.lhs, rhs=self.rhs)
# Run Benchmarks
exe_time, _ = nd_forward_backward_and_time(F=nd.add, runs=self.runs, lhs=self.lhs, rhs=self.rhs)
self.results["MX_Add_Forward_Backward_Time"] = exe_time / self.runs |
API / User Experience
We can define 2 types of users of the library and describe API interface for each of these users.
...
Code Block |
---|
{'lhs': (1024, 1024), 'rhs': (1024, 1024), 'initializer': <function normal at 0x117b607b8>, 'run_backward': True, 'dtype': 'float64'} |
Future Development and Ideas
The current proposed library is basic bare bone utility for easily running the operator benchmarks across different settings and generate the results in different formats. However, below are some ideas to evolve or build high level abstractions on top of this library:
- Graphs and insights to understand implication of various settings, such as precision/storage_type/accelerator_type/tensor shapes/etc..., on a given operator.
- Integration with tools such as MXBoard for debugging and visualization of performance bottlenecks, resource utilization and more fine grained visibility for users when training a model.
- Integration with nnvm pass for optimization, dynamic accelerator choosing and other operator level decision making capabilities.
Development Plan / Milestones
current status
See this repo for more details - https://github.com/sandeep-krishnamurthy/dl-operator-benchmark
- 134 operators are supported:
- All Gluon Layers - Activation, Loss, Normalization, Basic like Dense, Convolutions, Recurrent (RNN, LSTM, GRU)
- NDArray operators like creation, random sampling, arithmetic, logical, comparison etc...
- Timing metric - forward only, forward+backward operation.
Development plan
...
0x117b607b8>, 'run_backward': True, 'dtype': 'float64'} |
Pros
- More control for users - default inputs, random inputs, specific user defined inputs.
- Deterministic and better suited for performance benchmarks, reproducibility and CI integration.
- With Python interface:
- Easy to maintain and develop.
- Reflects the performance as seen by the users. (Majority users using Python interface)
- Fastest way to get performance tests in place. We do not have any tests in place as of today.
- Ability to run and compare benchmarks from other deep learning frameworks.
- Extensible:
- Can be integrated with property based testing libraries like Hypothesis, to randomly generate test cases with different tensor shapes.
Cons
- Need to write base tests for every new operator. If a new operator is added to MXNet, then a new performance test class for the operator needs to be added in this library with default inputs for that new operator to run performance tests.
- It is ideal to capture performance close to Kernel. Call from Python operator APIs may hide performance regression when operator computation is small.
Alternate Solution 2 - Autogenerate test with Property Based Testing Technique
(Credits - Thanks to Pedro Larroy for this suggestion)
Approach
- Automatically query all operators registered with MXNet engine.
- Infer the inputs and outputs for the operators.
- Use property based testing technique and library such as Hypothesis to generate random inputs and run the tests.
Pros
- Any new operator added to MXNet, will be automatically queried. Hence, no need to write tests explicitly for every operator.
- Inputs are randomly generated. Hence, better suited to capture performance regression on corner cases.
Cons
- Non deterministic inputs. Hence, better suitable for functionality testing. It will be hard to use this technique for performance tests.
- Still requires us to write many custom strategies or conditional property files. Example:
- For testing Add operator, we need to set conditions on input to generate same shapes or broadcastable shapes for lhs and rhs.
- For Convolution operator, we need to match Kernel, Padding and other parameter shapes appropriately.
- Querying operators and inferring the input conditions may be hard and complex logic.
- Example: add is an operator, that takes 2 input tensors - lhs, rhs. Now we need to infer that lhs and rhs tensor should of same size or broadcastable. Logic to handle such conditions may soon become complex enough to not give us advantage of auto generated operator benchmarks.
- MXNet currently do no support a standard way of querying the registered operators. It would be ideal if MXNet can expose NNVM APIs for querying registered operators and expected inputs, outputs, types and more.
- Complex and time consuming. We do not have any operator performance tests for MXNet. It would be ideal to revisit this approach for future enhancement.
Alternate Solution 3 - Extend existing unit tests to cover performance parameters
<To add more details> In summary, it is hard and complex to modify all unit tests to measure performance along with currently designed way of writing tests which is designed towards - consistency across context, correctness, gradient checks.
Appendix
Phase 1
Functionality supported:
- Able to run benchmarks for critical MXNet operators (see below for the list) with:
- Default inputs - Varying input shapes. Ex: Small Tensors, Large Tensors, Skewed Tensor, common tensor shapes.
- Custom inputs - Users should be able to specify input tensors to run benchmarks
- Random inputs - Randomly generate input tensors for operator benchmarks
- Able to save results as JSON or get results as Python Dictionary
- Able to run benchmarks on CPU/GPU.
Below are the operators covered in
- Our objective is to capture the performance at basic individual operator level.
- Symbol APIs is planned to be deprecated soon for users.
- Users currently use NDArray operations or use Gluon Layers in imperative (NDArray) or hybrid mode (Symbolic).
- In Phase 2 we will be benchmarking Gluon Hybrid Layers individually that should cover the symbolic operations exposed to users.
- Also, under the hood, the kernel is same for NDArray and Symbols. hence, we are not missing any tests.
In Phase 2, we will cover the remaining operators left out from phase 1 and also include Gluon Hybrid model this indirectly covers Symbol APIs of prominent operators.
In Phase 3, based on the requirement we will add equivalent PyTorch Operator benchmarks for having neutral baseline for MXNet operator benchmarks.
We will be running benchmarks on CPU (C5.18x) and GPU (P3.2x). For GPU we will be using GPU(0) only. In Phase 1, we will be running for FP32 and dense tensor only. In Phase 2, we will benchmark other Precisions (INT64, FP64 etc...) and also add support for Sparse Tensors. Phase 1
- Hardware
- CPU - C5.18X
- GPU - P3.2X (Single GPU)
- NDArray Operations
- Copy, CopyTo, as_in_context, asnumpy, asscalar, astype
- zeros, zeros_like, ones, ones_like, full, arange
- Transpose (T), shape_array, size_array, reshape, reshape, reshape_like, flatten, expand_dims, split, diag
- tile, pad
- sum, nansum, prod, nanprod, mean, max, min, norm
- sort, argsort, topk, argmax, argmin, argmax_channel
- add, sub, neg, mul, div, mod, pow
- iadd (+=), isub (-=), imul (*=), idiv (/=), imod (%=)
- lesser, lesser_equal, greater, greater_equal, equal, not_equal
- get_item (x[i]), set_item (x[i]=)
- slice, slice_axis, take, batch_take, pick
- one_hot
- exp, log
- sqrt, square
- concat, split, stack
- dot, batch_dot
- normal, poisson, uniform, random, randn, randint
- shuffle
- clip
- where
- abs
- Precision - FP32
- Conversion Operations
- Creation Operations
- Shape (view) change Operations
- Reduction Operations
- Sorting and Searching Operations
- Arithmetic Operations
- Inplace Arithmetic Operations
- Comparison Operations
- Indexing Operations
- Exponents and Logarithms
- Powers Operations
- Join and Split Operations
- GEMM
- Random Sampling
- Others
- Neural Network Operations
- Gluon Layers (Neural Network Operations)
- Dense, Lambda, Flatten, Embedding
- Dropout, BatchNorm
- Conv1D, Conv2D
- Conv1DTranspose, Conv2DTranspose
- MaxPool1D, MaxPool2D, AvgPool1D, AvgPool2D, GlobalMaxPool1D, GlobalMaxPool2D, GlobalAvgPool1D, GlobalAvgPool2D
- LeakyRelu, PRelu, Sigmoid, Softmax, Log_Softmax, Activation
- RNNCell, LSTMCell, GRUCell, RecurrentCell, SequentialRNNCell, BiDirectionalCell
- L1Loss, L2Loss, SigmoidBinaryCrossEntropyLoss, SoftmaxCrossEntropyLoss, KLDivLoss, HuberLoss, HingeLoss, SquaredHingeLoss, LogisticLoss, TripletLoss, CTCLoss
- Modes: Imperative (Hybrid will be added in Next Phase)
- Basic
- Convolutions
- Pooling
- Activations
- Recurrent Cells
- Loss
- Custom Operator Benchmark
...
- Other DataTypes
- INT64, INT8, FP64, FP16
- NDArray Operations
- tostype
- swapaxes, flip, depth_to_space, space_to_depth
- round, rint, fix, floor, ceil, trunc
- sin, cos, tan, arcsin, arccos, arctan, degrees, radians
- sinh, cosh, tahnh, arcsinh, arccosh, arctanh
- expm1, log10, log2, log1p
- logical_and, logical_or, logical_xor, logical_not
- rsqrt, cbrt, rcbrt, reciprocal
- exponential, gamma, generalized_negative_binomial, multinomial, negative_binomial,
- SequenceLast, SequenceMask, SequenceReverse
- unravel_index, ravel_multi_index
- Sparse
- View Operations
- Rounding Operations
- Trigonometric Operations
- Exponent and Logarithmic Operations
- Logical Operations
- Powers Operations
- Random Operations
- Sequence Operations
- Others
- Gluon
- Conv + Relu, Conv + BatchNorm (More to be added when we start this work)
- HybridLambda
- InstanceNorm, LayerNorm
- Conv3D
- Conv3DTranspose
- MaxPool3D, AvgPool3D, GlobalMaxPool3D, GlobalAvgPool3D
- , Elu, Selu, Swish
- ZoneOutCell, ResidualCell, DropoutCell
- CosineEmbeddingLoss, PoissonNLLoss
- Mode: Hybrid Mode for all layers covered in Phase 1. Additional coverage of layers as below.
- Fused Operators
- Basic
- Convolutions
- Pooling
- Activations
- Recurrent
- Loss
- Important Contrib Layers
- Other Items to be explored
- Image APIs
- Data APIs
- Metric APIs
- Initializers and Optimizers
Phase 3 (To be discussed/scoped)
Benchmark PyTorch Operators to have neutral baseline for comparing MXNet operator performance. This is yet to be discussed and finalized.
...
Q1) Why not use check_speed(..) utility in MXNet test_util?
A) Supports Symbol APIs only. Do not support benchmarking NDArray, Gluon Blocks. It is a lightweight simple symbol executor and expects users to create symbol graph to execute along with inputs. The proposed library in this document is more sophisticated by supporting benchmarking operators with NDArray, Gluon Block, provide various default inputs, provide high-level drivers, provide interface for users to specify different inputs, prepare results in different formats - Python dictionary, CSV, JSON.
Q2) Why not Symbol execution? Why only NDArray and Gluon?
A) MXNet users are encouraged and are mainly using NDArray APIs or Gluon APIs. MXNet community is moving towards deprecating symbol APIs with works on numpy compatible operators. To measure what our users use and observe, we propose to use NDArray and Gluon blocks for benchmarking operators in this library.
Q3) Why not extend/repurpose current MXNet unit tests?
Q4) Why Python? Shouldn't we benchmark operators as close to Kernel as possible i.e., C++ benchmarks?