Page History

...

1. Benchmarks for Apache MXNet operators.
2. Common combination (Fused) of operators. Conv + Relu, Conv + BatchNorm.
3. Individual operator benchmarks to capture - time for operator execution (speed), memory usage.
4. Fine grained individual operator benchmarks to capture - time for forward pass, time for backward pass and both.
5. Ability to run operator benchmarks with default inputs or customize with user specific inputs.
6. Ability to run operator benchmarks on CPU/GPU with different flavors of MXNet (mxnet-mkl, mxnet-cu90mkl etc.)
7. Benchmarks for operators with varying inputs to uncover any performance issues due to skewed input data. Ex: Measuring operator performance on small input tensors, large input tensors along with average normally used tensor sizes.
8. Ability to run one, group or all operator benchmarks.
9. Ability to extract results in multiple usable format - Python Dictionary, JSON, CSV, MD
10. Statistics:
    1. Mean, Median, P50, P90, P99

Design Tenets

1. Defaults => Common use cases should be extremely easy, customized complex use cases should be possible.
1. Example: I should be able to run Add operator benchmarks without specifying any inputs and library should provide benchmarks on valid default inputs. At the same time, as a power user, I should be able to provide my own inputs such as Tensor Shapes and context to run the benchmarks.
2. Minimum Learning Curve => Keep APIs same or close to native NDArray / Gluon Operators being benchmarked.
1. Example: If I am doing benchmarks on nd.add(lhs, rhs) operator, interface in the benchmark utility should be similar with zero learning curve.
3. Modular and Reusable
4. For a programmer or an automated system
1. Example: Developer using the library or integration with CI/CD

...