THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
- Provide a generic utility for executing an operator benchmarks and performance tests.
- This is responsible to creating input tensors of required shape on a given dtype, context.
- Execute the provided operator - forward or forward + backward.
- This generic utility will be integrated with MXNet profiler.
- Captures the Capture profile output from MXNet profiler - time, memory.
- Return a dictionary of results.
- Input for the performance tests will be a key/value config.
...
| Code Block | ||
|---|---|---|
| ||
"""
MXNet operator performance benchmarks.
NOTE:
1. You can pass list of input dictionary to run benchmarks for an operator with different input configuration.
2. Results are dictionary of time, memory for the benchmark runs.
"""
# Run performance test for Add operator
results = run_performance_test(F=mx.nd.add, ctx=mx.cpu(), warmup=10, runs=50, inputs=[{"lhs": (1024, 1024),
"rhs": (1024, 1024),
"initializer": nd.normal,
"run_backward": True,
"dtype": "float32"}])
# Run performance test for Conv2D operator
results += run_performance_test(F=nn.gluon.Conv2D, ctx=mx.cpu(), warmup=10, runs=50, inputs = [{"data": (32, 3, 256, 256),
"data_initializer": nd.normal,
"channels": 64,
"kernel_size": (3, 3),
"strides": (1, 1),
"padding": (0, 0),
"dilation": (1, 1),
"layout": "NCHW",
"activation": None,
"run_backward": True,
"dtype": "float32"]}
|
How does the backend profiling utility code looks like?
Below we take an example of profiling Add operator.
| Code Block | ||
|---|---|---|
| ||
# Configurations
warmup = 25
runs = 50
run_backward = True
# Operator to benchmark
F = mx.nd.add
# Prepare data for the operator
lhs = mx.nd.ones(shape=(1024, 1024))
rhs = mx.nd.ones(shape=(1024, 1024))
lhs.attach_grad()
rhs.attach_grad()
mx.nd.waitall()
# Warmup
print("Warming up....")
for _ in range(warmup):
with mx.autograd.record():
res = mx.nd.add(lhs, rhs)
res.backward()
mx.nd.waitall()
print("Done warming up....")
# Run Performance Runs
print("Running performance runs....")
profiler.set_config(profile_all=True, aggregate_stats=True)
# Start Profiler
profiler.set_state('run')
for _ in range(runs):
with mx.autograd.record():
res = mx.nd.add(lhs1, rhs1)
res.backward()
mx.nd.waitall()
# Stop Profiler
profiler.set_state('stop')
# Fetch Results from Profiler
# We will add 2 new APIs in Profiler - profiler.get_summary(), profiler.reset()
# profiler.get_summary() => will be a JSON string representing the output as shown below.
# profiler.reset() => Resets all the counter in the current profiler.
print("Done Running performance runs....")
print(profiler.dumps(reset=True))
|
How to capture Time?
We will be using MXNet profiler
Pros
- No need to write 1 class per operator to set up a performance test. Whenever a new operator is created, developer needs to add a `run_performance_test(..)` line with a list of inputs to run performance tests. A generic utility will handle the execution.
- Less code, easy to maintain.
- More control for users - default inputs, random inputs, specific user defined inputs.
- Deterministic and better suited for performance benchmarks, reproducibility and CI integration.
- With Python interface:
- Easy to maintain and develop.
- Reflects the performance as seen by the users. (Majority users using Python interface)
- Fastest way to get performance tests in place. We do not have any tests in place as of today.
...