Table of Contents |
---|
PR
https://github.com/apache/incubator-mxnet/pull/17510
Problem
Currently MXNet FFI overhead is significantly larger than official numpy [1]. MXNet FFI passes its arguments mainly via ctypes. All keyword arguments are serialized to string at frontend and deserialized from string at backend, which is slow. Also the number of APIs exposed to dll is very large, which is difficult to maintain. For now, few of them have cython interfaces, which worsens the FFI performance.
...
Adapt TVM registry for MXNet PackedFunc registration in backend
Adapt TVM
function
as MXNet PackedFunc wrapper in frontendExtend TVMValue for customized and efficient MXNet argument passing.
Example
A demo op np.zeros1zeros with the new FFI interface is implemented as an example, whose behavior is identical to np.zeros with the new FFI interface. Files mentioned in the example can be found in the POC mentioned above. It will help to read both of the frontend and backend registration if one wants to register some other ops.
Firstly, in frontend, np.zeros1 is zeros is registered in python/mxnet/ndarray/numpy/_op.py as usual:
Code Block | ||||
---|---|---|---|---|
| ||||
@set_module('mxnet.ndarray.numpy') def zeros1zeros(shape, dtype=None, order='C', ctx=None): # pylint: disable=redefined-outer-name if order != 'C': raise NotImplementedError # If the following code (4 lines) regarding ctx is removed # np.zeros((3, 4)) can be as fast as 4.96 us if ctx is None: ctx = str(current_context()) else: ctx = str(ctx) if dtype is not None and not isinstance(dtype, str): dtype = _np.dtype(dtype).name return _api_npiinternal.zeros1zeros(shape, dtype, ctx) |
Secondly, in backend, _npi.zeros1 is zeros is registered in src/api/api_npioperator/numpy/np_init_op.cc, as follows.
Code Block | ||||
---|---|---|---|---|
| ||||
MXNET_REGISTER_API("_npi.zeros1zeros") .set_body([](runtime::MXNetArgs args, runtime::MXNetRetValue* ret) { // Part1: populate NodeAttrs // part1Part1: populate attrsNodeAttrs using namespace runtime; const static nnvm::Op* op = Op::Get("_npi_zeros"); nnvm::NodeAttrs attrs; op::InitOpParam param; if (args[0].type_code() == kDLInt) { param.shape = TShape(1, args[0].operator int64_t()); } else { param.shape = TShape(args[0].operator ObjectRef()); } if (args[1].type_code() == kNull) { param.dtype = mshadow::kFloat32; } else { param.dtype = runtime::String2MXNetTypeWithBool(args[1].operator std::string()); } attrs.parsed = std::move(param); attrs.op = op; SetAttrDict<op::InitOpParam>(&attrs); if (args[2].type_code() != kNull) { attrs.dict["ctx"] = args[2].operator std::string(); } // part2Part2: invoke int num_outputs = 0; auto ndoutputs = Invoke(op, &attrs, 0, nullptr, &num_outputs, nullptr); *ret = ndoutputs[0]; }); |
The first part of npi.zeros1 is zeros is mainly responsible for populating NodeAttrs, and the second part invokes the op in an imperative way, which is similar to the api MXImperativeInvokeImpl.
As is seen, _npi.zeros1 is zeros is not directly exposed to the dll boundary but can be invoked in the frontend. If we go deeper into operator TShape(), we will see that TShape is initialized from ADTObj, which is like an array of integers in this case. As it works without (de)serialization between string and int, some speedup is obtained here.
Also, here are a few things to note:
- In the original np.zeros, the default value for dtype is specified in front end, while in in np.zeros1zeros with the new FFI interface as is shown above, it is specified in backend, which slightly speeds it up. Generally I think it may be better to move code to backend, as python is sometimes slow.
- ctx is still getting (de)serialized. Joint effort with engine overhead optimization may be required, as ctx gets deserialized in engine.
...
- python/mxnet/ndarray/numpy/_op.py: def zeros1zeros. The python entry.
- python/mxnet/_ffi/function.py: class Function. The front end wrapper to hide the details about ctypes and cython.
- python/mxnet/_ffi/_cython/function.pxi: cdef class FunctionBase. The cython entry.
- python/mxnet/_ffi/_cython/function.pxi: def __call__. Here we call make_ret to convert MXNetValue into python object.
- python/mxnet/_ffi/_cython/function.pxi: cdef inline int FuncCall. Here we call make_arg to convert python object into MXNetValue.
- src/runtime/c_runtime_api.cc: int MXNetFuncCall. The cpp entry. It is an API exposed to dll. It wraps the abi-compatible c struct MXNetValue into easy-to-use cpp class MXNetArgs and MXNetRetValue.
- include/mxnet/runtime/packed_func.h: inline void PackedFunc::CallPacked.
- src/api/api_npioperator/numpy/np_init_op.cc: MXNET_REGISTER_API(“_npi.zeros1”zeros”). Here we arrive at the user-defined backend function.
...
Benchmarked on c5n.4x Ubuntu 16.04 LTS with NaiveEngine and cython enabled.
Workload | Current FFI (us) | TVM FFI (us) |
zeros((3, 4)) | 27.2 | 4.9 |
zeros((3, 4), dtype=‘float64’) | 28.3 | 5.6 |
zeros((3, 4), ctx = “cpu(0)”, dtype=‘float64’) | 26.4 | 6.7 |
tensordot(a, b, ((1, 0), (0, 1))) | 31.8 | 13.3 |
References
[1] [RFC] MXNet Imperative Op Invocation Overhead https://github.com/apache/incubator-mxnet/issues/17097
[2] [PR] Fix collect_params().zero_grad() in gluon numpy interface https://github.com/apache/incubator-mxnet/pull/16716
[3] [DGL] https://docs.dgl.ai/en/latest/developer/ffi.html
[4] [COMMENT] https://github.com/apache/incubator-mxnet/issues/17097#issuecomment-568325041