Deep learning frameworks are core libraries that are increasingly used in a number of important real-world scenarios. MXNet is no exception, being used in range of production environments from embedded hardware, with very little RAM, to multi-million dollar web services running tens of thousands of requests per second. MXNet is written in C++ and has a lot of raw pointer operations due to its high-performance, mathematical nature. This introduces the potential for serious native coding errors to negatively affect important services and devices. This document will describe how we're attempting to avoid these errors by introducing a heavily instrumented build (an ASAN build) that's designed to catch overflows and leaks in our automatic testing process. The document will also describe how a developer can create an ASAN build and test for memory leaks locally when they're reported by users.
ASAN, or the address sanitizer, is one of many C++ sanitizers developed by Google with the primary initial goal of securing Chrome from use-after-free and buffer-overflow errors. It was originally launched as a feature for clang, but is now available in recent versions of GCC. ASAN on it's own will detect:
The advantage of ASAN compared to other similar tools such as Valgrind is that it's fast, and well supported. This is also why it's the primary mechanism of detecting leaks for the communities that manage both Chrome and Firefox. For more information on ASAN (and other sanitizer) basics and motivations check out this talk.
The capabilities of ASAN are similar whether the compiler used is clang or g++, as long as you're using very recent version of the compilers. There's a good comparison of clang versus gcc7, in terms of ASAN capabilities, here. With MXNet we've tested various versions of compilers and found that clang does not work well with our library. We haven't completely tracked down the issue, but trying various methods of enabling ASAN has not worked for us when using clang, including forcing clang to dynamically link the ASAN library. Luckily GCC ASAN seems to work correctly, and GCC 8 has the ASAN capabilities that we'd like to use. Because of this we recommend using GCC 8 w/ ASAN when attempting to detect leaks or buffer overflows in MXNet, and we use GCC8 in CI and the Dockerfiles referred to below.
We're currently experimenting with a variety of methods for incorporating automatic ASAN checks in our CI system. We will likely run these checks per-PR request while the detection is under active development. We'll likely migrate the checks to nightly builds to reduce cost once the development is stable. Thus far we've only enabled one variant of MXNet's build type, a basic CPU build. We hope this will serve a basis for other developers to integrate ASAN with other MXNet build flavours.
CI reports are nice, but it's sometimes more useful to build an ASAN build locally and to run some specific sections of code you're afraid may be leaking. This is easy to do with MXNet. We've installed all the prerequisites in or CI build Dockerfiles, so we can use docker to build ASAN builds without having to install or configure dependencies. To build a CPU build with ASAN run the following commands in a new folder:
git clone --recurse https://github.com/apache/incubator-mxnet.git cd incubator-mxnet/ci # Build our dockerfile with all required deps for ASAN docker build -f docker/Dockerfile.build.ubuntu_cpu -t mxnetci/build.ubuntu_cpu docker cd .. mkdir -p build # Build an ASAN instrumented MXNet library # Privileged probably not required for all steps, but in general ASAN requires some capabilities in order to inspect process memory. docker run --privileged -v `pwd`:/work/mxnet -v `pwd`/build:/work/build -ti mxnetci/build.ubuntu_cpu /work/runtime_functions.sh build_ubuntu_cpu_cmake_asan # Choose an example of something you'd like to test with ASAN. This could be a specific python test we want to run in a loop, it could be a C++ unit test written to expose leaks, etc. # In our case we use a small Gluon python tests as an example. # First we'll enter our container, and then we'll run tests within the container. docker run --privileged -v `pwd`:/work/mxnet -v `pwd`/build:/work/build -ti mxnetci/build.ubuntu_cpu bash # Now within the container: export PYTHONPATH=./python/ export MXNET_MKLDNN_DEBUG=1 # Ignored if not present export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0 # Feel free to export any ASAN options via export ASAN_OPTIONS=... # Importantly we need to make sure ASAN is the first library loaded (before our other libraries have a chance to allocate memory) # To do this we add the library to the library preload list export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libasan.so.5 nosetests-3.4 --verbose tests/python/unittest/test_rnn.py |
When ASAN builds are enabled we have leaks that are reported when running almost any MXNet test. If you want to focus on the possibly more important memory errors such as buffer overflows, you can turn off leak detection by setting ASAN_OPTIONS=detect_leaks=0.
After enabling and addressing issues reported by ASAN we can enable other sanitizers following the same template. The two most applicable sanitizers are described below.
TSAN is a sanitizer that detects data races and other thread-saftey errors in native libraries. TSAN works in a similar fashion to ASAN. It instruments builds and surrounds memory with protect access buffers. It then uses this instrumented code and specially protected buffers to ensures that each thread accesses memory in a threadsafe way. TSAN supports C++11 atomics and other modern C++ features. TSAN has more overhead (especially in memory usage) than ASAN.
MSAN detects uninitialized memory accesses. This could help us reduce errors in MXNet, especially difficult to reproduce, non-deterministic errors. MSAN has a slowdown of roughly 3x when it instruments MXNet.