Page History

Deep learning frameworks are core libraries that are increasingly used in a number of important real-world scenarios.  MXNet is no exception, being used  in range of production environments from embedded hardware, with very little RAM, to multi-million dollar web services running tens of thousands of requests per second.  MXNet is written in C++ and has a lot of raw pointer operations due to its high-performance, mathematical nature.  This introduces the potential for serious native coding errors to negatively affect important services and devices.  This document will describe how we're attempting to avoid these errors by introducing a heavily instrumented build (an ASAN build) that's designed to catch overflows and leaks in our automatic testing process.  The document will also describe how a developer can create an ASAN build and test for memory leaks locally when they're reported by users.

Background

ASAN

ASAN, or the address sanitizer, is one of many C++ sanitizers developed by Google with the primary initial goal of securing Chrome from use-after-free and buffer-overflow errors.  It was originally launched as a feature for clang, but is now available in recent versions of GCC.  ASAN on it's its own will detect:

• Use after free errors
• Heap buffer overflows
• Stack buffer overflows
• Global buffer overflows
• Use after returns
• Use after scopes
• Initialization order bugs
• Memory leaks

The advantage of ASAN compared to other similar tools such as Valgrind is that it's fast, and well supported.  This is also why it's the primary mechanism of detecting leaks for the communities that manage both Chrome and Firefox.   For more information on ASAN (and other sanitizer) basics and motivations check out this talk.

GCC versus Clang

The capabilities of ASAN are similar whether the compiler used is clang or g++, as long as you're using very recent version of the compilers.  There's a good comparison of clang versus gcc7, in terms of ASAN capabilities, here.  With MXNet we've tested various versions of compilers and found that clang does not work well with our library.  We haven't completely tracked down the issue, but trying various methods of enabling ASAN has not worked for us when using clang, including forcing clang to dynamically link the ASAN library.  Luckily GCC ASAN seems to work correctly, and GCC 8 has the ASAN capabilities that we'd like to use.  Because of this we recommend using GCC 8 w/ ASAN when attempting to detect leaks or buffer overflows in MXNet, and we use GCC8 in CI and the Dockerfiles referred to below.

ASAN versus Valgrind

ASAN is often compared to Valgrind.  Both perform similar actions and test for similar failures.  There are several differences, but one major one is that ASAN works by instrumenting the during compilation, Valgrind works by instrumenting binaries after compilation.  This design choice allows ASAN to run much faster than Valgrind, which makes it easier to run real world use cases.   Sanitizers also .   Another major difference in the context of MXNet is that Valgrind does not support the F16C instruction, which is built be default with MXNet (meaning default builds are incompatible with Valgrind).  Additionally, Sanitizers generally have better multithreading support than Valgrind.  ASAN also has wide community adoption, and several large technology communities are actively contributing to ASAN (Apple, Chrome, Firefox, Go-lang, LLVM).  One disadvantage to ASAN is that it's a relatively new tool compared to Valgrind, because of this it's best to use as up-to-date a version of ASAN as possible.

...

git clone --recurse https://github.com/apache/incubator-mxnet.git
cd incubator-mxnet/ci
# Build our dockerfile with all required deps for ASAN
docker build -f docker/Dockerfile.build.ubuntu_cpu -t mxnetci/build.ubuntu_cpu docker
cd ..
mkdir -p build
# Build an ASAN instrumented MXNet library
# Privileged probably not required for all steps, but in general ASAN requires some capabilities in order to inspect process memory.
docker run --privileged -v `pwd`:/work/mxnet -v `pwd`/build:/work/build  -ti mxnetci/build.ubuntu_cpu /work/runtime_functions.sh build_ubuntu_cpu_cmake_asan

# Now choose an example of something you'd like to test with ASAN.  This could be a specific python test we want to run in a loop, it could be a C++ unit test written to expose leaks, etc.
# In our case we use a small Gluon python tests as an example.
# First we'll will enter our container, and then we'll run tests within the container.
docker run --privileged -v `pwd`:/work/mxnet -v `pwd`/build:/work/build  -ti mxnetci/build.ubuntu_cpu bash

# Now within the container:
export PYTHONPATH=./python/
export MXNET_MKLDNN_DEBUG=1  # Ignored if not present
export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0
# Feel free to export any ASAN options via export ASAN_OPTIONS=...

# Importantly we need to make sure ASAN is the first library loaded (before our other libraries have a chance to allocate memory)
# To do this we add the library to the library preload list
export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libasan.so.5
nosetests-3.4 --verbose tests/python/unittest/test_rnn.py

...

Once the test is run, a developer will have to look at the stacks of memory allocations and determine which are important, and which are running as designed.  All of the reports in the sample output are displaying memory leak information, but other bugs will also be output and included in the summary if they are found.  If you are trying to reproduce a memory leak reported by users the recommend approach would be to reproduce the error by emulating the user's use case and running it in a loop with ASAN enabled.  You can then reduce the scope (for example to a C++ unit test) while continuing to run in a loop.  Eventually the stack traces should make it clear why the leak occurs.

MISC

Running in CLion

If you prefer debugging in an IDE, or you wish to break on ASAN errors you can use CLion.  You'll need to install all the required dependencies manually (but you can reference the CI Dockerfiles for help).  On Ubunut 16.04 this is as simple as installing gcc-8, and setting it as the project compiler in CLion.  You can then add the -DUSE_ASAN build flag for your project to enable ASAN support.  Finally you must add LD_PRELOAD to your run environment variables for your launch target and point it at the version of ASAN you have installed (libasan.so.5 for gcc8).

Disabling Memory Leak Detection

When ASAN builds are enabled we have leaks that are reported when running almost any MXNet test.  If you want to focus on the possibly more important memory errors such as buffer overflows, you can turn off leak detection by setting ASAN_OPTIONS=detect_leaks=0.

Other Sanitizers

After enabling and addressing issues reported by ASAN we can enable other sanitizers following the same template.  The two most applicable sanitizers are described below.

TSAN

TSAN is a sanitizer that detects data races and other thread-saftey errors in native libraries. TSAN works in a similar fashion to ASAN.  It instruments builds and surrounds memory with protect access buffers.  It then uses this instrumented code and specially protected buffers to ensures that each thread accesses memory in a threadsafe way.  TSAN supports C++11 atomics and other modern C++ features.  TSAN has more overhead (especially in memory usage) than ASAN.

MSAN

MSAN detects uninitialized memory accesses.  This could help us reduce errors in MXNet, especially difficult to reproduce, non-deterministic errors.  MSAN has a slowdown of roughly 3x when it instruments MXNet.

...