Deep learning frameworks are core libraries that are increasingly used in a number of important real-world scenarios. MXNet is no exception, being used in range of production environments from embedded hardware, with very little RAM, to multi-million dollar web services running tens of thousands of requests per second. MXNet is written in C++ and has a lot of raw pointer operations due to its high-performance, mathematical nature. This introduces the potential for serious native coding errors to negatively affect important services and devices. This document will describe how we're attempting to avoid these errors by introducing a heavily instrumented build (an ASAN build) that's designed to catch overflows and leaks in our automatic testing process. The document will also describe how a developer can create an ASAN build and test for memory leaks locally when they're reported by users.
ASAN, or the address sanitizer, is one of many C++ sanitizers developed by Google with the primary initial goal of securing Chrome from use-after-free and buffer-overflow errors. It was originally launched as a feature for clang, but is now available in recent versions of GCC. ASAN on it's own will detect:
The advantage of ASAN compared to other similar tools such as Valgrind is that it's fast, and well supported. This is also why it's the primary mechanism of detecting leaks for the communities that manage both Chrome and Firefox.
The capabilities of ASAN are similar whether the compiler used is clang or g++, as long as they're a very recent version of the compilers. There's a good comparison of clang versus gcc7, in terms of ASAN capabilities here. With MXNet we've tested various versions of compilers and found that clang does not work well with our library. We haven't completely tracked down the issue, but trying various methods of enabling ASAN has not worked for us when using clang, including forcing clang to dynamically link the ASAN library. Luckily GCC ASAN seems to work correctly, and GCC 8 has the ASAN capabilities that we'd like to use. Because of this we are using GCC 8 w/ ASAN in CI and the Dockerfiles referred to below.
We're currently experimenting with a variety of methods for incorporating automatic ASAN checks in our CI system. We will likely run these checks per-PR request while the detection is under active development. We'll likely migrate the checks to nightly builds to reduce cost once the development is stable. Thus far we've only enabled one variant of MXNet's build type, a basic CPU build. We hope this will serve a basis for other developers to integrate ASAN with other MXNet build flavours.
When ASAN builds are enabled we have leaks that are reported when running almost any MXNet test. If you want to focus on the possibly more important memory errors such as buffer overflows, you can turn off leak detection by setting ASAN_OPTIONS=detect_leaks=0.
Other Sanitizers