You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Deep learning frameworks are core libraries that are increasingly used in a number of important real-world scenarios.  MXNet is no exception, being used  in range of production environments from embedded hardware, with very little RAM, to multi-million dollar web services running tens of thousands of requests per second.  MXNet is written in C++ and has a lot of raw pointer operations due to its high-performance, mathematical nature.  This introduces the potential for serious native coding errors to negatively affect important services and devices.  This document will describe how we're attempting to avoid these errors by introducing a heavily instrumented build (an ASAN build) that's designed to catch overflows and leaks in our automatic testing process.  The document will also describe how a developer can create an ASAN build and test for memory leaks locally when they're reported by users.

ASAN

ASAN, or the address sanitizer, is one of many C++ sanitizers developed by Google with the primary initial goal of securing Chrome from use-after-free and buffer-overflow errors.  It was originally launched as a feature for clang, but is now available in recent versions of GCC.  ASAN on it's own will detect:

  • Use after free errors
  • Heap buffer overflows
  • Stack buffer overflows
  • Global buffer overflows
  • Use after returns
  • Use after scopes
  • Initialization order bugs
  • Memory leaks

The advantage of ASAN compared to other similar tools such as Valgrind is that it's fast, and well supported.  This is also why it's the primary mechanism of detecting leaks for the communities that manage both Chrome and Firefox. 

GCC versus Clang compatibility

The capabilities of ASAN are similar whether the compiler used is clang or g++, as long as they're a very recent version of the compilers.  There's a good comparison of clang versus gcc7, in terms of ASAN capabilities here.  With MXNet we've tested various versions of compilers and found that clang does not work well with our library.  We haven't completely tracked down the issue, but trying various methods of enabling ASAN has not worked for us when using clang, including forcing clang to dynamically link the ASAN library.  Luckily GCC ASAN seems to work correctly, and GCC 8 has the ASAN capabilities that we'd like to use.  Because of this we are using GCC 8 w/ ASAN in CI and the Dockerfiles referred to below.

ASAN in CI

We're currently experimenting with a variety of methods for incorporating automatic ASAN checks in our CI system.  We will likely run these checks per-PR request while the detection is under active development.  We'll likely migrate the checks to nightly builds to reduce cost once the development is stable.  Thus far we've only enabled one variant of MXNet's build type, a basic CPU build.  We hope this will serve a basis for other developers to integrate ASAN with other MXNet build flavours.

C++ Training Test With Leak Detection

Example Output

Python Tests Without Leak Detection

Example Output

Using ASAN builds with MXNet

Disabling Memory Leak Detection

When ASAN builds are enabled we have leaks that are reported when running almost any MXNet test.  If you want to focus on the possibly more important memory errors such as buffer overflows, you can turn off leak detection by setting ASAN_OPTIONS=detect_leaks=0.

Other Sanitizers



  • No labels