Page History

...

The advantage of ASAN compared to other similar tools such as Valgrind is that it's fast, and well supported.  This is also why it's the primary mechanism of detecting leaks for the communities that manage both Chrome and Firefox.     For more information on ASAN (and other sanitizer) basics and motivations check out this talk.

GCC versus Clang compatibility

The capabilities of ASAN are similar whether the compiler used is clang or g++, as long as theyyou're a using very recent version of the compilers.  There's a good comparison of clang versus gcc7, in terms of ASAN capabilities, here.  With MXNet we've tested various versions of compilers and found that clang does not work well with our library.  We haven't completely tracked down the issue, but trying various methods of enabling ASAN has not worked for us when using clang, including forcing clang to dynamically link the ASAN library.  Luckily GCC ASAN seems to work correctly, and GCC 8 has the ASAN capabilities that we'd like to use.  Because of this we are recommend using GCC 8 w/ ASAN when attempting to detect leaks or buffer overflows in MXNet, and we use GCC8 in CI and the Dockerfiles referred to below.

...

When ASAN builds are enabled we have leaks that are reported when running almost any MXNet test.  If you want to focus on the possibly more important memory errors such as buffer overflows, you can turn off leak detection by setting ASAN_OPTIONS=detect_leaks=0.

Other Sanitizers

After enabling and addressing issues reported by ASAN we can enable other sanitizers following the same template.  The two most applicable sanitizers are described below.

TSAN

TSAN is a sanitizer that detects data races and other thread-saftey errors in native libraries. TSAN works in a similar fashion to ASAN.  It instruments builds and surrounds memory with protect access buffers.  It then uses this instrumented code and specially protected buffers to ensures that each thread accesses memory in a threadsafe way.  TSAN supports C++11 atomics and other modern C++ features.  TSAN has more overhead (especially in memory usage) than ASAN.

MSAN

MSAN detects uninitialized memory accesses.  This could help us reduce errors in MXNet, especially difficult to reproduce, non-deterministic errors.  MSAN has a slowdown of roughly 3x when it instruments MXNet.