Page History

...

Code Block

git clone --recurse https://github.com/apache/incubator-mxnet.git
cd incubator-mxnet/ci
# Build our dockerfile with all required deps for ASAN
docker build -f docker/Dockerfile.build.ubuntu_cpu -t mxnetci/build.ubuntu_cpu docker
cd ..
mkdir -p build
# Build an ASAN instrumented MXNet library
# Privileged probably not required for all steps, but in general ASAN requires some capabilities in order to inspect process memory.
docker run --privileged -v `pwd`:/work/mxnet -v `pwd`/build:/work/build  -ti mxnetci/build.ubuntu_cpu /work/runtime_functions.sh build_ubuntu_cpu_cmake_asan
# Choose an example of something you'd like to test with ASAN.  This could be a specific python test we want to run in a loop, it could be a C++ unit test written to expose leaks, etc.

# In our case we use a small Gluon python tests as an example.
# First we'll enter our container, and then we'll run tests within the container.
docker run --privileged -v `pwd`:/work/mxnet -v `pwd`/build:/work/build  -ti mxnetci/build.ubuntu_cpu bash

# Now within the container:
export PYTHONPATH=./python/
export MXNET_MKLDNN_DEBUG=1  # Ignored if not present
export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0
# Feel free to export any ASAN options via export ASAN_OPTIONS=...

# Importantly we need to make sure ASAN is the first library loaded (before our other libraries have a chance to allocate memory)
# To do this we add the library to the library preload list
export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libasan.so.5
nosetests-3.4 --verbose tests/python/unittest/test_rnn.py

The output should look similar to:

Code Block

root@e77a083c4d00:/work/mxnet# nosetests-3.4 --verbose tests/python/unittest/test_rnn.py
test_rnn.test_deprecated ... ok
test_rnn.test_rnn ... ok
test_rnn.test_lstm ... ok
test_rnn.test_lstm_forget_bias ... ok
test_rnn.test_gru ... ok
test_rnn.test_residual ... ok
test_rnn.test_residual_bidirectional ... ok
test_rnn.test_stack ... ok
test_rnn.test_bidirectional ... ok
test_rnn.test_zoneout ... ok
test_rnn.test_unfuse ... ok
test_rnn.test_convrnn ... ok
test_rnn.test_convlstm ... ok
test_rnn.test_convgru ... ok
test_rnn.test_encode_sentences ... ok

----------------------------------------------------------------------
Ran 15 tests in 0.253s

OK

=================================================================
==93==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 2414848 byte(s) in 572 object(s) allocated from:
    #0 0x7f7bacb38b60 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xedb60)
    #1 0x59c889  (/usr/bin/python3.5+0x59c889)

...

Direct leak of 1640 byte(s) in 1 object(s) allocated from:
    #0 0x7f7bacb3a970 in operator new[](unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xef970)
    #1 0x7f7b53007bf0 in mxnet::profiler::Profiler::Profiler() /work/mxnet/src/profiler/profiler.cc:70
    #2 0x7f7b53028372 in void __gnu_cxx::new_allocator<mxnet::profiler::Profiler>::construct<mxnet::profiler::Profiler>(mxnet::profiler::Profiler*) (/work/mxnet/python/mxnet/../../build/libmxnet.so+0x68db372)
    #3 0x7f7b53027316 in void std::allocator_traits<std::allocator<mxnet::profiler::Profiler> >::construct<mxnet::profiler::Profiler>(std::allocator<mxnet::profiler::Profiler>&, mxnet::profiler::Profiler*) /usr/include/c++/8/bits/alloc_traits.h:475
    #4 0x7f7b53025b9d in std::_Sp_counted_ptr_inplace<mxnet::profiler::Profiler, std::allocator<mxnet::profiler::Profiler>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<>(std::allocator<mxnet::profiler::Profiler>) /usr/include/c++/8/bits/shared_ptr_base.h:549
    #5 0x7f7b530232fe in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<mxnet::profiler::Profiler, std::allocator<mxnet::profiler::Profiler>>(std::_Sp_make_shared_tag, mxnet::profiler::Profiler*, std::allocator<mxnet::profiler::Profiler> const&) /usr/include/c++/8/bits/shared_ptr_base.h:662
    #6 0x7f7b5302053f in std::__shared_ptr<mxnet::profiler::Profiler, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<mxnet::profiler::Profiler>>(std::_Sp_make_shared_tag, std::allocator<mxnet::profiler::Profiler> const&) /usr/include/c++/8/bits/shared_ptr_base.h:1328
    #7 0x7f7b5301cc03 in std::shared_ptr<mxnet::profiler::Profiler>::shared_ptr<std::allocator<mxnet::profiler::Profiler>>(std::_Sp_make_shared_tag, std::allocator<mxnet::profiler::Profiler> const&) /usr/include/c++/8/bits/shared_ptr.h:360
    #8 0x7f7b53019822 in std::shared_ptr<mxnet::profiler::Profiler> std::allocate_shared<mxnet::profiler::Profiler, std::allocator<mxnet::profiler::Profiler>>(std::allocator<mxnet::profiler::Profiler> const&) /usr/include/c++/8/bits/shared_ptr.h:707
    #9 0x7f7b53015d3b in std::shared_ptr<mxnet::profiler::Profiler> std::make_shared<mxnet::profiler::Profiler>() /usr/include/c++/8/bits/shared_ptr.h:723
    #10 0x7f7b530088e1 in mxnet::profiler::Profiler::Get(std::shared_ptr<mxnet::profiler::Profiler>*) /work/mxnet/src/profiler/profiler.cc:106
    #11 0x7f7b531d9778 in mxnet::engine::ThreadedEngine::ThreadedEngine() /work/mxnet/src/engine/./threaded_engine.h:310
    #12 0x7f7b531dc9ac in mxnet::engine::ThreadedEnginePerDevice::ThreadedEnginePerDevice() /work/mxnet/src/engine/threaded_engine_perdevice.cc:54
    #13 0x7f7b531d73a9 in mxnet::engine::CreateThreadedEnginePerDevice() /work/mxnet/src/engine/threaded_engine_perdevice.cc:342
    #14 0x7f7b5320633e in mxnet::engine::CreateEngine() /work/mxnet/src/engine/engine.cc:45
    #15 0x7f7b53205e4c in mxnet::Engine::_GetSharedRef() /work/mxnet/src/engine/engine.cc:62
    #16 0x7f7b53205fab in mxnet::Engine::Get() /work/mxnet/src/engine/engine.cc:67
    #17 0x7f7b5308422c in mxnet::LibraryInitializer::LibraryInitializer()::{lambda()#1}::operator()() const /work/mxnet/src/initialize.cc:54
    #18 0x7f7b5308428d in mxnet::LibraryInitializer::LibraryInitializer()::{lambda()#1}::_FUN() /work/mxnet/src/initialize.cc:55
    #19 0x7f7bac5303a4 in __fork (/lib/x86_64-linux-gnu/libc.so.6+0xcc3a4)
    #20 0x5e9abc  (/usr/bin/python3.5+0x5e9abc)
...

SUMMARY: AddressSanitizer: 7521184 byte(s) leaked in 3071 allocation(s).

Once the test is run, a developer will have to look at the stacks of memory allocations and determine which are important, and which are running as designed. All of the reports in the sample output are displaying memory leak information, but other bugs will also be output and included in the summary if they are found. If you are trying to reproduce a memory leak reported by users the recommend approach would be to reproduce the error by emulating the user's use case and running it in a loop with ASAN enabled. You can then reduce the scope (for example to a C++ unit test) while continuing to run in a loop. Eventually the stack traces should make it clear why the leak occurs.

Running in CLion

If you prefer debugging in an IDE, or you wish to break on ASAN errors you can use CLion. You'll need to install all the required dependencies manually (but you can reference the CI Dockerfiles for help). On Ubunut 16.04 this is as simple as installing gcc-8, and setting it as the project compiler in CLion. You can then add the -DUSE_ASAN build flag for your project to enable ASAN support. Finally you must add LD_PRELOAD to your run environment variables for your launch target and point it at the version of ASAN you have installed (libasan.so.5 for gcc8).

Disabling Memory Leak Detection

...

Page tree

Versions Compared

Old Version 3

New Version 4

Key

Running in CLion

Disabling Memory Leak Detection