Page History

...

This CI task will inform users primarily about memory leaks.  To reduce output we are running a limited test, and are avoiding the use of python.  Currently this means we're running training and detection on a MNIST network based on MLPs.  To do this we're using an ASAN instrumented MXNet library and cpp package executable.  Failures for this CI task will be displayed in Jenkins logs, but will not fail the build.  The output is informational only, but could be used to help us bisect in the future (for example if we're investigating which commit may have introduced a new leak).

Example Output

...
+ ./mlp_cpu
[10:48:23] /work/mxnet/src/io/iter_mnist.cc:110: MNISTIter: load 60000 images, shuffle=1, shape=(100,784)
[10:48:24] /work/mxnet/src/io/iter_mnist.cc:110: MNISTIter: load 10000 images, shuffle=1, shape=(100,784)
[10:48:26] /work/mxnet/cpp-package/example/mlp_cpu.cpp:134: Epoch: 0 30241.9 samples/sec Accuracy: 0.1135
[10:48:28] /work/mxnet/cpp-package/example/mlp_cpu.cpp:134: Epoch: 1 32715.4 samples/sec Accuracy: 0.5632
[10:48:30] /work/mxnet/cpp-package/example/mlp_cpu.cpp:134: Epoch: 2 32573.3 samples/sec Accuracy: 0.8454
[10:48:32] /work/mxnet/cpp-package/example/mlp_cpu.cpp:134: Epoch: 3 32397.4 samples/sec Accuracy: 0.8792
[10:48:34] /work/mxnet/cpp-package/example/mlp_cpu.cpp:134: Epoch: 4 32362.5 samples/sec Accuracy: 0.9125
[10:48:36] /work/mxnet/cpp-package/example/mlp_cpu.cpp:134: Epoch: 5 32345 samples/sec Accuracy: 0.9245
[10:48:38] /work/mxnet/cpp-package/example/mlp_cpu.cpp:134: Epoch: 6 32591 samples/sec Accuracy: 0.9309
[10:48:40] /work/mxnet/cpp-package/example/mlp_cpu.cpp:134: Epoch: 7 32485.1 samples/sec Accuracy: 0.9362
[10:48:42] /work/mxnet/cpp-package/example/mlp_cpu.cpp:134: Epoch: 8 32591 samples/sec Accuracy: 0.9392
[10:48:44] /work/mxnet/cpp-package/example/mlp_cpu.cpp:134: Epoch: 9 32573.3 samples/sec Accuracy: 0.9417

=================================================================
==34==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 12560000 byte(s) in 80 object(s) allocated from:
    #0 0x7fedd84c7980 in __interceptor_posix_memalign (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xee980)
    #1 0x159677c in mxnet::storage::CPUDeviceStorage::Alloc(unsigned long) (/work/mxnet/build/cpp-package/example/mlp_cpu+0x159677c)
    #2 0x15a4faa in mxnet::storage::NaiveStorageManager<mxnet::storage::CPUDeviceStorage>::Alloc(mxnet::Storage::Handle*) /work/mxnet/src/storage/./naive_storage_manager.h:61
    #3 0x15913c8 in mxnet::StorageImpl::Alloc(mxnet::Storage::Handle*) /work/mxnet/src/storage/storage.cc:147
    #4 0x14250eb in mxnet::Storage::Alloc(unsigned long, mxnet::Context) /work/mxnet/include/mxnet/./storage.h:70
    #5 0x14f631f in mxnet::NDArray::Chunk::CheckAndAlloc() /work/mxnet/include/mxnet/ndarray.h:882
    #6 0x166fe69 in mxnet::NDArray::Chunk::Chunk(nnvm::TShape, mxnet::Context, bool, int) /work/mxnet/include/mxnet/ndarray.h:784
    #7 0x167fa79 in void __gnu_cxx::new_allocator<mxnet::NDArray::Chunk>::construct<mxnet::NDArray::Chunk, nnvm::TShape const&, mxnet::Context&, bool&, int&>(mxnet::NDArray::Chunk*, nnvm::TShape const&, mxnet::Context&, bool&, int&) /usr/include/c++/8/ext/new_allocator.h:136
    #8 0x167f1cd in void std::allocator_traits<std::allocator<mxnet::NDArray::Chunk> >::construct<mxnet::NDArray::Chunk, nnvm::TShape const&, mxnet::Context&, bool&, int&>(std::allocator<mxnet::NDArray::Chunk>&, mxnet::NDArray::Chunk*, nnvm::TShape const&, mxnet::Context&, bool&, int&) /usr/include/c++/8/bits/alloc_traits.h:475
    #9 0x167e739 in std::_Sp_counted_ptr_inplace<mxnet::NDArray::Chunk, std::allocator<mxnet::NDArray::Chunk>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<nnvm::TShape const&, mxnet::Context&, bool&, int&>(std::allocator<mxnet::NDArray::Chunk>, nnvm::TShape const&, mxnet::Context&, bool&, int&) /usr/include/c++/8/bits/shared_ptr_base.h:549
    #10 0x167cdb8 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<mxnet::NDArray::Chunk, std::allocator<mxnet::NDArray::Chunk>, nnvm::TShape const&, mxnet::Context&, bool&, int&>(std::_Sp_make_shared_tag, mxnet::NDArray::Chunk*, std::allocator<mxnet::NDArray::Chunk> const&, nnvm::TShape const&, mxnet::Context&, bool&, int&) /usr/include/c++/8/bits/shared_ptr_base.h:662
    #11 0x167ab03 in std::__shared_ptr<mxnet::NDArray::Chunk, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<mxnet::NDArray::Chunk>, nnvm::TShape const&, mxnet::Context&, bool&, int&>(std::_Sp_make_shared_tag, std::allocator<mxnet::NDArray::Chunk> const&, nnvm::TShape const&, mxnet::Context&, bool&, int&) /usr/include/c++/8/bits/shared_ptr_base.h:1328
    #12 0x1677cdc in std::shared_ptr<mxnet::NDArray::Chunk>::shared_ptr<std::allocator<mxnet::NDArray::Chunk>, nnvm::TShape const&, mxnet::Context&, bool&, int&>(std::_Sp_make_shared_tag, std::allocator<mxnet::NDArray::Chunk> const&, nnvm::TShape const&, mxnet::Context&, bool&, int&) /usr/include/c++/8/bits/shared_ptr.h:360
    #13 0x1674016 in std::shared_ptr<mxnet::NDArray::Chunk> std::allocate_shared<mxnet::NDArray::Chunk, std::allocator<mxnet::NDArray::Chunk>, nnvm::TShape const&, mxnet::Context&, bool&, int&>(std::allocator<mxnet::NDArray::Chunk> const&, nnvm::TShape const&, mxnet::Context&, bool&, int&) /usr/include/c++/8/bits/shared_ptr.h:707
    #14 0x16717b1 in std::shared_ptr<mxnet::NDArray::Chunk> std::make_shared<mxnet::NDArray::Chunk, nnvm::TShape const&, mxnet::Context&, bool&, int&>(nnvm::TShape const&, mxnet::Context&, bool&, int&) /usr/include/c++/8/bits/shared_ptr.h:723
    #15 0x166f2ca in mxnet::NDArray::NDArray(nnvm::TShape const&, mxnet::Context, bool, int) /work/mxnet/include/mxnet/ndarray.h:98
    #16 0x68de1e6 in mxnet::io::PrefetcherIter::Init(std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&)::{lambda(mxnet::DataBatch**)#1}::operator()(mxnet::DataBatch**) const (/work/mxnet/build/cpp-package/example/mlp_cpu+0x68de1e6)
    #17 0x69102e5 in std::_Function_handler<bool (mxnet::DataBatch**), mxnet::io::PrefetcherIter::Init(std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&)::{lambda(mxnet::DataBatch**)#1}>::_M_invoke(std::_Any_data const&, mxnet::DataBatch**&&) /usr/include/c++/8/bits/std_function.h:282
    #18 0x6910bc2 in std::function<bool (mxnet::DataBatch**)>::operator()(mxnet::DataBatch**) const /usr/include/c++/8/bits/std_function.h:687
    #19 0x6907121 in dmlc::ThreadedIter<mxnet::DataBatch>::Init(std::function<bool (mxnet::DataBatch**)>, std::function<void ()>)::{lambda()#1}::operator()() const /work/mxnet/3rdparty/dmlc-core/include/dmlc/threadediter.h:357
    #20 0x6914e3c in void std::__invoke_impl<void, dmlc::ThreadedIter<mxnet::DataBatch>::Init(std::function<bool (mxnet::DataBatch**)>, std::function<void ()>)::{lambda()#1}>(std::__invoke_other, dmlc::ThreadedIter<mxnet::DataBatch>::Init(std::function<bool (mxnet::DataBatch**)>, std::function<void ()>)::{lambda()#1}&&) /usr/include/c++/8/bits/invoke.h:60
    #21 0x6910e12 in std::__invoke_result<dmlc::ThreadedIter<mxnet::DataBatch>::Init(std::function<bool (mxnet::DataBatch**)>, std::function<void ()>)::{lambda()#1}>::type std::__invoke<dmlc::ThreadedIter<mxnet::DataBatch>::Init(std::function<bool (mxnet::DataBatch**)>, std::function<void ()>)::{lambda()#1}>(std::__invoke_result&&, (dmlc::ThreadedIter<mxnet::DataBatch>::Init(std::function<bool (mxnet::DataBatch**)>, std::function<void ()>)::{lambda()#1}&&)...) /usr/include/c++/8/bits/invoke.h:95
    #22 0x691f21f in decltype (__invoke((_S_declval<0ul>)())) std::thread::_Invoker<std::tuple<dmlc::ThreadedIter<mxnet::DataBatch>::Init(std::function<bool (mxnet::DataBatch**)>, std::function<void ()>)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/include/c++/8/thread:234
    #23 0x691f1ac in std::thread::_Invoker<std::tuple<dmlc::ThreadedIter<mxnet::DataBatch>::Init(std::function<bool (mxnet::DataBatch**)>, std::function<void ()>)::{lambda()#1}> >::operator()() /usr/include/c++/8/thread:243
    #24 0x691ee15 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<dmlc::ThreadedIter<mxnet::DataBatch>::Init(std::function<bool (mxnet::DataBatch**)>, std::function<void ()>)::{lambda()#1}> > >::_M_run() /usr/include/c++/8/thread:186
    #25 0x7fedd5c5153e  (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xbd53e)

...

Direct leak of 3560 byte(s) in 1 object(s) allocated from:
    #0 0x7fedd84c8970 in operator new[](unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xef970)
    #1 0x1792b24 in mxnet::profiler::Profiler::Profiler() /work/mxnet/src/profiler/profiler.cc:70
    #2 0x17b1b20 in void __gnu_cxx::new_allocator<mxnet::profiler::Profiler>::construct<mxnet::profiler::Profiler>(mxnet::profiler::Profiler*) /usr/include/c++/8/ext/new_allocator.h:136
    #3 0x17b0bb4 in void std::allocator_traits<std::allocator<mxnet::profiler::Profiler> >::construct<mxnet::profiler::Profiler>(std::allocator<mxnet::profiler::Profiler>&, mxnet::profiler::Profiler*) /usr/include/c++/8/bits/alloc_traits.h:475
    #4 0x17af6f5 in std::_Sp_counted_ptr_inplace<mxnet::profiler::Profiler, std::allocator<mxnet::profiler::Profiler>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<>(std::allocator<mxnet::profiler::Profiler>) (/work/mxnet/build/cpp-package/example/mlp_cpu+0x17af6f5)
    #5 0x17ad0ca in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<mxnet::profiler::Profiler, std::allocator<mxnet::profiler::Profiler>>(std::_Sp_make_shared_tag, mxnet::profiler::Profiler*, std::allocator<mxnet::profiler::Profiler> const&) (/work/mxnet/build/cpp-package/example/mlp_cpu+0x17ad0ca)
    #6 0x17aa6bb in std::__shared_ptr<mxnet::profiler::Profiler, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<mxnet::profiler::Profiler>>(std::_Sp_make_shared_tag, std::allocator<mxnet::profiler::Profiler> const&) (/work/mxnet/build/cpp-package/example/mlp_cpu+0x17aa6bb)
    #7 0x17a70ab in std::shared_ptr<mxnet::profiler::Profiler>::shared_ptr<std::allocator<mxnet::profiler::Profiler>>(std::_Sp_make_shared_tag, std::allocator<mxnet::profiler::Profiler> const&) /usr/include/c++/8/bits/shared_ptr.h:360
    #8 0x17a4112 in std::shared_ptr<mxnet::profiler::Profiler> std::allocate_shared<mxnet::profiler::Profiler, std::allocator<mxnet::profiler::Profiler>>(std::allocator<mxnet::profiler::Profiler> const&) /usr/include/c++/8/bits/shared_ptr.h:707
    #9 0x17a091d in std::shared_ptr<mxnet::profiler::Profiler> std::make_shared<mxnet::profiler::Profiler>() /usr/include/c++/8/bits/shared_ptr.h:723
    #10 0x1793815 in mxnet::profiler::Profiler::Get(std::shared_ptr<mxnet::profiler::Profiler>*) /work/mxnet/src/profiler/profiler.cc:106
    #11 0x15d7a7c in mxnet::on_enter_api(char const*) /work/mxnet/src/c_api/c_api_profile.cc:123
    #12 0x1686f64 in MXListDataIters /work/mxnet/src/c_api/c_api.cc:683
    #13 0x13d9b28 in mxnet::cpp::MXDataIterMap::MXDataIterMap() (/work/mxnet/build/cpp-package/example/mlp_cpu+0x13d9b28)
    #14 0x13da6f2 in mxnet::cpp::MXDataIter::mxdataiter_map() (/work/mxnet/build/cpp-package/example/mlp_cpu+0x13da6f2)
    #15 0x13da868 in mxnet::cpp::MXDataIter::MXDataIter(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/work/mxnet/build/cpp-package/example/mlp_cpu+0x13da868)
    #16 0x13b1e13 in main /work/mxnet/cpp-package/example/mlp_cpu.cpp:65
    #17 0x7fedd52c982f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

...

SUMMARY: AddressSanitizer: 22682224 byte(s) leaked in 272 allocation(s).

...

Python Tests Without Leak Detection

...

git clone --recurse https://github.com/apache/incubator-mxnet.git
cd incubator-mxnet/ci
# Build our dockerfile with all required deps for ASAN
docker build -f docker/Dockerfile.build.ubuntu_cpu -t mxnetci/build.ubuntu_cpu docker
cd ..
mkdir -p build
# Build an ASAN instrumented MXNet library
# Privileged probably not required for all steps, but in general ASAN requires some capabilities in order to inspect process memory.
docker run --privileged -v `pwd`:/work/mxnet -v `pwd`/build:/work/build  -ti mxnetci/build.ubuntu_cpu /work/runtime_functions.sh build_ubuntu_cpu_cmake_asan

# ChooseNow choose an example of something you'd like to test with ASAN.  This could be a specific python test we want to run in a loop, it could be a C++ unit test written to expose leaks, etc.

# In our case we use a small Gluon python tests as an example.
# First we'll enter our container, and then we'll run tests within the container.
docker run --privileged -v `pwd`:/work/mxnet -v `pwd`/build:/work/build  -ti mxnetci/build.ubuntu_cpu bash

# Now within the container:
export PYTHONPATH=./python/
export MXNET_MKLDNN_DEBUG=1  # Ignored if not present
export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0
# Feel free to export any ASAN options via export ASAN_OPTIONS=...

# Importantly we need to make sure ASAN is the first library loaded (before our other libraries have a chance to allocate memory)
# To do this we add the library to the library preload list
export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libasan.so.5
nosetests-3.4 --verbose tests/python/unittest/test_rnn.py

...