Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Using ASAN builds with MXNet

CI reports are nice, but it's sometimes more useful to build an ASAN build locally and to run some specific sections of code you're afraid may be leaking.  This is easy to do with MXNet.  We've installed all the prerequisites in or CI build Dockerfiles, so we can use docker to build ASAN builds without having to install or configure dependencies.  To build a CPU build with ASAN run the following commands in a new folder:


Code Block
git clone --recurse https://github.com/apache/incubator-mxnet.git
cd incubator-mxnet/ci
# Build our dockerfile with all required deps for ASAN
docker build -f docker/Dockerfile.build.ubuntu_cpu -t mxnetci/build.ubuntu_cpu docker
cd ..
mkdir -p build
# Build an ASAN instrumented MXNet library
# Privileged probably not required for all steps, but in general ASAN requires some capabilities in order to inspect process memory.
docker run --privileged -v `pwd`:/work/mxnet -v `pwd`/build:/work/build  -ti mxnetci/build.ubuntu_cpu /work/runtime_functions.sh build_ubuntu_cpu_cmake_asan
# Choose an example of something you'd like to test with ASAN.  This could be a specific python test we want to run in a loop, it could be a C++ unit test written to expose leaks, etc.

# In our case we use a small Gluon python tests as an example.
# First we'll enter our container, and then we'll run tests within the container.
docker run --privileged -v `pwd`:/work/mxnet -v `pwd`/build:/work/build  -ti mxnetci/build.ubuntu_cpu bash

# Now within the container:
export PYTHONPATH=./python/
export MXNET_MKLDNN_DEBUG=1  # Ignored if not present
export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0
# Feel free to export any ASAN options via export ASAN_OPTIONS=...

# Importantly we need to make sure ASAN is the first library loaded (before our other libraries have a chance to allocate memory)
# To do this we add the library to the library preload list
export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libasan.so.5
nosetests-3.4 --verbose tests/python/unittest/test_rnn.py


Disabling Memory Leak Detection

...