See also: development guide.
Objective
This document helps you to get the development setup on Mac for working with Apache MXNet codebase. This document covers installing all required tools and packages, IDEs, debugging tools and is intended for developers who wants to dive into MXNet codebase, develop, debug, test and contribute.
...
If not already installed, download and install Xcode (or insall install it from the App Store) for macOS. Xcode is an integrated development environment for macOS containing a suite of software development tools like C/C++ compilers, BLAS library and more.
Note: You can also just install XCode Command Line Tools for this setup.
xcode-select --install
1.2 Install brew
Refer here for the steps - https://brew.sh/
...
- Open VSCode
- Go to View → Extensions
- In the search bar type - “C/C++”.
- Install extension “C/C++ IntelliSense, debugging, and code browsing” by Microsoft.
(Some developers have reported bugs using this extension on MacOS Yosemite. An alternative is to install the extension "CodeLLDB" by Vadim Chugunov)
- Install extension “C/C++ IntelliSense, debugging, and code browsing” by Microsoft.
- In the search bar type - “Python“CodeLLDB”.
- Install extension CodeLLDB. (There are some issues on debugging with macos LLDB, see this )
- In the search bar type - “Python”.
- Install extension “Python extension - “Python extension - Linting, Debugging (multi-threaded, remote), Intellisense, code formatting...” by Microsoft.
- In the search bar type - “CudaCPP”.
- Install extension “vscode-cudacpp - CUDA C++ language support for Visual Studio Code” by kriegalex.
1.6 Install Anaconda
Option 1 - Use Anaconda graphical installer from https://www.anaconda.com/download/#macos and click on the package to follow through the installation instructions.
...
brew update
brew tap homebrew/core
brew install pkgconfig
brew install opencv
brew install graphviz
pip install graphviz
pip install opencv-python
pip install nosepytest
2.4 Build MXNet core engine(After MXNet 1.6 release)
cp makeconfig/osxdarwin.mk ./config.mk
vi config.mk# Set/update USE_PROFILER, DEBUG compile flags with below values and leave the rest as defaults.
# This setting is required for enabling handles for developers.
# Enable profiling
USE_PROFILER = 1
# whether compile with debug
DEBUG = 1
# choose the version of blas you want to use
# can be: mkl, blas, atlas, openblas
USE_BLAS = apple
# Build MXNet (-j8 => use 8 threads to compile and build)
# Optimal number of threads is typically the number of cores available on your Mac. To get the number of cores available, use this command: "sysctl -n hw.ncpu"
make -j8
cmake config.cmake
mkdir build; cd build
CC=clang-10 CXX=clang++-10 CXXFLAGS='-fstandalone-debug' cmake -GNinja -DCMAKE_BUILD_TYPE=Debug -DUSE_CUDA=0 ..; ninja
Or with optimization
CC=clang-10 CXX=clang++-10 CXXFLAGS='-fstandalone-debug' cmake -GNinja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DUSE_CUDA=0 ..; ninja
Note: the exact versions of `clang` and `clang++` may be different on your computer. For example, this is an example of a command that currently works on MacOS Catalina after installing xcode
CC=clang CXX=clang++ CXXFLAGS='-fstandalone-debug' cmake -GNinja -DCMAKE_BUILD_TYPE=Debug -DUSE_CUDA=0 ..; ninja
To clean any previous build, use:
make cleanrm -rf build
2.5 Build MXNet Python binding
...
Typically, you implement the core logic in CPP, write python binding and write a python based test case to verify. Majority of MXNet python tests uses 'nosetests' with an exception of few in tests uses 'pytest'.
Let us run one single unit test for 'dot operator'
nosetests pytest tests/python/unittest/test_operator.py::test_dot
OR, you can run all operator tests
nosetests pytest tests/python/unittest/test_operator.py
OR, all unit tests
nosetests pytest tests/python/unittest/
Step 3 - Open MXNet codebase in VSCode - Develop and Debug
- File → Open → Select the MXNet directory.
- Setup VSCode for Debugging Python and C++ code
- View → Debug
- Click on the Current configuration drop-down (in screenshot below it says "No Configurations")Setup VSCode for Debugging Python and C++ code
- In the drop-down choose "Add Configuration"
- Choose "Python" as the configuration type
- And then you'll see the new configuration file "launch.json" with few default debug configurations. Delete all the contents and copy below provided launch.json.
...
Copy below snippet to 'launch.json' and save. We have 3 launch configurations below:
...
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "lldb Attach Debug",
"type": "cppdbglldb",
"request": "attach",
"program": "/anaconda3/bin/python",
"processId": "${command:pickProcess}",
"MIMode": "lldb"
},
{
"name": "lldb Launch Debug",
"type": "cppdbg",
"request": "launch",
"program": "${file}",
"args": [],
"stopAtEntry": false,
"cwd": "${workspaceFolder}",
"environment": [],
"externalConsole": true,
"MIMode": "lldb"
},
{
"name": "python Launch Debug",
"type": "python",
"request": "launch",
"stopOnEntry": false,
"pythonPath": "/anaconda3/bin/python",
"program": "${file}",
"cwd": "",
"console": "none",
"env": {},
"envFile": "${workspaceRoot}/.env"
},
]
}
...
Let us debug a unit test to see how we can debug a Mix of Python and C++ codebase.
- Open 'incubator-mxnet/tests/python/unittest/test_operator.py
- Go to test case - 'test_new_softmax()'. We will debug softmax operator. (Note: You can copy the softmax test function to the top of file to avoid waiting for other tests to pass)
- Add breakpoint. You can click to left of the line number where you want to add a breakpoint for debugging.
- Go to softmax implementation code - 'incubator-mxnet/src/operator/nn/softmax-inl.h'
- Add breakpoint in the softmax function - 'inline void Softmax(..)'
Now, let us debug.
Step 1: Start Python code in debug mode
- Go back to the test_operator.py file.
- Open Debug window: View→ Debug
- From the debug dropdown, select 'Python Launch Debug'. This was the debug configuration we had created in earlier steps. This will launch the current python file (test_operator.py) in debug mode.
At the top you can see the debugging controls. (Or, you can use shortcut commands, see appendix below)
Now, you should hit the debug point in your softmax test function like below.
Step 2: Start lldb debugger and attach to the python program being tested
From Debug dropdown, select 'lldb Attach Debug'. Select and start. Now, choose the 'test_operator.py' python process to attach.
You can now see that debug point is attached to the softmax CPP implementation code.
On the left, debug panel, you use 'python' and 'lldb' debugger to switch between
Now, you can continue execution, debug, inspect values and more.
You can also use Debug console to inspect and debug.
NOTE: If your C++ code debug point is not active, probably it means, symbols loaded in IDE is not the same that your python code is using. Most likely this will happen if you have pip installed mxnet and not compiled and build from mxnet source in the workspace. My pip list of search for MXNet looks like below, observe that MXNet python package is coming from my workspace.
...
Creating your first Pull Request
Follow instructions from this development guide - https://cwiki.apache.org/confluence/display/MXNET/MXNet+Development+Guide this Git Setup and Workflow
At high level below are the steps you do to contribute your changes:
...
- You can format an entire file with Format Document (⇧⌥F) or just the current selection with Format Selection (⌘K ⌘F) in right-click context menu.
- F10 → Step Over, F11 → Step Into, Shift+F11→ Step Out, F5 → Continue
...