THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
A build failure like shown below can be reproduced by copying the failed command, starting which starts with ci/build.py, and running it on your local machine while being in the root of your mxnet source directory. This step does NOT require a GPU, nor CUDA dependencies.
| Info |
|---|
In the most recent version of ci, the build command is not displayed in the tab as in the image above. To find it, simply click on show complete log and scroll to the top. |
In this case, you would like to run ci/build.py --platform ubuntu_build_cuda /work/runtime_functions.sh build_ubuntu_gpu_cuda8_cudnn5, which would produce an output like the following image:
...
First we have to generate these dependecies before a test can be executed. These can be resolved by the stash commands, which are indicated by the message "Restore files previously stashed".
| Info |
|---|
Files ending with the suffix _gcov_data are used for test coverage reporting, and are therefore not needed to reproduce test results. |
In this case, the stash is labelled as mkldnn_gpu. The easiest way to map this to a build-step, is by opening the corresponding Jenkinsfile and searching for pack_lib('mkldnn_gpu' In this case, you will find a block like the following:
| Code Block | ||||
|---|---|---|---|---|
| ||||
def compile_unix_mkldnn_gpu() { |
...
return ['GPU: MKLDNN': { |
...
node(NODE_LINUX_CPU) { |
...
ws('workspace/build-mkldnn-gpu') { |
...
timeout(time: max_time, unit: 'MINUTES') { |
...
utils.init_git() |
...
utils.docker_run('ubuntu_build_cuda', 'build_ubuntu_gpu_mkldnn', false) |
...
utils.pack_lib('mkldnn_gpu', mx_mkldnn_lib, true) |
...
} |
...
} |
...
} |
...
}] |
...
} |
...
The important code here is line 7, which contains the three arguments we need to use to build our dependencies. Substitute them in the command below and run it to obtain them.
- PLATFORM: in this case, ubuntu_build_cuda
- FUNCTION_NAME: in this case, build_ubuntu_gpu_mkldnn
- USE_NVIDIA: this argument toggles whether the --nvidiadocker should be used. Here, it is false, meaning <USE_NVIDIA> below should be removed. On the contrary, if set to TRUE, one would replace it with --nvidiadocker.
| Code Block | ||||
|---|---|---|---|---|
| ||||
ci/build.py --docker-registry mxnetci <USE_NVIDIA> -p <PLATFORM> /work/runtime_functions.sh <FUNCTION_NAME> |
Test execution
After the binaries have been generated successfully, please take the failed command from the screenshot above and execute it in the root of your MXNet workspace. To get such command, you need to expand the Shell Script tab and view the complete log. Scrolling to the top will show the command we are interested in:
In this case, you would like to run run:
| Code Block | ||||
|---|---|---|---|---|
| ||||
ci/build. |
...
py --docker-registry mxnetci --nvidiadocker --platform ubuntu_gpu --docker-build-retries 3 --shm-size 500m /work/runtime_functions.sh unittest_ubuntu_python2_gpu |
. Please note the parameter --nvidiadocker in this example. This indicates that this test requires a GPU and is thus only executable on a Ubuntu machine with Nvidia-Docker and a GPU installed. The result of this execution should look like follows:
Tips and Tricks
Repeating test execution
...