Page History

...

Thus, an application can include multiple OpenMP implementations. The explicitly built and linked, the one linked implicitly by the compiler and the one provided with mklml_intel.

As stated here:

Having more than one OpenMP runtime initialised may lead to undefined behaviour including incorrect results or crashes.

A discussion has been started on the dev list to review a possible solution to the problem.

Currently, we assume these issues might be realatedrelated: 

• Failed OpenMP assertion when loading MXNet compiled with DEBUG=1
  https://github.com/apache/incubator-mxnet/issues/10856
• libomp.so dependency (need REAL fix)
  https://github.com/apache/incubator-mxnet/issues/11417
• mxnet-mkl (v0.12.0) crash when using (conda-installed) numpy with MKL
  https://github.com/apache/incubator-mxnet/issues/8532
• Performance regression when OMP_NUM_THREADS environment variable is not set
  https://github.com/apache/incubator-mxnet/issues/9744
• Poor concat CPU performance on CUDA builds
  https://github.com/apache/incubator-mxnet/issues/11905
• Poor performance of the libmxnet if OMP_PLACES environment variable is present
  https://github.com/apache/incubator-mxnet/issues/14087

...

As of now (1/2019) we have 2 build systems: make and cmake. Current production binaries are delivered by make and the compiler optimization flags are more aggressive. CMake is under development, some settings are behind that of Make (like sse2 vs sse3). More than that, current cmake produces critically slower binaries. See: 

• !!!GPU performance of Cmake built mxnet is worse than Make built one
  https://github.com/apache/incubator-mxnet/issues/6685

One of the reasons with CPU (e.g. not CUDA version) is the OpenBLAS preceding MKL ML in the linker commands. See: 

• Default cmake build uses openblas instead of MKL
  https://github.com/apache/incubator-mxnet/issues/14085

...

Currently, there are several problems with MXNet compilation if compiled with ICC (Intel C++ Compiler). See: 

• Intel Compiler fails to build mxnet
  https://github.com/apache/incubator-mxnet/issues/14086

...

We use the current CMake, considering most of the deviating flags (like SSE or explicit loop unrolling) to be insignificant for our experiments.

cmake

...

 \
    -DUSE_CUDA=OFF

...

 \
    -DWITH_TESTS=OFF

...

 \ 
    -DWITH_EXAMPLES=OFF

...

 \
    -DCMAKE_CXX_COMPILER=$CXXCOMP

...

 \
    -DCMAKE_C_COMPILER=$CCOMP

...

 \
    -DMKLDNN_THREADING=$THREADING

...

 \
    $LD_ARG ..

See details in the attached benchmark.sh file.

...

Obviously, two factors contribute to the performance values: 

1. OpenMP implementation
2. Quality of generated machine code

...

Same behaviour we see in the treatment group no matter which OpenMP is used.

 Image Added

Control group

Treatment group shows no difference other than that "GCC-swing". Normalizing the data gives us average scores with ~1% difference, which is close to standard error.

Image Added 

Treatment group

ResNet152

Image AddedImage Added
Now we can observe a beautiful saturation of the throughput. Optimal batch size is between 16 and 32.

...

Very similar data we get for other models.

Total scores

Control group shows the following pretty close numbers: 

Combining the treatment group with clang7_omp which is the best performer (again, with devastating margin of 1%) of the control group we have the following data. 

...

But the overall differences are pretty close to standard error and don't even reach 2%.

faster-rcnn Benchmark

Image Added

As we can see, GOMP delivers ~3-5% worse performance than OMP. 

...