Overview

In recently performance testing, we uncovered some performance discrepancy between Scala package and python package. And a JIRA issue was created to track the issue: https://issues.apache.org/jira/browse/MXNET-1224

Issues in Scala maven build

• Scala release is a manual process, the jni library linked to Linear Algebra library might be different from MXNet released shared library (libmxnet.so). This different causes performance discrepancy between Scala and python release.
• Scala maven build depends on make build flags, missing required build parameters will generate different (may unexpected) jni library.
• Scala maven build should be able to run standalone: mvn package, currently Scala build has to use: make scalapkg
• Scala maven build should automatically detect platform and generate corresponding artifacts.
• Scala build should support incremental build, current maven build rebuild everything on each mvn package
• unittest and integrationtest are not implemented correctly. Current Scala project doesn't have unittest class ( class name must end with Test”, all the test class can only be executed with mvn integration-test. All the test are disabled by default, and has use mvn -Punittest or mvn -Pintegrationtest to activate the test run.
• for each submodule, run “mvn package” doesn't work properly. Only top level maven build works.
• the current directory must be in scala-package, otherwise mvn build will fail, this is due to poor implementation of org.apache.mxnet.init.Base class loading jni library using relative path to current directory.
• mvn clean doesn't clean up workspace, must use mvn clean -Plinux_x86-64-cpu,scala-2.11
• mvn clean doesn't delete some generated .scala code. e.g. src/main/scala/org/apache/mxnet/NDArrayAPIBase.scala
• The minimal Scala version support should not be an option, it should be carefully decided and hardcoded in the pom.xml file. User should be able to use scala-2.11 or scala-2.12 to build Scala project, but runtime requirement should be a fixed version.
• The Scala build plugin we are using is https://mvnrepository.com/artifact/org.scala-tools/maven-scala-plugin, which has not been updated since 2011, we should update to official scala-maven-plugin
• Invalid project dependency, mxnet-core should depend on libmxnet-scala, not libmxnet-scala depend on mxnet-core, the reversed dependency causes, mxnet-core project is not able run unittest during normal build, the project must use -DskipTests the build everything first, and then run mvn -Punittest
• scalastyle check isn't configured properly, code violation is missed from mxnet-core project
• Scala final jar files contains unnecessary scala runtime classes. Removing those classes can reduce around 25M in size.
• Scala final jar files doesn't contains top level pom.xml, mvn install:install-file -Dfile=scala.jar doesn't work.

Proposed solution

Decouple make build maven build

To make sure in sync with release MXNet library (libmxnet.so), scala jni will stop static linking libmxnet.a, instead dynamically link to libmxnet.so. Scala jni library become a thin wrapper around libmxnet.so.

Changes need to dynamically link to libmxnet.so:

• Use -lmxnet to dynamically link libmxnet.so
• make sure libmxnet-scala.so can load libmxnet.so from the same directory:
• Linux, use “-Wl,-rpath,”$ORGINAL” as link flag
• OSX, use install_name_tool to update loading path to: @loader_path/libmxnet.so
• Use fixed compile/link flags in pom.xml, this will make sure Scala build consistency.
• Extract libmxnet.so file from jar file to temp folder
• Pick up prebuilt libmxnet.so file and package it into artifact.

Packaging

Since we no longer static link with libmxnet.a, we no longer need to generate cpu/gpu flavor of Scala jni, the released package on maven central can be:

• libmxnet-scala-core (This will contain native jni library for all platform osx/linux)
• This package will not contain libmxnet.so, user can use this package if they installed libmxnet.so by themselves.
• libmxnet-scala-osx (This package will depends on libmxnet-scala-core, and contains only osx version of libmxnet.so)
• libmxnet-scala-linux-cpu (This package will depends on libmxnet-scala-core, and contains only linux cpu version of libmxnet.so)
• libmxnet-scala-linux-gpu (This package will depens on libmxnet-scala-core, and contains only linux gpu version of libmxnet.so)

Other build improvement

Auto detect platform and cpu/gpu package

Current Scala build doesn't support cross platform build. osx and linux build has to be build on each platform. We should be able to auto detect platform and generate correct artifact for corresponding platform. Further more, we should detect if bundled libmxnet.so is for gpu or cpu and name the package name properly.

With this change, we can consolidate linux_x86-64-cpu and linux_x86-64-gpu into single profile. For developer the only thing they need to do is: mvn package

Improve unittest and integrationtest

In current maven pom.xml file, we defined two profiles “unittest” and “integrationtest”. This is not the maven way of doing thing.

The right approach is:

• skipTests should not be turned off by default, developer can use mvn package -DskipTests to bypass the test or they can create their own profile to deactivate testing by default.
• unittest should be executed by default, in the unittest code, the class name should end with Test, and they will be picked up during mvn package
• integrationtest will not be executed by default, they can be executed with mvn integration-test

Better support for incremental build

• In the native build, javah will be executed every time. This causes the build think the header file is changed and run a full build each time. one of the possible workaround is checkin the generated header file, and introduce a static analysis check to see if the header file is changed.
• init-native project generates source code for native project, which defeats incremental build.

Proposal: See next section

Resolve cyclic module dependencies

Current there are cyclic module dependencies:

• init-native depends on init compiled classes to generate javah, init module's test depends on init-native's .so file
• native depends on core module's compiled classes for javah, core depends on macros generated .scala file, macros's java-doc depends on both core and native module.
• There are more tangle dependencies in current mvn project.

Proposal:

• check in generated javah header file as source code, this will break javah dependency.
• Move all test tasks into core module, move javadoc tasks into core module. Only core module depends on other modules. other modules won't have dependency on each other.