Page History

...

In recently performance testing, we uncovered some performance discrepancy between Scala package and python package. A COE is created: https://coe.amazon.com/coes/100087. And a JIRA issue is was created to track the issue: https://issues.apache.org/jira/browse/MXNET-1224

...

• Scala release is a manual process, the jni library linked to Linear Algebra library might be different from MXNet released shared library (libmxnet.so). This different causes performance discrepancy between Scala and python release.
• Scala maven build depends on make build flags, missing required build parameters will generate different (may unexpected) jni library.
• Scala maven build should be able to run standalone: mvn package, currently Scala build has to use: make scalapkg
• Scala maven build should automatically detect platform and generate corresponding artifacts.
• Scala build should support incremental build, current maven build rebuild everything on each mvn package
• unittest and integrationtest are not implemented correctly. Current Scala project doesn't have unittest class ( class name must end with Test”, all the test class can only be executed with mvn integration-test. All the test are disabled by default, and has use mvn -Punittest or mvn -Pintegrationtest to activate the test run.
• for each submodule, run “mvn package” doesn't work properly. Only top level maven build works.
• the current directory must be in scala-package, otherwise mvn build will fail, this is due to poor implementation of org.apache.mxnet.init.Base class loading jni library using relative path to current directory.
• mvn clean doesn't clean up workspace, must use mvn clean -Plinux_x86-64-cpu,scala-2.11
• mvn clean doesn't delete some generated .scala code. e.g. src/main/scala/org/apache/mxnet/NDArrayAPIBase.scala
• The minimal Scala version support should not be an option, it should be carefully decided and hardcoded in the pom.xml file. User should be able to use scala-2.11 or scala-2.12 to build Scala project, but runtime requirement should be a fixed version.
• The Scala build plugin we are using is https://mvnrepository.com/artifact/org.scala-tools/maven-scala-plugin, which has not been updated since 2011, we should update to official scala-maven-plugin
• Invalid project dependency, mxnet-core should depend on libmxnet-scala, not libmxnet-scala depend on mxnet-core, the reversed dependency causes, mxnet-core project is not able run unittest during normal build, the project must use -DskipTests the build everything first, and then run mvn -Punittest
• scalastyle check isn't configured properly, code violation is missed from mxnet-core project
• Scala final jar files contains unnecessary scala runtime classes. Removing those classes can reduce around 25M in size.
• Scala final jar files doesn't contains top level pom.xml, mvn install:install-file -Dfile=scala.jar doesn't work.

Proposed solution

Decouple make build maven build

...