Page History

...

• Scala release is a manual process, the jni library linked to Linear Algebra library might be different from MXNet released shared library (libmxnet.so). This different causes performance discrepancy between Scala and python release.
• Scala maven build depends on make build flags, missing required build parameters will generate different (may unexpected) jni library.
• Scala maven build should be able to run standalone: mvn package, currently Scala build has to use: make scalapkg
• Scala maven build should automatically detect platform and generate corresponding artifacts.
• Scala build should support incremental build, current maven build rebuild everything on each mvn package
• unittest and integrationtest are not implemented correctly. Current Scala project doesn't have unittest class ( class name must end with Test”, all the test class can only be executed with mvn integration-test. All the test are disabled by default, and has use mvn -Punittest or mvn -Pintegrationtest to activate the test run.
• for each submodule, run “mvn package” doesn't work properly. Only top level maven build works.
• the current directory must be in scala-package, otherwise mvn build will fail, this is due to poor implementation of org.apache.mxnet.init.Base class loading jni library using relative path to current directory.
• mvn clean doesn't clean up workspace, must use mvn clean -Plinux_x86-64-cpu,scala-2.11
• mvn clean doesn't delete some generated .scala code. e.g. src/main/scala/org/apache/mxnet/NDArrayAPIBase.scala
• The minimal Scala version support should not be an option, it should be carefully decided and hardcoded in the pom.xml file. User should be able to use scala-2.11 or scala-2.12 to build Scala project, but runtime requirement should be a fixed version.
• The Scala build plugin we are using is https://mvnrepository.com/artifact/org.scala-tools/maven-scala-plugin, which has not been updated since 2011, we should update to official scala-maven-plugin
• Invalid project dependency, mxnet-core should depend on libmxnet-scala, not libmxnet-scala depend on mxnet-core, the reversed dependency causes, mxnet-core project is not able run unittest during normal build, the project must use -DskipTests the build everything first, and then run mvn -Punittest
• scalastyle check isn't configured properly, code violation is missed from mxnet-core project

...

Current Scala build doesn't support cross platform build. osx and linux build has to be build on each platform. We should be able to auto detect platform and generate correct artifact for corresponding platform. Further more, we should detect if bundled libmxnet.so is for gpu or cpu and name the package name properly.

With this change, we can consolidate linux_x86-64-cpu and linux_x86-64-gpu into single profile. For developer the only thing they need to do is: mvn package

Improve unittest and integrationtest

...

• skipTests should not be turned off by default, developer can use mvn package -DskipTests to bypass the test or they can create their own profile to deactivate testing by default.
• unittest should be executed by default, in the unittest code, the class name should end with Test, and they will be picked up during mvn package
• integrationtest will not be executed by default, they can be executed with mvn integration-test

Better support for incremental build

...

• In the native build, javah will be executed every time. This causes the build think the header file is changed and run a full build each time. one of the possible workaround is checkin the generated header file, and introduce a static analysis check to see if the header file is changed.
• init-native project generates source code for native project, which defeats incremental build.

Proposal: See next section

Resolve cyclic module dependencies

...

• init-native depends on init compiled classes to generate javah, init module's test depends on init-native's .so file
• native depends on core module's compiled classes for javah, core depends on macros generated .scala file, macros's java-doc depends on both core and native module.
• There are more tangle dependencies in current mvn project.

Proposal:

• check in generated javah header file as source code, this will break javah dependency.
• Move all test tasks into core module, move javadoc tasks into core module. Only core module depends on other modules. other modules won't have dependency on each other.