...
MXNet-Scala is a Scala binding for the MXNet Deep Learning framework. It currently supports Training and Inference
MXNet-Scala Modules
Module | Package Generated | Dependencies | Description |
---|---|---|---|
|
|
| |
2. core | mxnet-core_2.11 |
| |
3. macros | mxnet-macros_2.11 |
| |
4. init | mxnet-init_2.11 |
...
5. init-native |
| mxnet-init_2.11 | |
6. native |
| mxnet-core_2.11 | |
7. infer | mxnet-infer_2.11 |
| |
8. spark | mxnet-spark_2.11 |
| |
9. examples | mxnet-examples_2.11 |
|
Publishing Scala Packages to Nexus/Maven
Currently 3 Packages from the above generated are published to Maven. `mxnet_full_2.11-{platform}`.
The following flags are used to build the MXNet Backend(derived from mxnet pip compiler flags)
...
MXNet-Scala Linux CPU
...
USE_CUDNN
...
1
...
$ java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
...
ubuntu@ip:~$ java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (Zulu 8.20.0.5-linux64) (build 1.8.0_121-b15)
OpenJDK 64-Bit Server VM (Zulu 8.20.0.5-linux64) (build 25.121-b15, mixed mode)
Export GPG Keys
Code Block | ||
---|---|---|
| ||
gpg --export-secret-keys -a xxx@apache.org > ~/Downloads/keys.asc
gpg --import /tmp/keys.asc
#list the keys to make sure they are there.
gpg --list-keys |
Build MXNet backend on OSX
Clone MXNet repo and checkout the release branch
Code Block | ||
---|---|---|
| ||
export YOURFORK=whatever
export RELEASE_VERISON=1.2.0
git clone https://github.com/$YOURFORK$/incubator-mxnet
git remote add upstream https://github.com/apache/incubator-mxnet
git fetch upstream $RELEASE_VERSION && git checkout -b $RELEASE_VERSION --track upstream/$RELEASE_VERSION |
Install necessary libraries
Code Block | ||
---|---|---|
| ||
brew install lapack
brew install openblas |
Build MXNet Backend
Code Block | ||
---|---|---|
| ||
brew install lapackbrew install openblas |
Maven Publish
Follow Maven Publish Link https://www.apache.org/dev/publishing-maven-artifacts.html to setup repo settings, etc., before you proceed
Code Block | ||
---|---|---|
| ||
make scalarelease-dryrun
# Releases to Apache Snapshot
make scaladeploy |
Build MXNet backend on Linux-CPU
To build for linux-cpu platform, we use the AWS Deep Learning Base AMI that comes built with all the necessary
libraries such as opencv/cudnn etc,.
Install Maven on Ubuntu
Code Block | ||
---|---|---|
| ||
sudo apt-get install -y maven |
setup all the flags
Note: compiling with USE_SIGNAL_HANDLER throws a segmentation fault on both OSX and Linux platforms
Code Block | ||||
---|---|---|---|---|
| ||||
[INFO] Stack trace returned 10 entries: [INFO] [bt] (0) /home/ubuntu/incubator-mxnet/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(dmlc::StackTrace[abi:cxx11]()+0x1bc) [0x7f0afcea735c] [INFO] [bt] (1) /home/ubuntu/incubator-mxnet/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(+0x2fc40af) [0x7f0aff99e0af] [INFO] [bt] (2) /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7f0be3a204b0] [INFO] [bt] (3) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x8797bd) [0x7f0be0fed7bd] [INFO] [bt] (4) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3dac73) [0x7f0be0b4ec73] [INFO] [bt] (5) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x40594d) [0x7f0be0b7994d] [INFO] [bt] (6) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3ed3a0) [0x7f0be0b613a0] [INFO] [bt] (7) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x4047ac) [0x7f0be0b787ac] [INFO] [bt] (8) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x4032ce) [0x7f0be0b772ce] [INFO] [bt] (9) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3de959) [0x7f0be0b52959] [INFO] Stack trace returned 10 entries: [INFO] [bt] (0) /home/ubuntu/incubator-mxnet/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(dmlc::StackTrace[abi:cxx11]()+0x1bc) [0x7f0afcea735c] [INFO] [bt] (1) /home/ubuntu/incubator-mxnet/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(+0x2fc40af) [0x7f0aff99e0af] [INFO] [bt] (2) /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7f0be3a204b0] [INFO] [bt] (3) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3dfe3a) [0x7f0be0b53e3a] [INFO] [bt] (4) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x323ac4) [0x7f0be0a97ac4] [INFO] [bt] (5) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x323e12) [0x7f0be0a97e12] [INFO] [bt] (6) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x324b07) [0x7f0be0a98b07] [INFO] [bt] (7) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x4768db) [0x7f0be0bea8db] [INFO] [bt] (8) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x478717) [0x7f0be0bec717] [INFO] [bt] (9) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0xa12217) [0x7f0be1186217] |
Testing with Intel MKL
Install instructions here:
https://software.intel.com/en-us/articles/installing-intel-free-libs-and-python-apt-repo
Code Block | ||||
---|---|---|---|---|
| ||||
USE_BLAS=mkl wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB sudo apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB sudo wget https://apt.repos.intel.com/setup/intelproducts.list -O /etc/apt/sources.list.d/intelproducts.list sudo sh -c 'echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list' sudo apt-get update sudo apt-get install intel-mkl-2018.2-046 |
Failed with Segfault:
Code Block |
---|
[INFO] Stack trace returned 10 entries: [INFO] [bt] (0) /home/ubuntu/mxnet-master/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(dmlc::StackTrace[abi:cxx11]()+0x1bc) [0x7f2f04ca58ec] [INFO] [bt] (1) /home/ubuntu/mxnet-master/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(+0x31d7a4f) [0x7f2f07971a4f] [INFO] [bt] (2) /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7f3096cd24b0] [INFO] [bt] (3) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3e4afc) [0x7f3093e0aafc] [INFO] [bt] (4) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0xa239d6) [0x7f30944499d6] [INFO] [bt] (5) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0xa24cdc) [0x7f309444acdc] [INFO] [bt] (6) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0xa24e4c) [0x7f309444ae4c] [INFO] [bt] (7) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x7c3252) [0x7f30941e9252] [INFO] [bt] (8) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x5b00d6) [0x7f3093fd60d6] [INFO] [bt] (9) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x5b2c44) [0x7f3093fd8c44] [INFO] ------------------------------------------------------------------------ |
Publishing to Maven
Code Block | ||
---|---|---|
| ||
export GPG_TTY=$(tty) make scalapkg make scalarelease-dryrun |
...