Page History

...

MXNet-Scala is a Scala binding for the MXNet Deep Learning framework. It currently supports Training and Inference

MXNet-Scala Modules

Module	Package Generated	Dependencies
assembly	mxnet-full-parent_2.11 mxnet-full_2.11-{platform}	mxnet-core_2.11 libmxnet-scala-{platform}
2. core	mxnet-core_2.11	mxnet-init_2.11 mxnet-macros_2.11
3. macros	mxnet-macros_2.11	mxnet-init_2.11 libmxnet-init-scala-{platform}
4. init	mxnet-init_2.11

...


5. init-native	mxnet-scala-init-native-parent libmxnet-init-scala-osx-x86_64 libmxnet-init-scala-linux-x86_64	mxnet-init_2.11
6. native	mxnet-parent_2.11 libmxnet-scala-{platform}	mxnet-core_2.11
7. infer	mxnet-infer_2.11	mxnet-core_2.11 libmxnet-scala-{platform}
8. spark	mxnet-spark_2.11	mxnet-core_2.11 libmxnet-scala-{platform} spark-mllib_2.11
9. examples	mxnet-examples_2.11	mxnet-core_2.11 mxnet-infer_2.11

Publishing Scala Packages to Nexus/Maven

Currently 3 Packages from the above generated are published to Maven. `mxnet_full_2.11-{platform}`.

The following flags are used to build the MXNet Backend(derived from mxnet pip compiler flags)

...

MXNet-Scala Linux CPU

...

USE_CUDNN

...

1

...

$ java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

...

ubuntu@ip:~$ java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (Zulu 8.20.0.5-linux64) (build 1.8.0_121-b15)
OpenJDK 64-Bit Server VM (Zulu 8.20.0.5-linux64) (build 25.121-b15, mixed mode)

Setup GPG Keys

Build MXNet backend on OSX

Clone MXNet repo and checkout the release branch

Code Block

language	bash
collapse	true

https://github.com/apache/incubator-mxnet.git ~/$MXNET_LOC
git fetch all
git checkout -b $RELEASE_BRANCH
cp ~/$MXNET_LOC/make/config.mk ~/$MXNET_LOC

2. Install Lapack

Code Block

language	bash

brew install lapack

setup all the flags

Code Block

echo "ADD_CFLAGS  += -I/usr/local/opt/lapack/include" >> ~/$MXNET_LOC/config.mk
echo "ADD_LDFLAGS += -L/usr/local/opt/lapack/lib"     >> ~/$MXNET_LOC/config.mk
echo "USE_OPENMP = 0" >> ~/$MXNET_LOC/config.mk

Build MXNet backend on Linux-CPU

To build for linux-cpu platform, we use the AWS Deep Learning Base AMI that comes built with all the necessary

libraries such as opencv/cudnn etc,.

Install Maven on Ubuntu

Code Block

language	bash

sudo apt-get install -y maven

setup all the flags

Code Block

cd incubator-mxnet
cp make/config.mk ./ 
echo "ADD_CFLAGS += -I/usr/local/include/opencv" >> ./config.mk
echo "ADD_LDFLAGS += -L/usr/local/lib" >> ./config.mk
echo "USE_BLAS=openblas" >> ./config.mk 
echo "USE_SIGNAL_HANDLER=1" >> ./config.mk
echo "USE_DIST_KVSTORE=1" >> ./config.mk
echo "USE_S3=1" >> ./config.mk
echo "USE_OPENMP=1" >> ./config.mk

Note: compiling with USE_BLAS throws a segmentation fault on both OSX and Linux platforms

Code Block

language	bash
title	Segfaults when MXNet backend is built with openblas

[INFO] Stack trace returned 10 entries:
[INFO] [bt] (0) /home/ubuntu/incubator-mxnet/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(dmlc::StackTrace[abi:cxx11]()+0x1bc) [0x7f0afcea735c]
[INFO] [bt] (1) /home/ubuntu/incubator-mxnet/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(+0x2fc40af) [0x7f0aff99e0af]
[INFO] [bt] (2) /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7f0be3a204b0]
[INFO] [bt] (3) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x8797bd) [0x7f0be0fed7bd]
[INFO] [bt] (4) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3dac73) [0x7f0be0b4ec73]
[INFO] [bt] (5) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x40594d) [0x7f0be0b7994d]
[INFO] [bt] (6) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3ed3a0) [0x7f0be0b613a0]
[INFO] [bt] (7) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x4047ac) [0x7f0be0b787ac]
[INFO] [bt] (8) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x4032ce) [0x7f0be0b772ce]
[INFO] [bt] (9) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3de959) [0x7f0be0b52959]
[INFO] Stack trace returned 10 entries:
[INFO] [bt] (0) /home/ubuntu/incubator-mxnet/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(dmlc::StackTrace[abi:cxx11]()+0x1bc) [0x7f0afcea735c]
[INFO] [bt] (1) /home/ubuntu/incubator-mxnet/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(+0x2fc40af) [0x7f0aff99e0af]
[INFO] [bt] (2) /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7f0be3a204b0]
[INFO] [bt] (3) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3dfe3a) [0x7f0be0b53e3a]
[INFO] [bt] (4) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x323ac4) [0x7f0be0a97ac4]
[INFO] [bt] (5) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x323e12) [0x7f0be0a97e12]
[INFO] [bt] (6) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x324b07) [0x7f0be0a98b07]
[INFO] [bt] (7) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x4768db) [0x7f0be0bea8db]
[INFO] [bt] (8) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x478717) [0x7f0be0bec717]
[INFO] [bt] (9) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0xa12217) [0x7f0be1186217]

Testing with Intel MKL

Install instructions here:

https://software.intel.com/en-us/articles/installing-intel-free-libs-and-python-apt-repo

Code Block

language	bash
title	MKL Install instructions

USE_BLAS=mkl


wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB
sudo apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB
sudo wget https://apt.repos.intel.com/setup/intelproducts.list -O /etc/apt/sources.list.d/intelproducts.list
sudo sh -c 'echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list'
sudo apt-get update
sudo apt-get install intel-mkl-2018.2-046

Failed with Segfault:

Code Block

[INFO] Stack trace returned 10 entries:
[INFO] [bt] (0) /home/ubuntu/mxnet-master/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(dmlc::StackTrace[abi:cxx11]()+0x1bc) [0x7f2f04ca58ec]
[INFO] [bt] (1) /home/ubuntu/mxnet-master/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(+0x31d7a4f) [0x7f2f07971a4f]
[INFO] [bt] (2) /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7f3096cd24b0]
[INFO] [bt] (3) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3e4afc) [0x7f3093e0aafc]
[INFO] [bt] (4) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0xa239d6) [0x7f30944499d6]
[INFO] [bt] (5) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0xa24cdc) [0x7f309444acdc]
[INFO] [bt] (6) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0xa24e4c) [0x7f309444ae4c]
[INFO] [bt] (7) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x7c3252) [0x7f30941e9252]
[INFO] [bt] (8) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x5b00d6) [0x7f3093fd60d6]
[INFO] [bt] (9) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x5b2c44) [0x7f3093fd8c44]
[INFO] ------------------------------------------------------------------------

Publishing to Maven

Code Block

language	bash

export GPG_TTY=$(tty)
make scalapkg
make scalarelease-dryrun

...

Page tree

Versions Compared

Old Version 15

New Version Current

Key

MXNet-Scala Modules

Publishing Scala Packages to Nexus/Maven

Build MXNet backend on OSX

Build MXNet backend on Linux-CPU

Note: compiling with USE_BLAS throws a segmentation fault on both OSX and Linux platforms

Testing with Intel MKL

Publishing to Maven

Release Process