Page History

...

Module	Package Generated	Dependencies
assembly	mxnet-full-parent_2.11 mxnet-full_2.11-{platform}	mxnet-core_2.11 libmxnet-scala-{platform}
2. core	mxnet-core_2.11	mxnet-init_2.11 mxnet-macros_2.11
3. macros	mxnet-macros_2.11	mxnet-init_2.11 libmxnet-init-scala-{platform}
4. init	mxnet-init_2.11
5. init-native	mxnet-scala-init-native-parent libmxnet-init-scala-osx-x86_64 libmxnet-init-scala-linux-x86_64	mxnet-init_2.11
6. native	mxnet-parent_2.11 libmxnet-scala-{platform}	mxnet-core_2.11
7. infer	mxnet-infer_2.11	mxnet-core_2.11 libmxnet-scala-{platform}
8. spark	mxnet-spark_2.11	mxnet-core_2.11 libmxnet-scala-{platform} spark-mllib_2.11
9. examples	mxnet-examples_2.11	mxnet-core_2.11 mxnet-infer_2.11

Publishing Scala Packages to Nexus/Maven

Currently 3 Packages from the above generated are published to Maven. `mxnet_full_2.11-{platform}`.

The following flags are used to build the MXNet Backend(derived from mxnet pip compiler flags)

FLAGS	MXNet-Scala OSX	MXNet-Scala Linux CPU	MXNet-Scala Linux GPU
USE_BLAS		0
USE_LAPACK	1	1
USE_OPENCV	1	1 (opencv 3.4.1)
USE_S3	1	1
USE_DIST_KVSTORE	0	1
USE_OPENMP		1	1
USE_CUDNN		0	1
USE_NCCL		0	1
USE_MKLDNN		0
Java version	$ java -version java version "1.8.0_171" Java(TM) SE Runtime Environment (build 1.8.0_171-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)	ubuntu@ip:~$ java -version openjdk version "1.8.0_121" OpenJDK Runtime Environment (Zulu 8.20.0.5-linux64) (build 1.8.0_121-b15) OpenJDK 64-Bit Server VM (Zulu 8.20.0.5-linux64) (build 25.121-b15, mixed mode)

Setup GPG Keys

Export GPG Keys

Code Block

language	bash

gpg --export-secret-keys -a xxx@apache.org > ~/Downloads/keys.asc
gpg --import /tmp/keys.asc
#list the keys to make sure they are there.
gpg --list-keys

Build MXNet backend on OSX

Clone MXNet repo and checkout the release branch

Code Block

language	bash
collapse	true

export YOURFORK=whatever
export RELEASE_VERISON=1.2.0
git clone https://github.com/apache$YOURFORK$/incubator-mxnet.
git ~/$MXNET_LOCremote add upstream https://github.com/apache/incubator-mxnet
git fetch all
 upstream $RELEASE_VERSION && git checkout -b $RELEASE_BRANCH
cp ~/$MXNET_LOC/make/config.mk ~/$MXNET_LOC

2. Install Lapack

VERSION --track upstream/$RELEASE_VERSION

Install necessary libraries

Code Block

language	bash

brew install lapack

setup all the flags


brew install openblas

Build MXNet Backend

Code Block

language	bash

brew install lapackbrew install openblas

Maven Publish

Follow Maven Publish Link https://www.apache.org/dev/publishing-maven-artifacts.html to setup repo settings, etc., before you proceed

Code Block

title	Maven Publish Process

make scalarelease-dryrun
# Releases to Apache Snapshot
make scaladeploy

Code Block

echo "ADD_CFLAGS  += -I/usr/local/opt/lapack/include" >> ~/$MXNET_LOC/config.mk
echo "ADD_LDFLAGS += -L/usr/local/opt/lapack/lib"     >> ~/$MXNET_LOC/config.mk
echo "USE_OPENMP = 0" >> ~/$MXNET_LOC/config.mk

Build MXNet backend on Linux-CPU

To build for linux-cpu platform, we use the AWS Deep Learning Base AMI that comes built with all the necessary

libraries such as opencv/cudnn etc,.

Install Maven on Ubuntu

Code Block

language	bash

sudo apt-get install -y maven

setup all the flags

Code Block

cd incubator-mxnet
cp make/config.mk ./ 
echo "ADD_CFLAGS += -I/usr/local/include/opencv" >> ./config.mk
echo "ADD_LDFLAGS += -L/usr/local/lib" >> ./config.mk
echo "USE_BLAS=openblas" >> ./config.mk 
echo "USE_SIGNAL_HANDLER=1" >> ./config.mk
echo "USE_DIST_KVSTORE=1" >> ./config.mk
echo "USE_S3=1" >> ./config.mk
echo "USE_OPENMP=1" >> ./config.mk

Note: compiling with USE_

BLAS

SIGNAL_HANDLER throws a segmentation fault on both OSX and Linux platforms

Code Block

language	bash
title	Segfaults when MXNet backend is built with openblas

[INFO] Stack trace returned 10 entries:
 [INFO] [bt] (0) /home/ubuntu/incubator-mxnet/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(dmlc::StackTrace[abi:cxx11]()+0x1bc) [0x7f0afcea735c]
 [INFO] [bt] (1) /home/ubuntu/incubator-mxnet/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(+0x2fc40af) [0x7f0aff99e0af]
 [INFO] [bt] (2) /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7f0be3a204b0]
 [INFO] [bt] (3) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x8797bd) [0x7f0be0fed7bd]
 [INFO] [bt] (4) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3dac73) [0x7f0be0b4ec73]
 [INFO] [bt] (5) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x40594d) [0x7f0be0b7994d]
 [INFO] [bt] (6) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3ed3a0) [0x7f0be0b613a0]
 [INFO] [bt] (7) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x4047ac) [0x7f0be0b787ac]
 [INFO] [bt] (8) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x4032ce) [0x7f0be0b772ce]
 [INFO] [bt] (9) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3de959) [0x7f0be0b52959]
 [INFO] Stack trace returned 10 entries:
 [INFO] [bt] (0) /home/ubuntu/incubator-mxnet/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(dmlc::StackTrace[abi:cxx11]()+0x1bc) [0x7f0afcea735c]
 [INFO] [bt] (1) /home/ubuntu/incubator-mxnet/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(+0x2fc40af) [0x7f0aff99e0af]
 [INFO] [bt] (2) /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7f0be3a204b0]
 [INFO] [bt] (3) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3dfe3a) [0x7f0be0b53e3a]
 [INFO] [bt] (4) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x323ac4) [0x7f0be0a97ac4]
 [INFO] [bt] (5) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x323e12) [0x7f0be0a97e12]
 [INFO] [bt] (6) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x324b07) [0x7f0be0a98b07]
 [INFO] [bt] (7) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x4768db) [0x7f0be0bea8db]
 [INFO] [bt] (8) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x478717) [0x7f0be0bec717]
 [INFO] [bt] (9) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0xa12217) [0x7f0be1186217]

Testing with Intel MKL

Install instructions here:

https://software.intel.com/en-us/articles/installing-intel-free-libs-and-python-apt-repo

Code Block

language	bash
title	MKL Install instructions

USE_BLAS=mkl


 wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB
 sudo apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB
 sudo wget https://apt.repos.intel.com/setup/intelproducts.list -O /etc/apt/sources.list.d/intelproducts.list
 sudo sh -c 'echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list'
 sudo apt-get update
 sudo apt-get install intel-mkl-2018.2-046

Failed with Segfault:

Code Block

[INFO] Stack trace returned 10 entries:
 [INFO] [bt] (0) /home/ubuntu/mxnet-master/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(dmlc::StackTrace[abi:cxx11]()+0x1bc) [0x7f2f04ca58ec]
 [INFO] [bt] (1) /home/ubuntu/mxnet-master/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(+0x31d7a4f) [0x7f2f07971a4f]
 [INFO] [bt] (2) /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7f3096cd24b0]
 [INFO] [bt] (3) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3e4afc) [0x7f3093e0aafc]
 [INFO] [bt] (4) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0xa239d6) [0x7f30944499d6]
 [INFO] [bt] (5) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0xa24cdc) [0x7f309444acdc]
 [INFO] [bt] (6) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0xa24e4c) [0x7f309444ae4c]
 [INFO] [bt] (7) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x7c3252) [0x7f30941e9252]
 [INFO] [bt] (8) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x5b00d6) [0x7f3093fd60d6]
 [INFO] [bt] (9) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x5b2c44) [0x7f3093fd8c44]
 [INFO] ------------------------------------------------------------------------

Publishing to Maven

Code Block

language	bash

export GPG_TTY=$(tty)
 make scalapkg
 make scalarelease-dryrun

Page tree

Versions Compared

Old Version 16

New Version 17

Key

Publishing Scala Packages to Nexus/Maven

Build MXNet backend on OSX

Install necessary libraries

Build MXNet Backend

Maven Publish

Build MXNet backend on Linux-CPU

To build for linux-cpu platform, we use the AWS Deep Learning Base AMI that comes built with all the necessary

libraries such as opencv/cudnn etc,.

Install Maven on Ubuntu

setup all the flags

Note: compiling with USE_

SIGNAL_HANDLER throws a segmentation fault on both OSX and Linux platforms

Testing with Intel MKL

Install instructions here:

https://software.intel.com/en-us/articles/installing-intel-free-libs-and-python-apt-repo

Failed with Segfault:

Publishing to Maven

Set up your dev environment