Versions Compared

Old Version 17

changes.mady.by.user Naveen Swamy

Saved on

compared with

New Version 18

changes.mady.by.user Naveen Swamy

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...


ModulePackage GeneratedDependenciesDescription
  1. assembly
  1. mxnet-full-parent_2.11
  2. mxnet-full_2.11-{platform}
  1. mxnet-core_2.11
  2. libmxnet-scala-{platform}

2. core

mxnet-core_2.11

  1. mxnet-init_2.11
  2. mxnet-macros_2.11

3. macros

mxnet-macros_2.11

  1. mxnet-init_2.11
  2. libmxnet-init-scala-{platform}

4. init

mxnet-init_2.11

 

5. init-native

  1. mxnet-scala-init-native-parent
  2. libmxnet-init-scala-osx-x86_64

  3. libmxnet-init-scala-linux-x86_64

mxnet-init_2.11


6. native

  1. mxnet-parent_2.11
  2. libmxnet-scala-{platform}

 mxnet-core_2.11


7. infer

mxnet-infer_2.11

  1. mxnet-core_2.11
  2. libmxnet-scala-{platform}

8. spark

mxnet-spark_2.11

  1. mxnet-core_2.11
  2. libmxnet-scala-{platform}
  3. spark-mllib_2.11

9. examples

mxnet-examples_2.11

  1. mxnet-core_2.11
  2. mxnet-infer_2.11


Publishing Scala Packages to Nexus/Maven

Currently 3 Packages from the above generated are published to Maven. `mxnet_full_2.11-{platform}`.

The following flags are used to build the MXNet Backend(derived from mxnet pip compiler flags)


FLAGSMXNet-Scala OSX

MXNet-Scala Linux CPU

MXNet-Scala Linux GPU
USE_BLAS
0 
USE_LAPACK11 
USE_OPENCV11 (opencv 3.4.1) 
USE_S311 
USE_DIST_KVSTORE01 
USE_OPENMP 11

USE_CUDNN

 01
USE_NCCL 0

1

USE_MKLDNN
0
Java version

$ java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

ubuntu@ip:~$ java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (Zulu 8.20.0.5-linux64) (build 1.8.0_121-b15)
OpenJDK 64-Bit Server VM (Zulu 8.20.0.5-linux64) (build 25.121-b15, mixed mode)




Export GPG Keys

Code Block
languagebash
gpg --export-secret-keys -a xxx@apache.org > ~/Downloads/keys.asc
gpg --import /tmp/keys.asc
#list the keys to make sure they are there.
gpg --list-keys

Build MXNet backend on OSX

Clone MXNet repo and checkout the release branch

Code Block
languagebash
export YOURFORK=whatever
export RELEASE_VERISON=1.2.0
git clone https://github.com/$YOURFORK$/incubator-mxnet
git remote add upstream https://github.com/apache/incubator-mxnet
git fetch upstream $RELEASE_VERSION && git checkout -b $RELEASE_VERSION --track upstream/$RELEASE_VERSION

Install necessary libraries

Code Block
languagebash
brew install lapack
brew install openblas

Build MXNet Backend

Code Block
languagebash
brew install lapackbrew install openblas

Maven Publish

Follow Maven Publish Link https://www.apache.org/dev/publishing-maven-artifacts.html to setup repo settings, etc., before you proceed

Code Block
titleMaven Publish Process
#if you don't set this it will fail with the error "gpg: signing failed: Inappropriate ioctl for device"
export GPG_TTY=$(tty)
make scalarelease-dryrun
# Releases to Apache Snapshot
make scaladeploy


Build MXNet backend on Linux-CPU

To build for linux-cpu platform, we use the AWS Deep Learning Base AMI that comes built with all the necessary 

libraries such as opencv/cudnn etc,.

Install Maven on Ubuntu

Code Block
languagebash
sudo apt-get install -y maven

setup all the flags


Note: compiling with USE_SIGNAL_HANDLER throws a segmentation fault on both OSX and Linux platforms

Code Block
languagebash
titleSegfaults when MXNet backend is built with openblas
[INFO] Stack trace returned 10 entries: [INFO] [bt] (0) /home/ubuntu/incubator-mxnet/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(dmlc::StackTrace[abi:cxx11]()+0x1bc) [0x7f0afcea735c] [INFO] [bt] (1) /home/ubuntu/incubator-mxnet/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(+0x2fc40af) [0x7f0aff99e0af] [INFO] [bt] (2) /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7f0be3a204b0] [INFO] [bt] (3) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x8797bd) [0x7f0be0fed7bd] [INFO] [bt] (4) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3dac73) [0x7f0be0b4ec73] [INFO] [bt] (5) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x40594d) [0x7f0be0b7994d] [INFO] [bt] (6) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3ed3a0) [0x7f0be0b613a0] [INFO] [bt] (7) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x4047ac) [0x7f0be0b787ac] [INFO] [bt] (8) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x4032ce) [0x7f0be0b772ce] [INFO] [bt] (9) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3de959) [0x7f0be0b52959] [INFO] Stack trace returned 10 entries: [INFO] [bt] (0) /home/ubuntu/incubator-mxnet/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(dmlc::StackTrace[abi:cxx11]()+0x1bc) [0x7f0afcea735c] [INFO] [bt] (1) /home/ubuntu/incubator-mxnet/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(+0x2fc40af) [0x7f0aff99e0af] [INFO] [bt] (2) /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7f0be3a204b0] [INFO] [bt] (3) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3dfe3a) [0x7f0be0b53e3a] [INFO] [bt] (4) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x323ac4) [0x7f0be0a97ac4] [INFO] [bt] (5) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x323e12) [0x7f0be0a97e12] [INFO] [bt] (6) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x324b07) [0x7f0be0a98b07] [INFO] [bt] (7) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x4768db) [0x7f0be0bea8db] [INFO] [bt] (8) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x478717) [0x7f0be0bec717] [INFO] [bt] (9) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0xa12217) [0x7f0be1186217]





Testing with Intel MKL

Install instructions here: 

https://software.intel.com/en-us/articles/installing-intel-free-libs-and-python-apt-repo

Code Block
languagebash
titleMKL Install instructions
USE_BLAS=mkl wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB sudo apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB sudo wget https://apt.repos.intel.com/setup/intelproducts.list -O /etc/apt/sources.list.d/intelproducts.list sudo sh -c 'echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list' sudo apt-get update sudo apt-get install intel-mkl-2018.2-046


Failed with Segfault:

Code Block
[INFO] Stack trace returned 10 entries: [INFO] [bt] (0) /home/ubuntu/mxnet-master/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(dmlc::StackTrace[abi:cxx11]()+0x1bc) [0x7f2f04ca58ec] [INFO] [bt] (1) /home/ubuntu/mxnet-master/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(+0x31d7a4f) [0x7f2f07971a4f] [INFO] [bt] (2) /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7f3096cd24b0] [INFO] [bt] (3) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3e4afc) [0x7f3093e0aafc] [INFO] [bt] (4) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0xa239d6) [0x7f30944499d6] [INFO] [bt] (5) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0xa24cdc) [0x7f309444acdc] [INFO] [bt] (6) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0xa24e4c) [0x7f309444ae4c] [INFO] [bt] (7) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x7c3252) [0x7f30941e9252] [INFO] [bt] (8) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x5b00d6) [0x7f3093fd60d6] [INFO] [bt] (9) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x5b2c44) [0x7f3093fd8c44] [INFO] ------------------------------------------------------------------------


Publishing to Maven

Code Block
languagebash
export GPG_TTY=$(tty) make scalapkg make scalarelease-dryrun

Set up your dev environment