Versions Compared

Old Version 12

changes.mady.by.user Naveen Swamy

Saved on

compared with

New Version 13

changes.mady.by.user Naveen Swamy

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...


ModulePackage GeneratedDependenciesDescription
  1. assembly
  1. mxnet-full-parent_2.11
  2. mxnet-full_2.11-{platform}
  1. mxnet-core_2.11
  2. libmxnet-scala-{platform}

2. core

mxnet-core_2.11

  1. mxnet-init_2.11
  2. mxnet-macros_2.11

3. macros

mxnet-macros_2.11

  1. mxnet-init_2.11
  2. libmxnet-init-scala-{platform}

4. init

mxnet-init_2.11

 

5. init-native

  1. mxnet-scala-init-native-parent
  2. libmxnet-init-scala-osx-x86_64

  3. libmxnet-init-scala-linux-x86_64

mxnet-init_2.11


6. native

  1. mxnet-parent_2.11
  2. libmxnet-scala-{platform}

 mxnet-core_2.11


7. infer

mxnet-infer_2.11

  1. mxnet-core_2.11
  2. libmxnet-scala-{platform}

8. spark

mxnet-spark_2.11

  1. mxnet-core_2.11
  2. libmxnet-scala-{platform}
  3. spark-mllib_2.11

9. examples

mxnet-examples_2.11

  1. mxnet-core_2.11
  2. mxnet-infer_2.11


Publishing Scala Packages to Nexus/Maven

Currently 3 Packages from the above generated are published to Maven. `mxnet_full_2.11-{platform}`.

The following flags are used to build the MXNet Backend(derived from mxnet pip compiler flags)


FLAGSMXNet-Scala OSX

MXNet-Scala Linux CPU

MXNet-Scala Linux GPU
USE_BLAS

 
USE_LAPACK11 
USE_OPENCV11 (opencv 3.4.1) 
USE_S311 
USE_DIST_KVSTORE01 
USE_SIGNAL_HANDLER11 
USE_OPENMP 11

USE_CUDNN

 01
USE_NCCL 0

1

Java version

$ java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

ubuntu@ip:~$ java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (Zulu 8.20.0.5-linux64) (build 1.8.0_121-b15)
OpenJDK 64-Bit Server VM (Zulu 8.20.0.5-linux64) (build 25.121-b15, mixed mode)



Setup GPG Keys

Build MXNet backend on OSX

  1. Clone MXNet repo and checkout the release branch
Code Block
languagebash
collapsetrue
https://github.com/apache/incubator-mxnet.git ~/$MXNET_LOC
git fetch all
git checkout -b $RELEASE_BRANCH
cp ~/$MXNET_LOC/make/config.mk ~/$MXNET_LOC


2. Install Lapack

Code Block
languagebash
brew install lapack


setup all the flags

Code Block
echo "ADD_CFLAGS  += -I/usr/local/opt/lapack/include" >> ~/$MXNET_LOC/config.mk
echo "ADD_LDFLAGS += -L/usr/local/opt/lapack/lib"     >> ~/$MXNET_LOC/config.mk
echo "USE_OPENMP = 0" >> ~/$MXNET_LOC/config.mk


Build MXNet backend on Linux-CPU

To build for linux-cpu platform, we use the AWS Deep Learning Base AMI that comes built with all the necessary 

libraries such as opencv/cudnn etc,.

Install Maven on Ubuntu

Code Block
languagebash
sudo apt-get install -y maven

setup all the flags

Code Block
cd incubator-mxnet
cp make/config.mk ./ 
echo "ADD_CFLAGS += -I/usr/local/include/opencv" >> ./config.mk
echo "ADD_LDFLAGS += -L/usr/local/lib" >> ./config.mk
echo "USE_BLAS=openblas" >> ./config.mk 
echo "USE_SIGNAL_HANDLER=1" >> ./config.mk
echo "USE_DIST_KVSTORE=1" >> ./config.mk
echo "USE_S3=1" >> ./config.mk
echo "USE_OPENMP=1" >> ./config.mk

Note: compiling with OpenBlas throws a segmentation fault on both OSX and Linux platforms

Code Block
languagebash
titleSegfaults when MXNet backend is built with openblas
[INFO] --- maven-scala-plugin:2.15.2:compile (default) @ mxnet-core_2.11 ---
[INFO] Checking for multiple versions of scala
[WARNING]  Expected all dependencies to require Scala version: 2.11.8
[WARNING]  org.apache.mxnet:mxnet-init_2.11:1.2.0-SNAPSHOT requires scala version: 2.11.8
[WARNING]  org.apache.mxnet:mxnet-init_2.11:1.2.0-SNAPSHOT requires scala version: 2.11.8
[WARNING]  org.apache.mxnet:mxnet-core_2.11:1.2.0-SNAPSHOT requires scala version: 2.11.8
[WARNING]  org.scala-lang:scala-reflect:2.11.8 requires scala version: 2.11.8
[WARNING]  org.scalatest:scalatest_2.11:3.0.4 requires scala version: 2.11.11
[WARNING] Multiple versions of scala libraries detected!
[INFO] includes = [**/*.java,**/*.scala,]
[INFO] excludes = []
[INFO] /home/ubuntu/incubator-mxnet/scala-package/core/src/main/scala:-1: info: compiling
[INFO] Compiling 51 source files to /home/ubuntu/incubator-mxnet/scala-package/core/target/classes at 1527282462191
[INFO] compiler plugin: BasicArtifact(org.scalamacros,paradise_2.11.8,2.1.0)
[INFO]
[INFO] Segmentation fault: 11
[INFOStack trace returned 10 entries:
[INFO] [bt] (0) /home/ubuntu/incubator-mxnet/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(dmlc::StackTrace[abi:cxx11]()+0x1bc) [0x7f0afcea735c]
[INFO] [bt] (1) /home/ubuntu/incubator-mxnet/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(+0x2fc40af) [0x7f0aff99e0af]
[INFO] [bt] (2) /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7f0be3a204b0]
[INFO] [bt] (3) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x8797bd) [0x7f0be0fed7bd]
[INFO] [bt] (4) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3dac73) [0x7f0be0b4ec73]
[INFO] [bt] (5) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x40594d) [0x7f0be0b7994d]
[INFO] [bt] (6) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3ed3a0) [0x7f0be0b613a0]
[INFO] [bt] (7) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x4047ac) [0x7f0be0b787ac]
[INFO] [bt] (8) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x4032ce) [0x7f0be0b772ce]
[INFO] [bt] (9) /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3de959) [0x7f0be0b52959]
[INFO] Stack trace returned 10 entries:
[INFO] [bt] (0) /home/ubuntu/incubator-mxnet/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(dmlc::StackTrace[abi:cxx11]()+0x1bc) [0x7fd294ea735c0x7f0afcea735c]
[INFO] [bt] (1) /home/ubuntu/incubator-mxnet/scala-package/init-native/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so(+0x2fc40af) [0x7fd29799e0af0x7f0aff99e0af]
[INFO] [bt] (2) /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7fd33fea24b00x7f0be3a204b0]
[INFO] [bt] (3) /usr/homelib/ubuntu/anaconda3jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x40ddf10x3dfe3a) [0x7fd33d67edf10x7f0be0b53e3a]
[INFO] [bt] (4) /homeusr/lib/ubuntu/anaconda3jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x9f55c80x323ac4) [0x7fd33dc665c80x7f0be0a97ac4]
[INFO] [bt] (5) /usr/homelib/ubuntu/anaconda3jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x9ef2600x323e12) [0x7fd33dc602600x7f0be0a97e12]
[INFO] [bt] (6) /homeusr/lib/ubuntu/anaconda3jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x9f2e8e0x324b07) [0x7fd33dc63e8e0x7f0be0a98b07]
[INFO] [bt] (7) /usr/homelib/ubuntu/anaconda3jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x9f2db50x4768db) [0x7fd33dc63db50x7f0be0bea8db]
[INFO] [bt] (8) /homeusr/lib/ubuntu/anaconda3jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x3f0e520x478717) [0x7fd33d661e520x7f0be0bec717]
[INFO] [bt] (9) /usr/homelib/ubuntu/anaconda3jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x8e811d0xa12217) [0x7fd33db5911d]
[INFO] ------------------------------------------------------------------------0x7f0be1186217]


Publishing to Maven


Code Block
languagebash
export GPG_TTY=$(tty)
make scalapkg
make scalarelease-dryrun


Set up your dev environment