MXNet-Scala is a Scala binding for the MXNet Deep Learning framework. It currently supports Training and Inference
MXNet-Scala Modules
Module | Package Generated | Dependencies | Description |
---|---|---|---|
|
|
| |
2. core | mxnet-core_2.11 |
| |
3. macros | mxnet-macros_2.11 |
| |
4. init | mxnet-init_2.11 | ||
5. init-native |
| mxnet-init_2.11 | |
6. native |
| mxnet-core_2.11 | |
7. infer | mxnet-infer_2.11 |
| |
8. spark | mxnet-spark_2.11 |
| |
9. examples | mxnet-examples_2.11 |
|
Publishing Scala Packages to Nexus/Maven
Currently 3 Packages from the above generated are published to Maven. `mxnet_full_2.11-{platform}`.
Compile Flags Used to build the Scala package
The following flags are used to build the MXNet Backend(derived from mxnet pip compiler flags)
FLAGS | MXNet-Scala OSX | MXNet-Scala Linux CPU | MXNet-Scala Linux GPU |
---|---|---|---|
USE_BLAS | 0 | ||
USE_LAPACK | 1 | 1 | 1 |
USE_OPENCV | 1 | 1 (opencv 3.4.1) | 1 |
USE_S3 | 1 | 1 | 1 |
USE_DIST_KVSTORE | 0 | 1 | 1 |
USE_OPENMP | 1 | 1 | |
USE_CUDA | 1 | ||
USE_CUDNN | 0 | 1 | |
USE_NCCL | 0 | 1 | |
USE_MKLDNN | 0 | 0 | |
Java version | $ java -version | ubuntu@ip:~$ java -version |
Building on OS-X
Setup GPG Keys https://www.apache.org/dev/openpgp.html#key-gen-install-latest-gnupg
Export/Import GPG Keys
gpg --export-secret-keys -a xxx@apache.org > ~/Downloads/keys.asc gpg --import /tmp/keys.asc #list the keys to make sure they are there. gpg --list-keys #you will need this to enter your gpg-key password sudo apt install gnupg-agent
Build MXNet backend on OSX
Clone MXNet repo and checkout the release branch
export YOURFORK=whatever export RELEASE_VERISON=1.2.0 git clone https://github.com/$YOURFORK$/incubator-mxnet git remote add upstream https://github.com/apache/incubator-mxnet git fetch upstream $RELEASE_VERSION && git checkout -b $RELEASE_VERSION --track upstream/$RELEASE_VERSION
Install necessary libraries
brew install lapack brew install openblas
Build MXNet Backend
#This will build the MXNet backend with the tested configuration for publishing MXNet-Scala to Maven. scala-package/dev/compile-mxnet-backend.sh osx-x86_64-cpu ./
Maven Publish
Follow Maven Publish Link https://www.apache.org/dev/publishing-maven-artifacts.html to setup repo settings, etc., before you proceed
#if you don't set this it will fail with the error "gpg: signing failed: Inappropriate ioctl for device" export GPG_TTY=$(tty) #make sure everything is working as expected make scalarelease-dryrun # Releases to Apache Snapshot make scaladeploy # Verify that the package has been uploaded in https://repository.apache.org/content/repositories/snapshots/ # Preparing the release will create the new tag, automatically checking in on your behalf.(We have disabled CheckIn) make scalarelease-prepare # this will upload to the staging repository and make it ready for closing make scalarelease-perform
Build MXNet backend on Linux-CPU
To build for linux-cpu platform, we use the AWS Deep Learning Base AMI that comes built with all the necessary
libraries such as opencv/cudnn etc,.
Install Maven on Ubuntu
sudo apt-get install -y maven
Debugging Info
git push --force --delete ./ refs/heads/mxnet-parent_2.11-1.2.0 git tag -d mxnet-parent_2.11-1.2.0
If you face an issue because of gpg such as ```Enter passphrase: gpg: gpg-agent is not available in this session```
sudo apt-get install gnupg2 sudo mv /usr/bin/gpg /usr/bin/gpg.1 sudo ln -s /usr/bin/gpg2 /usr/bin/gpg
gpg: signing failed: Inappropriate ioctl for device
```export GPG_TTY=$(tty)```
Note: compiling with USE_SIGNAL_HANDLER throws a segmentation fault on both OSX and Linux platforms
Publishing to Maven
export GPG_TTY=$(tty) make scalapkg make scalarelease-dryrun