Page History

...

MXNet-Scala is a Scala binding for the MXNet Deep Learning framework. It currently supports Training and Inference

MXNet-Scala Modules

Module	Package Generated	Dependencies
assembly	mxnet-full-parent_2.11 mxnet-full_2.11-{platform}	mxnet-core_2.11 libmxnet-scala-{platform}
2. core	mxnet-core_2.11	mxnet-init_2.11 mxnet-macros_2.11
3. macros	mxnet-macros_2.11	mxnet-init_2.11 libmxnet-init-scala-{platform}
4. init	mxnet-init_2.11
5. init-native	mxnet-scala-init-native-parent libmxnet-init-scala-osx-x86_64 libmxnet-init-scala-linux-x86_64	mxnet-init_2.11
6. native	mxnet-parent_2.11 libmxnet-scala-{platform}	mxnet-core_2.11
7. infer	mxnet-infer_2.11	mxnet-core_2.11 libmxnet-scala-{platform}
8. spark	mxnet-spark_2.11	mxnet-core_2.11 libmxnet-scala-{platform} spark-mllib_2.11
9. examples	mxnet-examples_2.11	mxnet-core_2.11 mxnet-infer_2.11

Publishing Scala Packages to Nexus/Maven

Currently 3 Packages from the above generated are published to Maven. `mxnet_full_2.11-{platform}`.

Compile Flags Used to build the Scala package

The following flags are used to build the MXNet Backend(derived from mxnet pip compiler flags)

...

MXNet-Scala Linux CPU

...

USE_CUDNN

...

1

...

$ java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

...

ubuntu@ip:~$ java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (Zulu 8.20.0.5-linux64) (build 1.8.0_121-b15)
OpenJDK 64-Bit Server VM (Zulu 8.20.0.5-linux64) (build 25.121-b15, mixed mode)

Building on OS-X

Setup GPG Keys https://www.apache.org/dev/openpgp.html#key-gen-install-latest-gnupg

Build MXNet backend on OSX

Clone MXNet repo and checkout the release branch

Code Block

language	bash

export YOURFORK=whatever
export RELEASE_VERISON=1.2.0
git clone https://github.com/$YOURFORK$/incubator-mxnet
git remote add upstream https://github.com/apache/incubator-mxnet
git fetch upstream $RELEASE_VERSION && git checkout -b $RELEASE_VERSION --track upstream/$RELEASE_VERSION

Install necessary libraries

Code Block

language	bash

brew install lapack
brew install openblas

Build MXNet Backend

Code Block

language	bash

#This will build the MXNet backend with the tested configuration for publishing MXNet-Scala to Maven.
scala-package/dev/compile-mxnet-backend.sh osx-x86_64-cpu ./

Maven Publish

Follow Maven Publish Link https://www.apache.org/dev/publishing-maven-artifacts.html to setup repo settings, etc., before you proceed

Code Block

title	Maven Publish Process

#if you don't set this it will fail with the error "gpg: signing failed: Inappropriate ioctl for device"
export GPG_TTY=$(tty)

#make sure everything is working as expected
make scalarelease-dryrun
# Releases to Apache Snapshot
make scaladeploy 
# Verify that the package has been uploaded in https://repository.apache.org/content/repositories/snapshots/
# Preparing the release will create the new tag, automatically checking in on your behalf.(We have disabled CheckIn)
make scalarelease-prepare
# this will upload to the staging repository and make it ready for closing
make scalarelease-perform

Build MXNet backend on Linux-CPU

To build for linux-cpu platform, we use the AWS Deep Learning Base AMI that comes built with all the necessary

libraries such as opencv/cudnn etc,.

Export/Import GPG Keys

First you need to export the gpg keys you created on Mac and import to the linux host

Install Maven, gnupg, gnupg2 & gnupg-agent

Code Block

language	bash

sudo apt-get install -y maven gnupg gnupg2 gnupg-agent && 
#soft link gnupg2 to gnupg, Update the below when you find a better solution
sudo mv /usr/bin/gpg /usr/bin/gpg.1 && 
sudo ln -s /usr/bin/gpg2 /usr/bin/gpg

Export and ship gpg keys you created on the Mac

Code Block

language	bash

gpg --export-secret-keys -a xxx@apache.org > ~/Downloads/keys.asc
scp ~/Downloads/keys.asc ubuntu@host:~/keys.asc

Import and verify they look fine

Code Block

language	bash

gpg --import ~/keys.asc
gpg --list-keys

Update the gnupg-agent cachettl, it takes a long time to build on linux and linux-gpu instances and will time out during release if you don't baby sit the build

Code Block

language	bash

mkdir -p ~/.gnupg && touch ~/.gnupg/gpg-agent.conf
#set to 4 hours 
echo "default-cache-ttl 14400" > ~/.gnupg/gpg-agent.conf && 
echo "max-cache-ttl 14400" >> ~/.gnupg/gpg-agent.conf

Start gpg-agent to use the above

Code Block

language	bash

eval $(gpg-agent --daemon)

Maven Password Encryption for use in release

Now setup your maven create a master password and paste the encrypted the more info Password Encryption

Code Block

language	bash

touch ~/.m2/settings-security.xml &&
mvn --encrypt-master-password

paste the encrypted master-password you get from above in `~/.m2/settings-security.xml`

Code Block

language	xml

<settingsSecurity>
  <master>ENCRYPTEDPASSWORD</master>
</settingsSecurity>

encrypt your password for `apache` login

Code Block

language	bash

touch ~/.m2/settings.xml && mvn --encrypt-password

Paste the encrypted password for use in apache snapshot and staging in `~/.m2/settings.xml`.

Code Block

language	xml

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                          https://maven.apache.org/xsd/settings-1.0.0.xsd">
  <servers>
    <!-- To publish a snapshot of some part of Maven -->
    <server>
      <id>apache.snapshots.https</id>
      <username>APACHE_USERNAME</username>
      <password>{COQLCE6DU6GtcS5P=}</password>
    </server>
    <!-- To stage a release of some part of Maven -->
    <server>
      <id>apache.releases.https</id>
      <username>APACHE_USERNAME</username>
      <password>{COQLCE6DU6GtcS5P=}</password>
    </server>
  </servers>
</settings>

Install Maven on Ubuntu

Code Block

language	bash

sudo apt-get install -y maven

Using Package in Staging Repository

Update your pom file to include the repository info like below:

Code Block

language	xml
title	Staging Repo

  <repository>
    <id>snapshot</id>
    <url>https://repository.apache.org/content/repositories/staging</url>
  </repository>
</repositories>

<dependencies>
  <dependency>
    <groupId>org.apache.mxnet</groupId>
    <artifactId>mxnet-full_2.11-osx-x86_64-cpu</artifactId>
    <version>1.2.0</version>
  </dependency>
</dependencies

Debugging Info

Code Block

language	bash

git push --force --delete ./ refs/heads/mxnet-parent_2.11-1.2.0
git tag -d mxnet-parent_2.11-1.2.0

If you face an issue because of gpg such as ```Enter passphrase: gpg: gpg-agent is not available in this session```

Code Block
sudo apt-get install gnupg2 sudo mv /usr/bin/gpg /usr/bin/gpg.1 sudo ln -s /usr/bin/gpg2 /usr/bin/gpg

gpg: signing failed: Inappropriate ioctl for device

```export GPG_TTY=$(tty)```

Note: compiling with USE_SIGNAL_HANDLER throws a segmentation fault on both OSX and Linux platforms

Publishing to Maven

...

language	bash

...

Page tree

Versions Compared

Old Version 22

New Version Current

Key

MXNet-Scala Modules

Publishing Scala Packages to Nexus/Maven

Compile Flags Used to build the Scala package

Building on OS-X

Build MXNet backend on OSX

Install necessary libraries

Build MXNet Backend

Maven Publish

Build MXNet backend on Linux-CPU

Export/Import GPG Keys

Maven Password Encryption for use in release

Install Maven on Ubuntu

Using Package in Staging Repository

Debugging Info

gpg: signing failed: Inappropriate ioctl for device

Note: compiling with USE_SIGNAL_HANDLER throws a segmentation fault on both OSX and Linux platforms

Publishing to Maven

Release Process