...
Module | Package Generated | Dependencies | Description |
---|---|---|---|
|
|
| |
2. core | mxnet-core_2.11 |
| |
3. macros | mxnet-macros_2.11 |
| |
4. init | mxnet-init_2.11 | ||
5. init-native |
| mxnet-init_2.11 | |
6. native |
| mxnet-core_2.11 | |
7. infer | mxnet-infer_2.11 |
| |
8. spark | mxnet-spark_2.11 |
| |
9. examples | mxnet-examples_2.11 |
|
Publishing Scala Packages to Nexus/Maven
Currently 3 Packages from the above generated are published to Maven. `mxnet_full_2.11-{platform}`.
Compile Flags Used to build the Scala package
The following flags are used to build the MXNet Backend(derived from mxnet pip compiler flags)
...
FLAGS | MXNet-Scala OSX | MXNet-Scala Linux CPU | MXNet-Scala Linux GPU |
---|---|---|---|
USE_BLAS | 0 | ||
USE_LAPACK | 1 | 1 | 1 |
USE_OPENCV | 1 | 1 (opencv 3.4.1) | 1 |
USE_S3 | 1 | 1 | 1 |
USE_DIST_KVSTORE | 0 | 1 | 1 |
USE_OPENMP | 1 | 1 | |
USE_CUDA | 1 | ||
USE_CUDNN | 0 | 1 | |
USE_NCCL | 0 | 1 | |
USE_MKLDNN | 0 | 0 | |
Java version | $ java -version | ubuntu@ip:~$ java -version |
Building on OS-X
Setup GPG Keys https://www.apache.org/dev/openpgp.html#key-gen-install-latest-gnupg
Build MXNet backend on OSX
Clone MXNet repo and checkout the release branch
Code Block | ||
---|---|---|
| ||
export YOURFORK=whatever export RELEASE_VERISON=1.2.0 git clone https://github.com/$YOURFORK$$YOURFORK/incubator-mxnet ~/mxnet-$RELEASEVERSION cd ~/mxnet-$RELEASEVERSION git remote add upstream https://github.com/apache/incubator-mxnet git fetch upstream $RELEASE_VERSIONv1.2.0 && git checkout -b $RELEASE_VERSION$RELEASEVERISON --track upstream/$RELEASE_VERSION |
...
v1.2.0
git submodule update --init --recursive
|
Install lapack, openblas, opencv packages
Code Block | ||
---|---|---|
| ||
brew install lapack#try opencv3 next time you release brew install lapack openblas opencv@2 |
Build MXNet
...
backend on OSX
Code Block | ||
---|---|---|
| ||
#This will build the MXNet backend with the tested configuration for publishing MXNet-Scala to Maven.
cd ~/mxnet-$RELEASEVERSION
scala-package/dev/compile-mxnet-backend.sh osx-x86_64-cpu |
Maven Password Encryption for use in release
Now setup your maven create a master password and paste the encrypted the more info Password Encryption
Code Block | ||
---|---|---|
| ||
touch ~/.m2/settings-security.xml &&
mvn --encrypt-master-password |
Paste the encrypted master-password you get from above in `~/.m2/settings-security.xml`
Code Block | ||
---|---|---|
| ||
<settingsSecurity>
<master>ENCRYPTEDMASTERPASSWORD</master>
</settingsSecurity> |
Encrypt your password for `apache` login
Code Block | ||
---|---|---|
| ||
cp /usr/share/maven/conf/settings.xml ~/.m2/ && mvn --encrypt-password |
Paste the below with your apache username and encrypted apache password updated under `<servers>`tag, this will used to publish to apache snapshot and staging in `~/.m2/settings.xml`.
Code Block | ||
---|---|---|
| ||
<!-- To publish a snapshot of some part of Maven --> <server> <id>apache.snapshots.https</id> <username>APACHEUSERNAME</username> <password>APACHEPASSWORDENCRYPTED</password> </server> <!-- To stage a release of some part of Maven --> <server> <id>apache.releases.https</id> <username>APACHEUSERNAME</username> <password>APACHEPASSWORDENCRYPTED</password> </server> |
Maven Publish
Follow Maven Publish Link https://www.apache.org/dev/publishing-maven-artifacts.html to setup repo settings, etc., before you proceed
Code Block | ||
---|---|---|
| ||
#if you don't set this it will fail with the error "gpg: signing failed: Inappropriate ioctl for device" export GPG_TTY=$(tty) #build scalapkg make scalapkg #make sure everything is working as expected make scalarelease-dryrun # Releases to Apache Snapshot make scaladeploy. This will upload to the snapshot repo # Verify that the package has been uploaded in https://repository.apache.org/content/repositories/snapshots/ # Preparing the release will create the new tag, automatically checking in on your behalf.(We have disabled CheckIn) make scalarelease-prepare # this will upload to the staging repository and make it ready for closingtesting make scalarelease-perform |
...
You have to close the repo for it to become available on the Staging Repo, find details here
Once you have tested the staged package, you can promote the package and it will become available on maven central, find details here
Build Linux-CPU
To build for linux-cpu platform, we use the AWS Deep Learning Base AMI that comes built with all the necessary
...
First you need to export the gpg keys you created on Mac and import to the linux host
Install Maven, gnupg, gnupg2 & gnupg-agent
Code Block | ||
---|---|---|
| ||
sudo apt-get install -y maven gnupg gnupg2 gnupg-agent && #soft link gnupg2 to gnupg, Update the below when you find a better solution sudo mv /usr/bin/gpg /usr/bin/gpg.1 && sudo ln -s /usr/bin/gpg2 /usr/bin/gpg |
Export and ship gpg keys you created on the Mac
Code Block | ||
---|---|---|
| ||
gpg --export-secret-keys -a xxx@apache.org > ~/Downloads/keys.asc scp ~/Downloads/keys.asc ubuntu@host:~/keys.asc |
Import and verify they look fine
Code Block | ||
---|---|---|
| ||
gpg --import ~/keys.asc
#note the key id
gpg --list-keys
|
Update the gnupg-agent cachettl, it takes a long time to build on linux and linux-gpu instances and will time out during release if you don't baby sit the build
Code Block | ||
---|---|---|
| ||
mkdir -p ~/.gnupg && touch ~/.gnupg/gpg-agent.conf
#set to 4 hours
echo "default-cache-ttl 14400" > ~/.gnupg/gpg-agent.conf &&
echo "max-cache-ttl 14400" >> ~/.gnupg/gpg-agent.conf
#key id you go when you ran gpg --listkeys
echo "default-key KEYIDFROMABOVE" >> ~/.gnupg/gpg-agent.conf |
Start gpg-agent to use the above
Code Block | ||
---|---|---|
| ||
eval $(gpg-agent --daemon) |
Maven Password Encryption for use in release
...
Build Backend
Follow steps to clone the repo and checkout the correct release branch
Code Block | ||||
---|---|---|---|---|
| ||||
touchcd ~/.m2/settings-security.xml && mvn --encrypt-master-password |
- paste the encrypted master-password you get from above in `~/.m2/settings-security.xml`
Code Block | ||
---|---|---|
| ||
<settingsSecurity>
<master>ENCRYPTEDMASTERPASSWORD</master>
</settingsSecurity> |
- encrypt your password for `apache` login
Code Block | ||
---|---|---|
| ||
cp /usr/share/maven/conf/settings.xml ~/.m2/ && mvn --encrypt-password |
- Paste the below with your apache username and encrypted apache password updated under `<servers>`tag, this will used to publish to apache snapshot and staging in `~/.m2/settings.xml`.
mxnet-$RELEASEVERSION
./scala-package/dev/compile-mxnet-backend.sh linux-x86_64-gpu ./ |
Follow steps from OSX
Build on GPU
Follow all steps for Linux-CPU
Update the nccl location approriately in ~/incubator-mxnet/scala-package/dev/compile-backend.sh
```echo "ADD_LDFLAGS += -L/lib/nccl/cuda-9.0/lib```
Code Block | ||||
---|---|---|---|---|
| ||||
cd ~/mxnet-$RELEASEVERSION
./scala-package/dev/compile-mxnet-backend.sh linux-x86_64-gpu ./ | ||||
Code Block | ||||
| ||||
<!-- To publish a snapshot of some part of Maven -->
<server>
<id>apache.snapshots.https</id>
<username>APACHEUSERNAME</username>
<password>APACHEPASSWORDENCRYPTED</password>
</server>
<!-- To stage a release of some part of Maven -->
<server>
<id>apache.releases.https</id>
<username>APACHEUSERNAME</username>
<password>APACHEPASSWORDENCRYPTED</password>
</server> |
Using Package in Staging Repository
Update your pom file to include the repository info like below:
Code Block | ||||
---|---|---|---|---|
| ||||
<repository> <id>snapshot</id> <url>https://repository.apache.org/content/repositories/staging</url> </repository> </repositories> <dependencies> <dependency> <groupId>org.apache.mxnet</groupId> <artifactId>mxnet-full_2.11-osx-x86_64-cpu</artifactId> <version>1.2.0</version> </dependency> </dependencies |
...
Maven Publish Debugging
If you find that maven release-prepare fails with tag already exists
Code Block | ||
---|---|---|
| ||
git push --force --delete ./ refs/heads/mxnet-parent_2.11-1.2.0 git tag -d mxnet-parent_2.11-1.2.0 |
If you face an issue because of gpg such as ```Enter passphrase: gpg: gpg-agent is not available in this session```
Code Block |
---|
sudo apt-get install gnupg2 sudo mv /usr/bin/gpg /usr/bin/gpg.1 sudo ln -s /usr/bin/gpg2 /usr/bin/gpg |
gpg: signing failed: Inappropriate ioctl for device
```export GPG_TTY=$(tty)```
...
Compiling with USE_SIGNAL_HANDLER throws a segmentation fault on both OSX and Linux platforms
...
Publishing to Maven
...
language | bash |
---|
...