Table of Contents |
---|
Note: Following instructions are applicable for publishing Scala and Java package simultaneously as both bindings are inside the same package and built together.
Publishing Scala Packages to Nexus/Maven
Currently 3 Packages are published to Maven. `mxnet_full_2.11-{platform}` .
Compile Flags Used to build the Scala package
The following flags are used to build the MXNet Backend(derived from mxnet pip compiler flags)
...
MXNet-Scala Linux CPU
...
MXNet-Scala Linux CPU
...
1
(Last published opencv 3.4.1)
...
USE_CUDNN
...
1
(Last published with (CUDA 9.0)
...
0
...
1
(Last published with CUDNN 7.0.5)
...
$ java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
...
ubuntu@ip:~$ java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (Zulu 8.20.0.5-linux64) (build 1.8.0_121-b15)
OpenJDK 64-Bit Server VM (Zulu 8.20.0.5-linux64) (build 25.121-b15, mixed mode)
...
for the following platforms: linux-x86_64-cpu, linux-x86_64-gpu, osx-x86_64-cpu.
Building on OS-X
Setup GPG Keys https://www.apache.org/dev/openpgp.html#key-gen-install-latest-gnupg. Make sure to follow https://www.apache.org/dev/openpgp.html#update to send your key to Apache and onto the public PGP Keyservers. You also need to add your key to both the KEYS file on mxnet github and the KEYS file in the apache committers subversion.
Clone MXNet repo and checkout the release branch
Code Block | ||
---|---|---|
| ||
git clone https://github.com/$YOURFORKapache/incubator-mxnet ~/mxnet-$RELEASEVERSION cd ~/mxnet-$RELEASEVERSION git remote add upstream https://github.com/apache/incubator-mxnet git fetch upstream v1.2.0 && git checkout -b $RELEASEVERISON --track upstream/v1.2.0 git submodule update --init --recursive checkout $RESOURCE_CANDIDATE_TAG |
Install lapack, openblas, opencv packages
Code Block | ||
---|---|---|
| ||
#try opencv3 next time you release
brew install lapack openblas opencv@2 |
Build MXNet backend on OSX
Code Block | ||
---|---|---|
| ||
#This will build the MXNet backend with the tested configuration for publishing MXNet-Scala to Maven. cd ~/mxnet-$RELEASEVERSION scala-package/dev/compile-mxnet-backend.sh osx-x86_64-cpuopencv |
Maven Password Encryption for use in release
...
Encrypt your password for `apache` login`apache` login, note that this is not the passphrase of your gpg-key.
Code Block | ||
---|---|---|
| ||
cp /usr/share/maven/conf/settings.xml ~/.m2/ && mvn --encrypt-password |
...
Code Block | ||
---|---|---|
| ||
<!-- To publish a snapshot of some part of Maven --> <server> <id>apache.snapshots.https</id> <username>APACHEUSERNAME</username> <password>APACHEPASSWORDENCRYPTED</password> </server> <!-- To stage a release of some part of Maven --> <server> <id>apache.releases.https</id> <username>APACHEUSERNAME</username> <password>APACHEPASSWORDENCRYPTED</password> </server> |
Maven Publish
Follow Maven Publish Link https://www.apache.org/dev/publishing-maven-artifacts.html to setup repo settings, etc., before you proceedEdit the file `ci/publish/scala/build.sh` and add the flag `-Pstaging` to the mvn deploy command. Make sure to have your GPG key available on the machine. If you have multiple GPG keys, you can add `-Dgpg.keyname=$DESIRED_KEY_EMAIL` to the mvn deploy command to specify the desired key.
Code Block | ||
---|---|---|
| ||
#if you don't set this it will fail with the error "gpg: signing failed: Inappropriate ioctl for device" export GPG_TTY=$(tty) #build# scalapkg make scalapkg #make sure everything is working as expected make scalarelease-dryrun # Releases to Apache Snapshot make scaladeploy. This will upload to the snapshot repo # Verify that the package has been uploaded in Set MXNet variant to build export mxnet_variant=mkl # Run the build bash ci/publish/scala/build.sh |
Once you have finished, you should be able to find the newly added version on the Apache Nexus at https://repository.apache.org/
...
#stagingRepositories. It will have repository name orgapachemxnet-$RELEASE_NUMBER. Click on the repository and press the close button at the top to indicate that everything has been uploaded. At this point, Nexus will verify that all of the checksums are correct and the signatures are all valid (including that you have properly uploaded your key and are authorized to sign the package). Then, follow the instructions to build the CPU/GPU packages and close them before continuing to the end.
You have to close the repo for it to become available on the Staging Repo, When you close be sure to add the flags you used to build the package like below, You can find how to here
```
Built with flags for OPENCV, OPENBLAS, LAPACK, DIST_KVSTORE, S3
Flattened POM
```
Once you have tested the staged package, you can promote the package and it will become available on maven central, find details here
Build Linux-CPU and Linux-GPU
To build for linux-cpu platform, we use the AWS Deep Learning Base AMI that comes built with all the necessary libraries such as opencv/cudnn etc,a Ubuntu 14.04 instance. Install the necessary dependencies by running the install script found in the MXNet repo `ci/docker/install/ubuntu_publish.sh`.
Export/Import GPG Keys
First you need to export the gpg keys you created on Mac and import to the linux host
...
Code Block | ||
---|---|---|
| ||
sudo apt-get install -y maven gnupg gnupg2 gnupg-agent && #soft link gnupg2 to gnupg, Update the below when you find a better solution sudo mv /usr/bin/gpg /usr/bin/gpg.1gpg1 && sudo ln -s /usr/bin/gpg2 /usr/bin/gpg |
...
Code Block | ||
---|---|---|
| ||
mkdir -p ~/.gnupg && touch ~/.gnupg/gpg-agent.conf #set to 4 hours echo "default-cache-ttl 14400" > ~/.gnupg/gpg-agent.conf && echo "max-cache-ttl 14400" >> ~/.gnupg/gpg-agent.conf #key id you go when you ran gpg --listkeys echo. This isn't working, commenting out #echo "default-key KEYIDFROMABOVE" >> ~/.gnupg/gpg-agent.conf |
...
Code Block | ||
---|---|---|
| ||
eval $(gpg-agent --daemon) |
...
Checkout MXNet
Follow steps to clone the repo and checkout the correct release branch. We recommend using a clean copy of the repo for the CPU and GPU builds
Code Block | ||
---|---|---|
| ||
cd ~/mxnet-$RELEASEVERSION
./scala-package/dev/compile-mxnet-backend.sh linux-x86_64-gpu ./ |
Follow steps from OSX
Build on GPU
Follow all steps for Linux-CPU
Update the nccl location approriately in ~/incubator-mxnet/scala-package/dev/compile-backend.sh
```echo "ADD_LDFLAGS += -L/lib/nccl/cuda-9.0/lib```
Code Block | ||
---|---|---|
| ||
git clone https://github.com/apache/incubator-mxnet ~/mxnet-$RELEASEVERSION cd ~/mxnet-$RELEASEVERSION .git checkout $RESOURCE_CANDIDATE_TAG |
Maven Publish
Edit the file `ci/publish/scala
...
Using Packages in Staging Repository
Update your pom file to include the repository info like below:
Code Block | ||||
---|---|---|---|---|
| ||||
<repositories>
...
<repository>
<id>snapshot</id>
<url>https://repository.apache.org/content/repositories/staging</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.apache.mxnet</groupId>
<artifactId>mxnet-full_2.11-osx-x86_64-cpu</artifactId>
<version>1.2.0</version>
</dependency>
</dependencies> |
Maven Publish Debugging
If you find that maven release-prepare fails with tag already exists
...
language | bash |
---|
/build.sh` and add the flag `-Pstaging` to the mvn deploy command. Make sure to have your GPG key available on the machine. If you have multiple GPG keys, you can add `-Dgpg.keyname=$DESIRED_KEY_EMAIL` to the mvn deploy command to specify the desired key. Alter the script below to set the mxnet_variant to the desired version to build. We currently use `mkl` for the CPU build and `cu92mkl` for the GPU build.
Code Block | ||
---|---|---|
| ||
#if you don't set this it will fail with the error "gpg: signing failed: Inappropriate ioctl for device"
export GPG_TTY=$(tty)
# Set MXNet variant to build
export mxnet_variant=$DESIRED_VARIANT
# Run the build
bash ci/publish/scala/build.sh |
Once you have finished, you should be able to find the newly added version on the Apache Nexus at https://repository.apache.org/#stagingRepositories. It will have repository name orgapachemxnet-$RELEASE_NUMBER. Click on the repository and press the close button at the top to indicate that everything has been uploaded. At this point, Nexus will verify that all of the checksums are correct and the signatures are all valid (including that you have properly uploaded your key and are authorized to sign the package). Then, follow the instructions to build the CPU/GPU packages and close them before continuing to the end.
After Publishing
At this point, you should have three packages that are closed on the apache nexus. You should now have other members of the community verify these closed packages. You can find the packages for download in a Maven repository format (which can be included following a process like https://maven.apache.org/guides/mini/guide-multiple-repositories.html) at a URL similar to https://repository.apache.org/service/local/repositories/orgapachemxnet-$BUILD_NUM/content. The $BUILD_NUM can be found as the name of the staged repository on Nexus. Note that you would have one of these repositories for each of the three builds. These URLs should be shared to help the community members verify the packages and to help build the Clojure release jars.
Following the results of the vote, you should use Nexus to finish the release process. When you go to the staging repository and view the package, you will have a release button and a drop button. Click the release button if the vote passed to finalize the release and make the package available through Maven Central (this can not be reversed). Click the Drop button if the vote failed as the staged package will not be released. If you have to make a release candidate, it will create new staged items with incremented staging numbers instead of changing the old ones.
Staging Verification Checklist
- Test Inference examples on macOS
- Test Inference examples on Ubuntu CPU/GPU not on DL AMI
- Test Training examples on Ubuntu GPU, not on DL AMI.
- Test Training and Inference on Centos, not on DL AMI
Troubleshooting
...
If you face an issue because of gpg such as ```Enter passphrase: gpg: gpg-agent is not available in this session```. Without a session, you will need to enter your passphrase for each file that is to be signed. In total, you would need to enter it 6 times.
Code Block |
---|
sudo apt-get install gnupg2 sudo mv /usr/bin/gpg /usr/bin/gpg.1 sudo ln -s /usr/bin/gpg2 /usr/bin/gpg |
gpg: signing failed: Inappropriate ioctl for device
```export GPG_TTY=$(tty)```
Compiling with USE_SIGNAL_HANDLER throws a segmentation fault on both OSX and Linux platforms
Pre-Release/Post-Release
Release Checklist
- Test Inference examples on macOS
- Test Inference examples on Ubuntu CPU/GPU not using the DL AMI
- Test Training examples on Ubuntu GPU, not on DL AMI.
Open Issues with the existing Packages
...
Error during release
If you have an error during the release and it makes it to the Maven part of the build process, you only need to run the mvn deploy command from `ci/publish/scala/build.sh` instead of re-running the whole build. However, the GPU build requires copying some environment variables that are created during the build. Those can be found in `tools/staticbuild/build.sh` and then `tools/setup_gpu_build_tools.sh`.
Maven Publish fails with: Return code is: 401, ReasonPhrase: Unauthorized.
Make sure you are using the encrypted password of your apache login and not gpg passphrase
Released Artifacts do not show up in Maven Search after releasing through nexus.
Verify that the released artifacts of the version you are publishing are synced to Apache release repository and Maven Central Repository, It might be that Maven Search is not picking up.
I see two repos in the staging repo for packages(example: OSX)
do a make clean and make scala clean before you build, it might be that the size of the Jar is too big and its splitting across two repos.
Apache Release Repo
https://repository.apache.org/content/repositories/releases/org/apache/mxnet/
Maven Central Repo
http://central.maven.org/maven2/org/apache/mxnet/mxnet-full_2.11-osx-x86_64-cpu/1.2.1/
http://central.maven.org/maven2/org/apache/mxnet/mxnet-full_2.11-linux-x86_64-cpu/1.2.1
http://central.maven.org/maven2/org/apache/mxnet/mxnet-full_2.11-linux-x86_64-gpu/1.2.1/
...