Page History

...

Currently 3 Packages are published to Maven. `mxnet_full_2.11-{platform}` .

Compile Flags Used to build the Scala/Java package

The following flags are used to build the MXNet Backend(derived from mxnet pip compiler flags)

...

MXNet-Scala Linux CPU

...

MXNet-Scala Linux GPU

...

1

(Last published opencv 3.4.1)

...

USE_CUDNN

...

1

(Last published with (CUDA 9.2)

...

0

...

1

(Last published with CUDNN 7.0.5)

...

$ java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

...

ubuntu@ip:~$ java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (Zulu 8.20.0.5-linux64) (build 1.8.0_121-b15)
OpenJDK 64-Bit Server VM (Zulu 8.20.0.5-linux64) (build 25.121-b15, mixed mode)

...

for the following platforms: linux-x86_64-cpu, linux-x86_64-gpu, osx-x86_64-cpu.

Building on OS-X

Setup GPG Keys https://www.apache.org/dev/openpgp.html#key-gen-install-latest-gnupg. Make sure to follow https://www.apache.org/dev/openpgp.html#update to send your key to Apache and onto the public PGP Keyservers. You also need to add your key to both the KEYS file on mxnet github and the KEYS file in the apache committers subversion.

Clone MXNet repo and checkout the release branch

git clone https://github.com/$YOURFORKapache/incubator-mxnet ~/mxnet-$RELEASEVERSION
cd ~/mxnet-$RELEASEVERSION
git remote add upstream https://github.com/apache/incubator-mxnet
git fetch upstream v1.2.0 && git checkout -b $RELEASEVERISON --track upstream/v1.2.0
git submodule update --init --recursive
checkout $RESOURCE_CANDIDATE_TAG

Install lapack, openblas, opencv packages

#try opencv3 next time you release
brew install lapack openblas opencv

Build MXNet backend on OSX

#This will build the MXNet backend with the tested configuration for publishing MXNet-Scala to Maven.
cd ~/mxnet-$RELEASEVERSION
scala-package/dev/compile-mxnet-backend.sh osx-x86_64-cpu ./

Maven Password Encryption for use in release

...

    <!-- To publish a snapshot of some part of Maven -->
    <server>
      <id>apache.snapshots.https</id>
      <username>APACHEUSERNAME</username>
      <password>APACHEPASSWORDENCRYPTED</password>
    </server>
    <!-- To stage a release of some part of Maven -->
    <server>
      <id>apache.releases.https</id>
      <username>APACHEUSERNAME</username>
      <password>APACHEPASSWORDENCRYPTED</password>
    </server>

Maven Publish

Follow Maven Publish Link https://www.apache.org/dev/publishing-maven-artifacts.html to setup repo settings, etc., before you proceedEdit the file `ci/publish/scala/build.sh` and add the flag `-Pstaging` to the mvn deploy command. Make sure to have your GPG key available on the machine. If you have multiple GPG keys, you can add `-Dgpg.keyname=$DESIRED_KEY_EMAIL` to the mvn deploy command to specify the desired key.

#if you don't set this it will fail with the error "gpg: signing failed: Inappropriate ioctl for device"
export GPG_TTY=$(tty)
#build# scalapkg
make scalapkg
#make sure everything is working as expected
make scalarelease-dryrun
# Releases to Apache Snapshot
make scaladeploy. This will upload to the snapshot repo
# Verify that the package has been uploaded in Set MXNet variant to build
export mxnet_variant=mkl
# Run the build
bash ci/publish/scala/build.sh

Once you have finished, you should be able to find the newly added version on the Apache Nexus at https://repository.apache.org/

...

You have to close the repo for it to become available on the Staging Repo, When you close be sure to add the flags you used to build the package like below, You can find how to here

```

Built with flags for OPENCV, OPENBLAS, LAPACK, DIST_KVSTORE, S3
Flattened POM

```#stagingRepositories. It will have repository name orgapachemxnet-$RELEASE_NUMBER. Click on the repository and press the close button at the top to indicate that everything has been uploaded. At this point, Nexus will verify that all of the checksums are correct and the signatures are all valid (including that you have properly uploaded your key and are authorized to sign the package). Then, follow the instructions to build the CPU/GPU packages and close them before continuing to the end.

Once you have tested the staged package, you can promote the package and it will become available on maven central, find details here

Build Linux-CPU and Linux-GPU

To build for linux-cpu platform, we use the AWS Deep Learning Base AMI that comes built with all the necessary libraries such as opencv/cudnn etc,a Ubuntu 14.04 instance. Install the necessary dependencies by running the install script found in the MXNet repo `ci/docker/install/ubuntu_publish.sh`.

Export/Import GPG Keys

First you need to export the gpg keys you created on Mac and import to the linux host

...

eval $(gpg-agent --daemon)

...

Checkout MXNet

Follow steps to clone the repo and checkout the correct release branch. We recommend using a clean copy of the repo for the CPU and GPU builds

cd ~/mxnet-$RELEASEVERSION
./scala-package/dev/compile-mxnet-backend.sh linux-x86_64-gpu ./

Follow steps from OSX

Build on GPU

Follow all steps for Linux-CPU

Update the nccl location approriately in ~/incubator-mxnet/scala-package/dev/compile-backend.sh

```echo "ADD_LDFLAGS += -L/lib/nccl/cuda-9.2/lib```

git clone https://github.com/apache/incubator-mxnet ~/mxnet-$RELEASEVERSION
cd ~/mxnet-$RELEASEVERSION
.git checkout $RESOURCE_CANDIDATE_TAG

Maven Publish

Edit the file `ci/publish/scala

...

Using Packages in Staging Repository

Update your pom file to include the repository info like below:

<repositories>
...  
<repository>
    <id>snapshot</id>
    <url>https://repository.apache.org/content/repositories/staging</url>
  </repository>
</repositories>

<dependencies>
  <dependency>
    <groupId>org.apache.mxnet</groupId>
    <artifactId>mxnet-full_2.11-osx-x86_64-cpu</artifactId>
    <version>1.2.0</version>
  </dependency>
</dependencies>

Maven Publish Debugging 

If you find that maven release-prepare fails with tag already exists

...

/build.sh` and add the flag `-Pstaging` to the mvn deploy command. Make sure to have your GPG key available on the machine. If you have multiple GPG keys, you can add `-Dgpg.keyname=$DESIRED_KEY_EMAIL` to the mvn deploy command to specify the desired key. Alter the script below to set the mxnet_variant to the desired version to build. We currently use `mkl` for the CPU build and `cu92mkl` for the GPU build.

#if you don't set this it will fail with the error "gpg: signing failed: Inappropriate ioctl for device"
export GPG_TTY=$(tty)
# Set MXNet variant to build
export mxnet_variant=$DESIRED_VARIANT
# Run the build
bash ci/publish/scala/build.sh

Once you have finished, you should be able to find the newly added version on the Apache Nexus at https://repository.apache.org/#stagingRepositories. It will have repository name orgapachemxnet-$RELEASE_NUMBER. Click on the repository and press the close button at the top to indicate that everything has been uploaded. At this point, Nexus will verify that all of the checksums are correct and the signatures are all valid (including that you have properly uploaded your key and are authorized to sign the package). Then, follow the instructions to build the CPU/GPU packages and close them before continuing to the end.

After Publishing

At this point, you should have three packages that are closed on the apache nexus. You should now have other members of the community verify these closed packages. You can find the packages for download in a Maven repository format (which can be included following a process like https://maven.apache.org/guides/mini/guide-multiple-repositories.html) at a URL similar to https://repository.apache.org/service/local/repositories/orgapachemxnet-$BUILD_NUM/content. The $BUILD_NUM can be found as the name of the staged repository on Nexus. Note that you would have one of these repositories for each of the three builds. These URLs should be shared to help the community members verify the packages and to help build the Clojure release jars.

Following the results of the vote, you should use Nexus to finish the release process. When you go to the staging repository and view the package, you will have a release button and a drop button. Click the release button if the vote passed to finalize the release and make the package available through Maven Central (this can not be reversed). Click the Drop button if the vote failed as the staged package will not be released. If you have to make a release candidate, it will create new staged items with incremented staging numbers instead of changing the old ones. 

Staging Verification Checklist

• Test Inference examples on macOS
• Test Inference examples on Ubuntu CPU/GPU not on DL AMI
• Test Training examples on Ubuntu GPU, not on DL AMI.
• Test Training and Inference on Centos, not on DL AMI

Troubleshooting

...

If you face an issue because of gpg such as  ```Enter passphrase: gpg: gpg-agent is not available in this session```. Without a session, you will need to enter your passphrase for each file that is to be signed. In total, you would need to enter it 6 times.

sudo apt-get install gnupg2
sudo mv /usr/bin/gpg /usr/bin/gpg.1
sudo ln -s /usr/bin/gpg2 /usr/bin/gpg

gpg: signing failed: Inappropriate ioctl for device

```export GPG_TTY=$(tty)```

...

Error during release

If you have an error during the release and it makes it to the Maven part of the build process, you only need to run the mvn deploy command from `ci/publish/scala/build.sh` instead of re-running the whole build. However, the GPU build requires copying some environment variables that are created during the build. Those can be found in `tools/staticbuild/build.sh` and then `tools/setup_gpu_build_tools.sh`.

Maven Publish fails with: Return code is: 401, ReasonPhrase: Unauthorized.

Make sure you are using the encrypted password of your apache login and not gpg passphrase

Released Artifacts do not show up in Maven Search after releasing through nexus.

Verify that the released artifacts of the version you are publishing are synced to Apache release repository and Maven Central Repository, It might be that Maven Search is not picking up.

...

http://central.maven.org/maven2/org/apache/mxnet/mxnet-full_2.11-osx-x86_64-cpu/1.2.1/
http://central.maven.org/maven2/org/apache/mxnet/mxnet-full_2.11-linux-x86_64-cpu/1.2.1
http://central.maven.org/maven2/org/apache/mxnet/mxnet-full_2.11-linux-x86_64-gpu/1.2.1/

Release Checklist

• Test Inference examples on macOS
• Test Inference examples on Ubuntu CPU/GPU not on DL AMI
• Test Training examples on Ubuntu GPU, not on DL AMI.
• Test Training and Inference on Centos, not on DL AMI

Open Issues with the existing Packages

...