This document details how to release artifacts to the staging repository. It is currently a manual process, but will be automated in the future.
There are three architectures supported OSX cpu, Linux cpu, and Linux gpu. A build and deploy must be done for each to publish the 3 artifacts.
OSX CPU
Creating the artifact and testing it locally
First we need to modify the project file to create the correct artifact and test to make sure it is good.
- Check out the tag of the release with
- git tag —list - to list all the tags
- git checkout tags/<tag_name> -b <branch_name>
- git checkout tags/<tag_name> -b <branch_name>
- Go into the clojure directory
- cd contrib/clojure
- cd contrib/clojure
- In the project.clj file, make the following edits
- The main scala dependency jar must be edited to be the correct release and match the architecture [org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.3.0"]
- The main project name must be changed to match the artifact that it will produce. Add -osx-cpu to the name and rename the SNAPSHOT
- Example: (defproject org.apache.mxnet.contrib.clojure/clojure-mxnet-osx-cpu "1.3.0"
- Make sure a license and url key are present (will be put into a PR shortly)
:url "https://github.com/apache/incubator-mxnet"
:license {:name "Apache License"
:url "http://www.apache.org/licenses/LICENSE-2.0"}
- From the command line, verify that the Scala dependency is downloaded and the tests are run by running lein test
The tests should run without error
- From the command line, run lein install, which will create the jar in the target directory and install it in your local ~/.m2 directory
- Verify the jar structure in target and make sure it looks right
- Optionally, you can test this by
- cd examples/imclassification
- do lein run to use the jar and run the example
Deploying the artifact to staging
Now that we know we are producing a good jar, we can move to deploy it. There are two authentication parts to deploying, your ASF login used to login to the repository and your GPG signature that is used to sign the release.
The leiningen build tools handles configuration for both these parts. Good resources to review are:
https://github.com/technomancy/leiningen/blob/master/doc/DEPLOY.md
https://github.com/technomancy/leiningen/blob/master/doc/GPG.md
The first thing you will need is to create a credentials file for leiningen - this will be encypted with gpg
- Create a file ~./lein/credentials.clj file with the contents :
- {#"staging" {:username "user"
:password "pass"}}
- {#"staging" {:username "user"
- Encypt that file with gpg --sign ~/.lein/credentials.clj
- You can test decrypting with gpg -d ~/.lein/credentials.clj.gpg
- Delete the original unencrypted ~/.lein/credentials.clj
The next thing you will need is to modify the project.clj file to have the ASF deploy staging repo and also point to your ASF gpg key.
- Add the following to your file
:deploy-repositories [["staging" {:sign-releases true
:url "https://repository.apache.org/service/local/staging/deploy/maven2"
:creds :gpg
:signing {:gpg-key "your key"} ;; this can be email of key-id
:snapshots true
:checksum :fail
:update :always
:releases {:checksum :fail :update :always}}]]if it does not prompt for passphrase (known bug in OSX) - you can run
gpg --quiet --batch --decrypt ~/.lein/credentials.clj.gpg
Now you can run the deploy with lein deploy staging. You will be prompted for you gpg passphrase.
For automatic deploys in the future, we can look into more why this is prompting. It does not prompt for linux
Locate and Examine the Staging Repository
DO NOT PUSH RELEASE - IT WILL SEND IT TO RELEASES AND MAVEN CENTRAL!
If all went well with the deploy step, you need to login to the repository and verify the upload and then close it. See this document https://central.sonatype.org/pages/releasing-the-deployment.html#locate-and-examine-your-staging-repository for more details.
If you run into an error closing, it will show up here. For example, your apache gpg public key needs to be uploaded here http://pgp.mit.edu:11371/ or at another public server to verify the release.
Final testing of downloading the jar from staging and running example
To give the jar a final test, you can delete it from your local ~/.m2 cache and try to run the example again with
- cd examples/imclassification
- add the repository in the project.clj to pull from staging
:repositories [["staging-download" {:url "https://repository.apache.org/content/groups/staging"
:snapshots true
:checksum :fail
:update :always
:releases {:checksum :fail :update :always}}]]
- do lein run to use the jar and run the example - You should see the dependency download from staging
Retrieving org/apache/mxnet/contrib/clojure/clojure-mxnet-osx-cpu/1.3.0/clojure-mxnet-osx-cpu-1.3.0.pom from staging-download
Linux CPU
To build for linux-cpu platform and gpu platforms, we use the AWS Deep Learning Base AMI that comes built with all the necessary
libraries such as opencv/cudnn etc,.
Export/Import GPG Keys
First you need to export the gpg keys you created on Mac and import to the linux host
Install Maven, gnupg, gnupg2 & gnupg-agent
|
Export and ship gpg keys you created on the Mac
|
Import and verify they look fine
|
Update the gnupg-agent cachettl, it takes a long time to build on linux and linux-gpu instances and will time out during release if you don't baby sit the build
|
Start gpg-agent to use the above
|
Creating the artifact and testing it locally
Follow the same instructions as for OSX but use clojure-mxnet-linux-cpu for the project name
Deploying the artifact to staging
Follow the same instructions as for OSX.
If you run into trouble with it prompting you or not prompting you for signing key use this:
export GPG_TTY=$(tty)
Locate and Examine the Staging Repository
same process as OSX
Final testing of downloading the jar from staging and running example
Same process as OSX with correct linux-cpu artifact
Linux GPU
You can do this in the same instance as the linux cpu since you have all the keys setup
Creating the artifact and testing it locally
Follow the same instructions as for OSX but use clojure-mxnet-linux-gpu for the project name
Deploying the artifact to staging
Follow the same instructions as for OSX but you won't need to recreate your credentials.clj.gpg file
If you run into trouble with it prompting you or not prompting you for signing key use this:
export GPG_TTY=$(tty)
Locate and Examine the Staging Repository
same process as OSX
Final testing of downloading the jar from staging and running example
Same process as OSX with correct linux-gpu artifact
What to do if you accidentally release the jar
- Don't panic
- Mail dev list about it and find out how they want to proceed
- To clean up you will need to coordinate with infra
- Create an infra ticket - to ask for the cleanup to repository.apache.org
- Reference the dev list thread
- Create a ticket to issues.sonatype.org to delete it from central
- Here is an unfortunate example
- Create an infra ticket - to ask for the cleanup to repository.apache.org
- Figure out what steps to do to prevent it in the future