Overview

The release manager role in Apache MXNet means you are responsible for a few different things:

1. Preparing for release candidates:
2. cutting a release branch
1. informing the community of timing
working with component leads to clean up backlog
1. prepare release notes (bug fixes, features added)
2. making code changes
in that branch
1. with necessary version updates
2. cutting a release candidate
3. Running the voting process for a release:creating release candidates using automated tooling
1. calling votes and triaging issues, revote if necessary
4. Finalizing and posting a release:
1. marking & publishing the official release
2. updating the MXNet website
writing release notes
2. announcing the release

Preparing MXNet for Release

The main step towards preparing a release is to create a release branch. This is done via standard Git branching mechanism and should be announced to the community once the branch is created. A Jenkins jobs for the release branch has to be setup once the branch is in created and test results must be published on the mailing list (consult Ly Nguyen for help with this).

Next, ensure that all MXNet versions are correct in the code base on the release branch (TODO see this example commit). You should grep through the codebase to find all instances of the version string. Some known places to change are:

• TODO

Finally, update CHANGES.txt with this script in the MXNet repository. CHANGES.txt captures all the patches that have made it into this release candidate since the last release.

TODO

$ export MXNET_HOME=<your Spark home>
$ cd mxnet
# Update release versions
$ vim dev/create-release/generate-changelist.py
$ dev/create-release/generate-changelist.py

...

Branch creation announcement template

...

Pre-requisites:

1. You are a current project committer
2. You’ve gained access to configure build jobs at builds.apache.org as a part of the jobsadmin group (ask a PMC mentor to add you)

1. Preparing for release candidates:

1. prepare release notes (bug fixes, features added)

Draft release notes which should include the list of new features and bug fixes, along with limitations, known issues, credits, etc.

2. informing the community of timing

After every release, the community should decide on a list of features to deliver for the following release. When the clear target date for a release has been estimated, the release manager should inform the community of the release schedule. All features which the community wishes to include in the release should be merged and tested successfully before the date of the first release candidate. Below is a template for the information:

To: dev@mxnet.apache.org

...

Subject:
...
 [VOTE] Release plan - MXNET 0.11
We are starting the process to prepare for MXNET $release release. We have drafted release notes ($link) to cover the tasks under this release.
A release candidate will be cut on $utc_date and voting will commence $utc_date_vote0 - $utc_date_vote1. If you have any additional features in progress and would like to include it in this release, please assure they have been merged by $utc_date-24 comment so we may update the release notes.
Feel free to add any other comments/suggestions.
Thanks,
$RM
3. making code changes with necessary version updates
Bump up the version number. An example can be followed here: https://github.com/apache/incubator-mxnet/pull/6462
Update NEWS and README. An example can be followed here: https://github.com/apache/incubator-mxnet/pull/6471
Create a RSA GPG key of length 4096, upload it to the public server, and add it to the KEYS file (do this process once for each release manager): https://github.com/apache/incubator-mxnet/blob/master/KEYS See more detailed instructions on creating the key here: https://www.apache.org/dev/openpgp.html#generate-key
4. cutting a release candidate
...
If this is not the first RC, then make sure that all issues on github related to previous RC have been solved since the last RC are marked as FIXED in this release version.
A possible protocol for this is to mark such issues as FIXED in next maintenance release. For example if you are cutting RC for 1.0.2, mark such issues as FIXED in 1.0.3.
When cutting new RC, find all the issues that are marked as FIXED for next maintenance release, and change them to the current release.
Verify from gitlog whether they are actually making it in the new RC or not.
The process of cutting a release candidate has been automated via Jenkins. There are Jenkins jobs that can tag a release candidate and create various packages based on that candidate. The recommended process is to ask the previous release manager to walk you through the Jenkins jobs or consult with Ly Nguyen.
Call a Vote on the Release Candidate
The release voting takes place on the Apache MXNet developers list (the PMC is voting). Look at past voting threads to see how this proceeds. The email should follow this format.
...
The release candidate should contain all planned features, bug fixes, and code changes above. The release candidate commit should have passed the merge build, pip builds, nightly tests. Docs should also be manually built locally & checked for correctness. TODO: how to retrieve the commit hash.
Retrieve the commit hash and create artifacts for the release and push to the dist folder
apache-mxnet-src-#.#.#-rc0-incubating.tar.gz: Checkout the commit source and archive into this file, replace “#.#.#” part of the name with the release tag
apache-mxnet-src-#.#.#-rc0-incubating.tar.gz.asc: Create OpenGPG compatible ASCII armored detached signature 
http://www.apache.org/dev/release-signing.
...
html#openpgp-ascii-detach-sig
apache-mxnet-src-#.#.#-rc0-incubating.tar.gz.md5: Create MD5 checksum http://www.apache.org/dev/release-signing.
...
Voting email template
 Code Block
language
xml
title
Voting email template
collapse
true
To: dev@mxnet.apache.org
Subject: [VOTE] Release plan - MXNET 0.11

We are starting the process to prepare for MXNET $release release. I have created release plan ($link) to cover the tasks under this release.

If you have any tasks in progress and would like to include it in this release, please follow the process to do so.

Feel free to comment on the github if you have any comments/suggestions.

Thanks,
$RM
...
 Code Block
language
xml
title
Voting result template
collapse
true
To: dev@mxnet.apache.org
Subject: [RESULTS] [VOTE] Release MXNet version 0.10.0.0

This vote passes with 7 +1 votes (3 bindings) and no 0 or -1 votes.

+1 votes
PMC Members:
* $Name
* $Name 
* $Name

Committers:
* $Name
* $Name

Community:
* $Name
* $Name

0 votes
* No votes

-1 votes
* No votes

Vote thread:
https://lists.apache.org/list.html?commits@mxnet.apache.org/

I'll continue with the release process and the release announcement will follow in the next few days.

$RM
...
Be Careful!
THIS STEP IS IRREVERSIBLE so make sure you selected the correct staging repository. Once you move the artifacts into the release folder, they cannot be removed.
After the vote passes, create the release. To upload the binaries, you have to first upload them to the dev directory in the Apache Distribution repo, and then move the binaries from dev directory to release directory. This “moving” is the only way you can add stuff to the actual release directory.
 
...
language
bash
title
Release template
collapse
true
...
html#md5
apache-mxnet-src-#.#.#-rc0-incubating.tar.gz.sha1: Create SHA checksum http://www.apache.org/dev/release-signing.html#sha-checksum
KEYS: Add your public signing key to the dist folder
#.#.#-incubating.RC0 folder: Create a folder for this RC in the dist folder
# Checkout the Apache directory in Apache distribution SVN "dev" repo
...
$ svn co https://dist.apache.org/repos/dist/dev/
...
incubator/
...
mxnet
  
# Make directory for this RC in the above directory
...
mkdir 
...
#.
...
#.#-rc0
# Move artifacts into this folder
TODO
$ svn add mxnet-#.#.#-rc0
$ svn commit -m "Add 
...
mxnet-
...
#.
...
#.
...
#-
...
rc0" --username "
...
lxn2"
...
  
...
# Move the sub-directory in "dev" to the
...
# corresponding directory in "release"
...
$ export SVN_EDITOR=vim
...
$ svn mv https://dist.apache.org/repos/dist/dev/
...
incubator/mxnet/
...
mxnet-
...
#.
...
#.
...
#-
...
rc0 
...
http://
...
www.apache.org/
...
dist/
...
incubator/
...
mxnet/
...
mxnet-
...
#.
...
#.#-rc0
Verify that the resources are present in 
...
http://www.apache.org/dist/incubator/mxnet. It may take a while for them to be visible. This will be mirrored throughout the Apache network. There are a few remaining steps.
...
        
3.    Validate release package
As per the Apache documentation, verify that the release candidate artifacts satisfy the following:
PGP signatures and SHA256/MD5 checksum verification
Example (performed on a Macbook Pro: brew install gpg2 coreutils):
$ brew install gpg2 coreutils
 $ gpg2 --import ../KEYS
 $ gpg2 --verify apache-mxnet-src-#.#.#-rc-incubating.tar.gz.asc

 $ gsha256sum --check apache-mxnet-src-#.#.#-rc-incubating.tar.gz.sha256 
 $ gmd5sum --check apache-mxnet-src-#.#.#-rc-incubating.tar.gz.md5
Build is successful
DISCLAIMER is correct, filenames include “incubating”
LICENSE and NOTICE files are correct and dependency licenses are acceptable
LICENSE and NOTICE files at the root of the artifact directory must only reflect the contents of the artifact in which they are contained.
See:
LICENSE file requirements
LICENSE requirements for distribution artifacts with multiple licenses
NOTICE file requirements (Check Copyright year)
Apache Legal
Acceptable and Unacceptable Dependency Licenses
All source files have license headers where appropriate, RAT checks pass
The provenance of all source files is clear (ASF or software grants)
4.    Retrieve the commit hash and tag the release candidate:
Go to the GitHub repo’s “releases” tab
Click “Draft a new release”
Provide the release tag in the form of “v#.#.#-rc0” where 0 means it’s the first release candidate
Select the commit by clicking Target: master > Recent commits > $commit_hash
Copy and paste NEWS change into the description box
Select “This is a pre-release”
Upload the .tar.gz
2. Running the voting process for a release:
1. calling votes and triaging issues, revote if necessary
The full voting process is two phases: 3 days on dev@mxnet.apache.org, 3 days on http://general@incubator.apache.org.
For the dev@ vote, there must be at least 3 +1 votes and more +1 votes than -1 votes. Once that quorum has been reached, the vote then moves onto the general@ vote, where anyone can contribute a vote, but only MXNet PMC votes are binding. To pass, there must be 3 +1 PMC votes and more +1 votes than -1 votes. Send out a notice to vote. Email template:
 
To: dev@mxnet.apache.org
Subject: [VOTE] Release MXNet version 0.11.0
 
This is the vote to release Apache MXNet (incubating) version 0.11.0. Voting will start now and close $utc_now+72.
Link to release notes: <Link>
Link to release candidate 0.11.0-rc0: <Link>
Link to pip build: <Link>
 
Please test and vote accordingly:
+1 = approve
+0 = no opinion
-1 = disapprove (provide reason)
 
Once each vote is complete, send out a results email:
 
To: dev@mxnet.apache.org
Subject: [RESULTS] [VOTE] Release MXNet version 0.11.0
 
This vote passes with 7 +1 votes (3 bindings) and no 0 or -1 votes.
+1 votes
PMC Members:
* $Name
* $Name 
* $Name
 
Committers:
* $Name
* $Name
 
Community:
* $Name
* $Name
 
0 votes
* No votes
 
-1 votes
* No votes
 
Vote thread:
https://lists.apache.org/list.html?commits@mxnet.apache.org/
I'll continue with the release process and the release announcement will follow in the next few days.
 
$RM
 
Should any vote fail to reach quorum, the release manager should triage the issues, create GitHub issues, and move to fix the issues. Once fixed, mark the issues as FIXED in the new README, release notes, and description when drafting a new release for a second release candidate. Repeat until the vote passes.
3. Finalizing and posting a release:
1. marking & publishing the official release
Once the release candidate passes the vote, we draft the official release on GitHub and rename the release candidate as the official release.
Draft official release on GitHub:
Go to the GitHub repo’s “releases” tab
Click “Draft a new release”
Provide the release tag in the form of “v#.#.#”
Select the commit by clicking Target: master > the passing release candidate tag
Copy and paste NEWS change into the description box
Download the nightly pip build, rename, and publish:
TODO
Rename and publish the source package:
TODO
Build, test, and publish the Docker images:
TODO
test multinode and multi gpu (kvstore=1)
2. updating the MXNet website
Now that a new version has been drafted on GitHub with source and pip packages published, we need to update the MXNet website. Go to builds.apache.org and find job “incubator-mxnet-build-site”. Click “Build with Parameters”, select the release tag, and build. The docs will be published to incubator.mxnet.apache.org.
Manually update the website’s download links:
TODO: link to Apache mirror source, pgp, sha, md5
3. announcing the release
Remove Old Releases from Mirror Network
MXNet always keeps two releases in the mirror network: the most recent release on the current and previous branches. To delete older versions simply use TODO. The TODO file in the website ... must also be updated to reflect the changes. For instance, the two releases should be 1.1.1 and 1.0.2, but not 1.1.1 and 1.1.0.
TODO
$ svn rm https://dist.apache.org/repos/dist/release/spark/spark-1.1.0
Update the MXNet Apache Repository
Check out the tagged commit for the release candidate that passed and apply the correct version tag.
 Code Block
language
bash
title
UpdateMxnetApacheRepo
collapse
true
TODO
$ git checkout v1.1.1-rc2 # the RC that passed
$ git tag v1.1.1
$ git push apache v1.1.1
 
# Verify that the tag has been applied correctly
# If so, remove the old tag
$ git push apache :v1.1.1-rc2
$ git tag -d v1.1.1-rc2
...
In general, the rules are as follows:
grep through the repository to find such occurrences
References to the version just released. Upgrade them to next release version. If it is not a documentation related version (e.g. inside mxnet/docs/ or inside ...), add -SNAPSHOT to the end.
References to the next version. Ensure these already have -SNAPSHOT.
Update the MXNet Website
The website repository is located at TODO. Ensure the docs were generated with the PRODUCTION=1 environment variable - TODO
# TODO Add instructions how to build docs and update the website
Next, update the rest of the MXNet. See how the previous releases are documented. In particular, have a look at the changes to the *.md files in this commit.
TODO
# Add git cmds to update 
Then, create the release notes. The contributors list can be automatically generated through TODO this script. It accepts the tag that corresponds to the current release and another tag that corresponds to the previous (not including maintenance release). For instance, if you are releasing MXNet 1.2.0, set the current tag to v1.2.0-rc2 and the previous tag to v1.1.0. Once you have generated the initial contributors list, it is highly likely that there will be warnings about author names not being properly translated. To fix this, run this other script, which fetches potential replacements from Github.
Additionally, if you wish to give more specific credit for developers of larger patches, you may use the the following commands to identify large patches. Extra care must be taken to make sure commits from previous releases are not counted since git cannot easily associate commits that were back ported into different branches.
 Code Block
language
bash
title
UpdatePatchAndContributor
collapse
true
TODO
 
# TODO Update to reflect MXNet process
$ cd release-spark/dev/create-release
# Set RELEASE_TAG and PREVIOUS_RELEASE_TAG
$ vim generate-contributors.py
# Generate initial contributors list, likely with warnings
$ ./generate-contributors.py
# Set JIRA_USERNAME, JIRA_PASSWORD, and GITHUB_API_TOKEN
$ vim release-spark/dev/translate-contributors.py
# Translate names generated in the previous step, reading from known_translations if necessary
$ ./translate-contributors.py

# Determine PR numbers closed only in the new release
$ git log v1.1.1 | grep "Closes #" | cut -d " " -f 5,6 | grep Closes | sort > closed_1.1.1
$ git log v1.1.0 | grep "Closes #" | cut -d " " -f 5,6 | grep Closes | sort > closed_1.1.0
$ diff --new-line-format="" --unchanged-line-format="" closed_1.1.1 closed_1.1.0 > diff.txt
 
# Grep expression with all new patches
$ EXPR=$(cat diff.txt | awk '{ print "\\("$1" "$2" \\)"; }' | tr "\n" "|" | sed -e "s/|/\\\|/g" | sed "s/\\\|$//")
 
# Contributor list
$ git shortlog v1.1.1 --grep "$EXPR" > contrib.txt
 
# Large patch list (300+ lines)
$ git log v1.1.1 --grep "$expr" --shortstat --oneline | grep -B 1 -e "[3-9][0-9][0-9] insert" -e "[1-9][1-9][1-9][1-9] insert" | grep SPARK > large-patches.txt
 
Then, update the downloads page, and then the main page with a news item.
Create an Announcement
Once everything is working (website docs, website changes, 24 hours after upload) create an announcement on the website and then send an e-mail to the mailing list. http://www.apache.org/legal/release-policy.html#release-announcements
Send an email to announce@apache.org, general@incubator.apache.org, and dev@mxnet.incubator.apache.org with the subject: [ANNOUNCE] Apache MXNet <release number> Release and a body along the lines of:
Apache MXNet (incubating) Project Team is proud to announce Apache
 MXNet <release version> has been released.
*Download Link*: <Link>

 *About this release*
 <Link to release notes>

 *HAWQ Resources:*

    - JIRA: https://github.com/apache/incubator-mxnet/issues
    - Wiki: https://cwiki.apache.org/confluence/display/MXNET
    - Mailing list(s): dev@mxnet.incubator.apache.org 

 *Know more about HAWQ: incubator.mxnet.apache.org 

 - Apache MXNet(incubating) Team

 =====
 *Disclaimer*
Apache MXNet (incubating) is an effort undergoing incubation at The
 Apache Software Foundation (ASF), sponsored by the name of Apache
 Incubator PMC. Incubation is required of all newly accepted
 projects until a further review indicates that the
 infrastructure, communications, and decision making process have
 stabilized in a manner consistent with other successful ASF
 projects. While incubation status is not necessarily a reflection
 of the completeness or stability of the code, it does indicate
 that the project has yet to be fully endorsed by the ASF.
 
 
Enjoy an adult beverage (or bubble tea) of your choice, and congratulations on making a MXNet release.