Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Note: From version 1.3 all branches are created with a name v1.3.x to make patch releases easier. For patch releases cutting a new branch is not necessary. Fixes for patch releases should be merged to v#.#.x branch and latest commit should be tagged with an exact version number, ie v1.3.1.

  • To create a new major or minor branch use (replace v1.4.x with actual branch name): 
Code Block
languagebash
themeMidnight
titleGit branch creation
# checkout clean master branch
git clone https://github.com/apache/incubator-mxnet.git
cd incubator-mxnet
git branch v1.4.x
git push --set-upstream origin v1.4.x


  • The release candidate should contain all planned features, bug fixes, and code changes above. The release candidate commit should have passed the merge build, pip builds
  • Docs should also be manually built locally & checked for correctness. TODO: how to retrieve the commit hash.
  • Checkout the commit into a new branch "v#.#.x" and push it (It might be a good idea to do this in advance and freeze the branch)
  • Update NEWS and README. An example can be followed here: https://github.com/apache/incubator-mxnet/pull/6471. Note that this assumes the existence of the tag of next release, which is not created yet. Therefore, the update only happens on the release branch (not the master branch) so broken links are not visible to users.

Manual Docs Build

The build_all_versions.sh must also be updated with the correct tag

This script must be built locally to test docs (confirm procedure with Santhosh, Yao) and automate after 0.12.0.

Step 1.7. Test the release branch

...

Following jobs are run as a part of the nightly tests:

NightlyTests

NightlyTestsForBinaries

These jobs should already contain needed branches if they are cut. If not, check the jobs configuration. These are triggered by DailyTrigger. Either wait until the jobs have been triggered or trigger them manually.

Backward Compatibility checker (currently restricted).

Some jobs might take up to 4-5 hours, so plan accordingly.

In case of a failure of any job, check the console logs, the newly added PRs and try and resolve issues or contact community for a fix (on @dev list, for example). 

Jenkins CI Tests

Ensure that PR verification is enabled for the target release branch, for example v.1.3.x job

Ensure that nightly tests are enabled for the target release branch, for example v.1.3.x NightlyTestsForBinariesv1.3.x NightlyTests

Ensure that release branch is in the daily reports (contact Marco de Abreu )

RAT Check

make sure the RAT check passes (it's already part of the verification pipeline as a stage, no special actions needed)

Step 1.8. Tag release candidate (To be done by a committer)

Go to the GitHub repositories “releases” tab

Click “Draft a new release”

  • Provide the release tag in the form of “1.0.0.rc0” where 0 means it’s the first release candidate
  • Select the commit by clicking Target: branch > Recent commits > $commit_hash 
  • Copy and paste NEWS.md change into the description box from here
  • Select “This is a pre-release"

Step 1.9. This step has not been done in the past (question)

...

The src tar - apache-mxnet-src-#.#.#.rc0-incubating.tar.gz:

Clone the Repo and checkout the release branch 

Remove R-package until the licensing issue is resolved

Remove all the .git files

Remove all other files of the form .* (eg .travis, .DS_Store etc)

on MacOS use gnu-tar

Try to untar it in a different type of machine and try a classic build with `make`. Follow build instructions in the "Build from Source" section http://mxnet.incubator.apache.org/get_started/install.html

Code Block
languagebash
themeMidnight
titleCreating the src tar
linenumberstrue
git clone git@github.com:apache/incubator-mxnet.git apache-mxnet-src-0.12.0.rc0-incubating
git checkout v0.12.0
git submodule update --init --recursive
git checkout 0.12.0.rc0
rm -rf R-package/*
rm -rf R-package
rm -rf .DS_Store
rm -rf CODEOWNERS
find . -name ".git*" -print0 | xargs -0 rm -rf
brew install gnu-tar 
gtar -czvf apache-mxnet-src-0.12.0.rc0-incubating.tar.gz apache-mxnet-src-0.12.0.rc0-incubating

...

As per the Apache documentation, verify that the release candidate artifacts satisfy the following:

PGP signatures and SHA256 checksum verification

Code Block
languagebash
themeMidnight
svn co https://dist.apache.org/repos/dist/dev/incubator/mxnet && cd mxnet/#.#.#.rc0
brew install gpg coreutils
gpg --import ../KEYS

...

Step 1.12. Test the final release package (Committer or Contributor)

Download the package that was just uploaded to svn as above. 

Test that a build is successful - spinning up a deep learning Ubuntu AMI will give you an environment with all dependencies to build. For example:


Code Block
languagebash
themeMidnight
ci/build.py -p ubuntu_gpu

...

Step 1.13. Upload the release artifacts to Github release tag

Edit the release tag from Step 1.7, and upload the release artifacts from Step 1.9 (which were verified in 1.10-1.11).

After creating RC, give 2 days to build scala/ R packages if needed before starting vote on dev@.

Step 1.14. Language bindings

...

Step 3.1.1: Draft official release on GitHub

Go to the GitHub repo’s “releases” tab

Click “Draft a new release”

Provide the release tag in the form of “<major>.<minor>.<patch>”

Select the commit by clicking Target: master > the passing release candidate tag

Copy and paste NEWS change into the description box

Step 3.1.2: Rename and publish the source package

...

Step 3.1.4: Build, test, and publish the Docker images: (by Meghna)

This was done as follows in 0.11.0. In the future use the automated pipeline - 

Update the tag in the tool.sh file or select the correct tag in the Jenkins Job

Select the correct Repo name  = mxnet

Trigger the Job with IS_PUBLISH=false

Login to the slave and test each image for correct version by checking logs and/or Readme. 

Manually upload the 20 images that are built. 

Make sure to update the latest and previous docker images

Step 3.1.5: Verify the resources are uploaded

...

Now that a new version has been drafted on GitHub with source and pip packages published, we need to update the MXNet website.

Add the new installation version on the install page: 

Add news about the new release. See this example: docs/_static/mxnet-theme/index.html.

Add download links to the install archives. See this example: docs/install/download.md. Make sure to use the Apache distro servers. 

Update the pip package table: http://mxnet.incubator.apache.org/install/ubuntu_setup.html

Go to http://jenkins.mxnet-ci.amazon-ml.com/and find job “restricted-website-build”. Login if you're not already.

Click Configure and look for the section with the String Parameters.

Tags to build

  • If this is a minor release, the branch should already be in the first parameter "tags_to_build".
  • If this is a major release, you will need to add the branch to the list in the Default Value field. Note these are semicolon separated values.

Tag list

  • If this is a minor release, you will want to bump the version up for the branch you're updating. Example: change 1.2.0 to 1.2.1.
  • If this is a major release, you will need to add the new version number to the list. Example: add 1.3.0. Note these are semicolon separated values.

Click Save.

Click “Build with Parameters”, verify that the settings you just updated look correct, then click Build. 

The docs will be published to http://mxnet.incubator.apache.org/. There can be a delay due to some edge caching with Apache infra. The restricted-website-build job also runs four times a day.

Not Sure (question) Manually update the website’s download links:

TODO: link to Apache mirror source, pgp, sha 

3.3. Update NEWS.md & README.md on the MXNet master branch

...

3.7. Bump up the versions on the master  

Merge the PR you created for the release branch into the master

Add a new PR to update the version to one up/or next release for pip-releases

3.8. Clean up older releases on ASF mirrors  

Clean up older releases stored in https://dist.apache.org/repos/dist/release/incubator/mxnet/ and https://dist.apache.org/repos/dist/dev/incubator/mxnet/

Code Block
languagebash
themeMidnight
firstline1
titleDelete old releases from ASF mirrors (assuming new release version is 1.1.0 and the old one is 1.0.0
# checkout apache release repo
svn co https://dist.apache.org/repos/dist/release/incubator/mxnet apache-mxnet-release
cd apache-mxnet-release
# remove the previous release folder
svn rm 1.0.0
svn commit -m 'commit message'
# checkout apache dev repo
svn co https://dist.apache.org/repos/dist/dev/incubator/mxnet apache-mxnet-dev
cd apache-mxnet-dev
# remove the previous release candidate folders
svn rm 1.1.0.rc*
svn commit -m 'commit message'

...

Task #TaskOwnerRelative DateAbsolute Date
 PreReqs for Release Start:PM  
1Finalize the Release DatePMT-15 
2High Level List of Features and artifact specsPMT-15 
3Prepare Release Notes - draft 1PMT-15 
4Inform the communityReleaseLeadT-15 
5PR to update the Website into master/RBWebsiteLeadT-14 
6PR to update the Docs into master/RBDocsLeadT-14 
7PR to update the versions into master/RBReleaseLeadT-14 
8Validate License Headers and top Level LICENSE fileReleaseLeadT-14 
9Validate/update submodulesReleaseLeadT-14 
10Stabilize the code for CI to passReleaseLeadT-14 
11Merge all necessary PRs into masterCommunityT-13 
     
12Code Freeze and Release Start: Cut the Release BranchReleaseLeadT-10 
13Finalize the Release Notes based on PRs that got in - final draftPMT-10 
14PR to update the NEWS and README into the RBReleaseLeadT-9 
15Validate the docs build locally by checking release-specific docs like pip install, etc.DocsLeadT-9 
16Test part 1: Run the unit Tests on the Release BranchReleaseLeadT-8 
17Test part 2: Run the Nightly TestsReleaseLeadT-8 
18Create the Github Tag for the rc0ReleaseLeadT-7 
19Create the src tar and sign, Upload the src tarReleaseLeadT-7 
20Validate the signaturesRelease LeadT-7 
21Clone svn repo and do a manual testRelease LeadT-7 
     
 Begin Apache VotingReleaseLeadT-7 
22Start the vote on dev@ReleaseLead  
23Send out the results of the vote on dev@ReleaseLeadT-4 
24Revote if necessary (Steps 15-22)ReleaseLeadNot Accounted For 
25Start the vote on general@ReleaseLeadT-4 
26Send out the results of the vote on general@ReleaseLeadT-1 
 End Apache Voting   
     
27Create the final release tag on githubReleaseLeadT-1 
28Rename, resign and upload the src tar to final dirReleaseLeadT-1 
29Update the website using tagWebsite LeadT-1 
30Release the official pip packagepip LeadT-1 
31Release the official docker imagesDocker LeadT-1 
32After 24 hrs, validate the packages are uploadedRelease LeadT 
33Draft the offical announce email and reviewRelease LeadT-1 
34Send out the email on announce@Release LeadT 
35Update the apache blogRelease LeadT 
36update the aws blogPMT 
37send internal announcementPMT 
38Update the version on masterRelease LeadT 

Notes for reference

*NOTES FROM DOCS FOR REFERENCE:*
http://incubator.apache.org/guides/releasemanagement.html

   - 3 +1 votes from IPMC members (these are the votes that count but we
   should open up to the whole podling community)
   - For podlings, 2 additional constraints:
   - Release artifacts must include “incubating” in final file name (ex:
      apache-mxnet-src-0.10.1-incubating.tar.gz)
   - Release artifacts must include disclaimer in the release artifacts


   - The Incubator PMC expects the source releases to be staged on
   https://dist.apache.org/repos/dist/dev/incubator/podlingName so that
   they can easily be moved to the release location via svn mv   (
   http://www.apache.org/dist/incubator/)
   - After graduating, RC’s go into https://dist.apache.org/repos/dist/dev/
   and official releases go into https://dist.apache.org/repos/dist/release/


http://incubator.apache.org/guides/branding.html#disclaimers

   - Apache Press Team [http://www.apache.org/press/index.html#whoweare]
   must review and coordinate releases for branding
   - On website and in release DISCLAIMER file:
   - Apache Podling-Name is an effort undergoing incubation at The Apache
      Software Foundation (ASF), sponsored by the name of Apache TLP sponsor.
      Incubation is required of all newly accepted projects until a further
      review indicates that the infrastructure, communications, and decision
      making process have stabilized in a manner consistent with other
successful
      ASF projects. While incubation status is not necessarily a reflection of
      the completeness or stability of the code, it does indicate that the
      project has yet to be fully endorsed by the ASF.
      - Website should include Apache Incubator logo:
      http://incubator.apache.org/guides/press-kit.html


   - Release should include:
      - DISCLAIMER
      - LICENSE
      - NOTICE - attribution notices


http://www.apache.org/legal/release-policy.html

   - A release must contain source package which is cryptographically
   signed by Release lead with detached signature. It must be tested prior
   to voting for release.
   - Release must only contain appropriately licensed code
   - Please ensure you wait >=24 hours after uploading a release before
   making announcements so mirrors catch up
   - Releases of more than 1GB of artifacts require a heads-up to
   Infrastructure in advance.
   - Which directory for what build?
   http://www.apache.org/legal/release-policy.html#build-directories


http://www.apache.org/dev/release-distribution.html

   - Artifacts MUST be accompanied by:
      - apache-mxnet-src-0.10.1-incubating.asc - contains OpenPGP
      compatible ASCII armored detached signature
      - apache-mxnet-src-0.10.1-incubating.sha - SHA checksum (SHOULD)
   - Publish KEYS file in distribution directory root
      - Signing keys MUST be published in KEYS file, SHOULD be available in
      global public keyserver
      http://www.apache.org/dev/release-signing#keyserver, SHOULD be linked
      into web of trust
      - Keys MUST be RSA & 4096 bits


http://www.apache.org/dev/release-publishing.html

   - Apache RAT can assist in checking license compliance
   http://creadur.apache.org/rat/
   - Eventually we should set up a build system to sign our releases with
   cryptographic signatures. For now we’ll do it manually.


http://www.apache.org/dev/release-signing.html

   - Create a signature and sign releases as mentioned above