...
At least 3 days prior to cutting the first release candidate, bump up check the version number and address other changes that the release depends on.
Bump up Update the version number by following if it's not reflecting the new release version. Here is a previous version update PR (https://github.com/apache/incubator-mxnet/pull/1347819196). Please update this wiki with latest version update PR to stay current.
make sure no broken/non-existing links are added to the master/v1.x branch.
Check with the website lead and docs lead that all necessary updates have made it into the release branch
...
Create a RSA GPG key of length 4096, upload it to the public server, and add it to the KEYS file (do this process once for each release lead):
...
...
See more detailed instructions on creating the key here: https://www.apache.org/dev/openpgp.html#generate-key. Instructions for updating the KEYS file can be found here: https://github.com/apache/incubator-mxnet/blob/master/KEYS#L9-L13at the beginning of the KEYS file. Note that you must use apache email instead of personal email for the key.
The following instructions shows how to update the key using svn.
Code Blockcode | ||||||
---|---|---|---|---|---|---|
| ||||||
# checkout the apache mxnet repo svn co https://dist.apache.org/repos/dist/devrelease/incubator/mxnet apache-mxnet cd apache-mxnet # update the KEYS file ... # commit the update svn commit -m “update keys file for xxx ” --username your_username --password your_passwd |
...
- The release candidate should contain all planned features, bug fixes, and code changes above. The release candidate commit should have passed the merge build, pip buildsDocs should also be manually built locally & checked for correctness. TODO: how to retrieve the commit hash.
- Checkout the commit into a new branch "v#.#.x" and push it (It might be a good idea to do this in advance and freeze the branch)
- Update NEWS and README. Examples can be followed here: NEWS.md and README.md. Note that this assumes the existence of the tag of next release, which is not created yet. Therefore, the update only happens on the release branch (not the master branch) so broken links are not visible to users.
Manual Docs Build
The build_all_versions.sh must also be updated with the correct tag
This script must be built locally to test docs (confirm procedure with Santhosh, Yao) and automate after 0.12.0.
Step 1.7. Test the release branch
- Docs should also be manually built locally & checked for correctness.
- build and host the website locally to check the correctness.
- More reference here: Host a MXNet Website Preview. Contact MXNet engineering team for access if needed
Step 1.7. Test the release branch
Once all the Once all the PRs have been merged the final commit hash on the release branch must be tested. (You should begin testing as soon as the release branch is created, but make sure it is run at least once on the final commit)
...
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
git clone git@github.com:apache/incubator-mxnet.git apache-mxnet-src-0.12.0.rc0-incubating
git checkout v0.12.0
git submodule update --init --recursive
git checkout 0.12.0.rc0
rm -rf R-package/*
rm -rf R-package
rm -rf .DS_Store
rm -rf CODEOWNERS
find . -name ".git*" -print0 | xargs -0 rm -rf
brew install gnu-tar
gtar -czvf apache-mxnet-src-0.12.0.rc0-incubating.tar.gz apache-mxnet-src-0.12.0.rc0-incubating |
...
Code Block | ||||
---|---|---|---|---|
| ||||
svn co https://dist.apache.org/repos/dist/dev/incubator/mxnet && cd mxnet/#.#.#.rc0 brew install gpg coreutils wget https://downloads.apache.org/incubator/mxnet/KEYS gpg --import ../KEYS |
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
gpg --verify ./apache-mxnet-src-0.11.0.rc3-incubating.tar.gz.asc gsha512sum --check ./apache-mxnet-src-0.11.0.rc1-incubating.tar.gz.sha512 |
...
Test that a build is successful - spinning up a deep learning Ubuntu AMI will give you an environment with all dependencies to build. You can build it MXNet from source or from CI dockers. For example:
...
In ASF, votes are open "at least" 72hrs (3 days). If you don't get enough number of binding votes within that time, you cannot close the voting deadline. You need to extend it.
Make sure the dev community is aware of any ASF votes and results so they can be active in the ongoing voting process.
Send out a notice to vote with the template below.
To get the fingerprint of your keys, run `gpg2 --list-keys --fingerprint`
Info | ||||
---|---|---|---|---|
| ||||
To: general@incubator.apache.org Dear community, This is a call for a releasing Apache MXNet (incubating) #.#.#, release Apache MXNet (incubating) community has voted and approved the release. Vote thread: Result thread: The source tarball, including signatures, digests, etc. can be found at: The tag to be voted upon is #########: The release hash is #########: Release artifacts are signed with the following key: KEYS file available: For information about the contents of this release, see: The vote will be open for 72 hours. [ ] +1 release this package as ######### Best regards, |
...
Send out an email on general@ in the same voting thread which you sent in previous step to summarize the result.
3. Finalizing and posting a release
3.1. Marking & Publishing the Official release
Once the release candidate passes the vote, we draft the official release on GitHub and rename the release candidate as the official release.
Step 3.1.1: Draft official release on GitHub
Go to the GitHub repo’s “releases” tab
Click “Draft a new release”
Provide the release tag in the form of “<major>.<minor>.<patch>”
Select the commit by clicking Target: branch > the passing release candidate tag
Copy and paste NEWS change into the description box
Step 3.1.2: Rename and publish the source package
First upload to dev and then move to releases as follows:
#create a directory under releases - https://httpd.apache.org/dev/release.html
This process needs to change -> http://www.apache.org/dev/release-download-pages.html
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
# Step 1: Download and Untar
tar -zxvf apache-mxnet-src-0.11.0.rc3-incubating.tar.gz
# Step 2: rename (remove the rc) and tar again
mv apache-mxnet-src-0.11.0.rc3-incubating apache-mxnet-src-0.11.0-incubating
gtar -czvf apache-mxnet-src-0.11.0-incubating.tar.gz apache-mxnet-src-0.11.0-incubating
# Step 3: Resign now
# Step 4: First upload here -
svn mv https://dist.apache.org/repos/dist/dev/incubator/mxnet/0.11.0/
# Step 5: Move from dev to release
svn mv https://dist.apache.org/repos/dist/dev/incubator/mxnet/0.11.0/ https://dist.apache.org/repos/dist/release/incubator/mxnet/ --message "0.11.0 release moving from dev to release repo"
# Wait 24hours for the src mirroring. The final download page for the announce email is somewhere here (example 0.11.0) - https://www.apache.org/dyn/closer.cgi/incubator/mxnet/0.11.0-incubating/ |
Step 3.1.3: Download the nightly pip build, rename, and publish:(by Sheng)
Step 3.1.4: Build, test, and publish the Docker images: (by Meghna)
This was done as follows in 0.11.0. In the future use the automated pipeline -
Update the tag in the tool.sh file or select the correct tag in the Jenkins Job
Select the correct Repo name = mxnet
Trigger the Job with IS_PUBLISH=false
Login to the slave and test each image for correct version by checking logs and/or Readme.
Manually upload the 20 images that are built.
Make sure to update the latest and previous docker images
...
Send out a notice with the vote results with the template below.
Info |
---|
To: general@incubator.apache.org |
3. Finalizing and posting a release
3.1. Marking & Publishing the Official release
Once the release candidate passes the vote, we draft the official release on GitHub and rename the release candidate as the official release.
Step 3.1.1: Draft official release on GitHub
Go to the GitHub repo’s “releases” tab
Click “Draft a new release”
Provide the release tag in the form of “<major>.<minor>.<patch>”
Select the commit by clicking Target: branch > the passing release candidate tag
Copy and paste NEWS change into the description box
Step 3.1.2: Rename and publish the source package
First upload to dev and then move to releases as follows:
#create a directory under releases - https://httpd.apache.org/dev/release.html
This process needs to change -> http://www.apache.org/dev/release-download-pages.html
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
# Step 1: Download and Untar
tar -zxvf apache-mxnet-src-0.11.0.rc3-incubating.tar.gz
# Step 2: rename (remove the rc) and tar again
mv apache-mxnet-src-0.11.0.rc3-incubating apache-mxnet-src-0.11.0-incubating
gtar -czvf apache-mxnet-src-0.11.0-incubating.tar.gz apache-mxnet-src-0.11.0-incubating
# Step 3: Resign now
# Step 4: First upload here -
svn mv https://dist.apache.org/repos/dist/dev/incubator/mxnet/0.11.0/
# Step 5: Move from dev to release
svn mv https://dist.apache.org/repos/dist/dev/incubator/mxnet/0.11.0/ https://dist.apache.org/repos/dist/release/incubator/mxnet/ --message "0.11.0 release moving from dev to release repo"
# Wait 24hours for the src mirroring. The final download page for the announce email is somewhere here (example 0.11.0) - https://www.apache.org/dyn/closer.cgi/incubator/mxnet/0.11.0-incubating/ |
Step 3.1.3: Build, test, and publish pip wheels and python docker images
For linux, we can trigger a release pipeline that builds, tests and publishes the pip wheels and python docker images for us with just a few steps:
- Go to https://jenkins.mxnet-ci.com/job/restricted-mxnet-cd/job/mxnet-stable-release-pipeline/ and make sure you have write access to the Jenkins system.
- Click "Configure" and update the "Branch specifier" to point to the GitHub branch/tag and click "Save" (The GitHub repository URL is set to apache/incubator-mxnet, if you are releasing using a fork, update this URL too)
- For branch, use
<branchName>
- For tag, use
refs/tags/<tagName>
- For branch, use
- Click "Build with Parameters" from the left pane, modify the following fields as needed and click on "Build"
- MXNET_VARIANTS : Make sure all supported cu* flavors are present
- RELEASE_BUILD" : Keep the box Unchecked
- CD_RELEASE_JOB_NAME : Do not modify this field
- VERSION : Mention the MXNet release tag here e.g. 1.8.0, 2.0.0 (This tag is only used in naming python docker images.)
- This triggers the release pipelines for different cpu and cu* flavors which can be found here https://jenkins.mxnet-ci.com/job/restricted-mxnet-cd/job/mxnet-stable-release-job/ . You can click on "Open Blue Ocean" from the left pane to check pipeline status. There are 3 pipelines that get trigerred:
- First pipeline builds the libmxnet binaries for different flavors. Only after this pipeline completes (takes around 2.5 hours), the other two pipelines are triggered, that run in parallel.
- Second pipeline "python/pypi" publishes the pip wheels to https://dist.mxnet.io/python/.
- Third pipeline "python/docker" publishes the python docker images to https://gallery.ecr.aws/w6z5f7h2/mxnet/python.
- If any job in any of the three pipelines fails, go through the corresponding logs and either
- Re-trigger the pipeline in case it's a flaky failure or
- Root cause and fix the issue and then re-trigger the pipeline.
- Re-trigger the pipeline in case it's a flaky failure or
- Once all pipelines are green, Python wheels and Python docker images for all variants are published to their respective targets. Now we need to move these released artifacts to their final locations so that they are available to users:
- Python wheels (for this release) from https://dist.mxnet.io/python/ go to PyPI.
- Identify the wheels by the version number, date created, and variant string
- Download the above wheels, give them another test if needed, and upload them to PyPI
- For more info regarding publishing pip wheels to Pypi, please contact zhasheng@apache.org
- Python docker images (for this release) from https://gallery.ecr.aws/w6z5f7h2/mxnet/python go to dockerhub.
- Identify the docker images by the version number, date created, and variant string
- Pull each docker image to local (docker pull public.ecr.aws/w6z5f7h2/mxnet/python:<tag>)
- Tag downloaded images (docker tag …)
- Login to dockerhub (docker login ...)
- Push images (docker push …)
- Python wheels (for this release) from https://dist.mxnet.io/python/ go to PyPI.
For mac, the following automated script can be used to build the wheel.(MacOS machine is required to generate wheels for mac)
#!/bin/bash
# Install brew
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
# Install developement tools
brew install nasm
brew install automake
brew install libtool
brew install ninja
brew install pkg-config
python -m pip install contextvars
python -m pip install numpy
python -m pip install requests
# Clone MXNet
cd $HOME
git clone -b v1.x --single-branch https://github.com/apache/incubator-mxnet.git # Use release tag instead
cd $HOME/incubator-mxnet/
git submodule update --init --recursive
# Set ENV variables and build statically
export IS_RELEASE=True
export mxnet_variant=cpu
echo $(git rev-parse HEAD) >> python/mxnet/COMMIT_HASH
mkdir -p $HOME/pip_build/mxnet-build
cp -r * $HOME/pip_build/mxnet-build
export CODE_DIR=$(pwd)
cd $HOME/pip_build/mxnet-build
CMAKE_STATICBUILD=1 ./tools/staticbuild/build.sh $mxnet_variant pip
# Package wheel
cd $HOME/pip_build
cp -r mxnet-build/tools/pip/* .
python setup.py bdist_wheel
# Copy wheel to $HOME/pip_build
cp $HOME/pip_build/dist/mxnet* $HOME/pip_build
Wheels are publish manually to pypi.
Please note that the release tag is usually created on a branch that's forked from v1.x and master. In that case it may lack some of the latest ci/cd fixes on the v1.x and master branch. We might need to manually port back those fixes for the cd pipeline to work.
Step 3.1.4: Verify the resources are uploaded
Check that the resources are present in http://www.apache.org/dist/incubator/mxnet. It may take a while for them to be visible. This will be mirrored throughout the Apache network. There are a few remaining steps.
Step 3.1.6: Test
Test multinode and multi gpu (kvstore=1)
3.2. Update the MXNet website
Anchor | ||||
---|---|---|---|---|
|
...
Follow these steps to update website in release branch and create website artifacts.
- Add the new installation version on the get_start page: Set the new version to be default, and add the dropdown option. See this example
- Add download links to the install archives. See this example. Make sure to use archive.apache.org for all except the current one.
- Current: http://www.apache.org/dyn/closer.lua?filename=incubator/mxnet/#.#.#/apache-mxnet-src-#.#.#-incubating.tar.gz&action=download
- Legacy: http://archive.apache.org/dist/incubator/mxnet/
- Update the pip package table in /python/cpu/pip.md and /python/gpu/pip.md. See this exampleThis has a sketch file that can be updated to generate a new png image. Because this file is live and pulled from the web-data repo, you need to submit a PR to this repo with the new image. It is recommended that you add a new one with the version name. Then in your PR for the html updates, you refer to this specific new image, rather than updating the one that is used in production.
- Update and generate the image file for pip install table.
- Download pip-packages.sketch from dmlc/web-data repo
- Use Sketch or other diagram drawing tools to update the sketch file for the new version table (add rows/columns and checkmarks)
- Save the new pip table as a png file with name pip-packages-{NEW_RELEASE_VERSION}.png
- Upload the .sketch file and .png file to dmlc/web-data repo. See this example
- Update pip install table link in pip_snippet.md. See this example
- Add menu options for general version dropdown. Add the new release version number in _config.ym, _config_beta.tml and _config_prod.yml under '- master'._prod.yml under '- master'.
- Update the dropdown versions for python doc. See this example
- Update the dropdown versions in header.html. See this example
- Prepare and deploy the release version website artifacts, at the time this step requires certain access permissions to dev Jenkins server, MXNet S3 and codeCommit. Please reach out to MXNet Engineering team for more information. CI Jenkins server
- Log in to Jenkins pipeline restricted-website-update-versions
- Goto Configure to make sure the Git Repository URL points to https://github.com/apache/incubator-mxnet.git
- Click Build with Parameters on the left panel
- Specify COMMIT_ID to the release branch
- Specify FOLDER_NAME to the release version, e.g. 1.8.0 or 2.0.0
- Build and make sure the pipeline finish successfully
Step 3.2.2 Update website in master version
Ensure all the above processes in step 3.2.1 are all completed. Next task is to update master version website.
- Cherry-pick above changes (step 1-5 6 in 3.2.1) from release branch to master branch and v1.x branch. See this example
- Set default website version to new release version by updating the
RewriteRule
in /docs/static_site/src/.htaccess under comment "# Set default website version to current stable". For example: update the version 1.x to the new release version:- RewriteRule ^(.*)$ /versions/1.x/$1 [r=307,L] => RewriteRule ^(.*)$ /versions/{NEW_RELEASE_VERSION}/$1 [r=307,L]
- Publish website artifact to the main website
- Manually trigger pipeline restricted-website-build-master to publish the new website artifact. Make sure that the pipeline finishes successfully.
- The new website
- The docs will be published to http://mxnet.incubator.apache.org/. There can be a delay due to some edge caching with Apache infra. The restricted-website-build job also runs four times a day.
- After publishing, check the correctness of the main website
3.3. Update NEWS.md & README.md on the MXNet master branch
...
Info |
---|
There must be a link in the announcement to the downloads page, not to the dyn/closer page. Downloads must be mirrored from the official Apache distribution site, not from github or other site. e.g. "A Link to the httphttps://mxnet.incubator.apache.org/install/versions/1.8.0/get_started/download.html" |
Send an email to announce@apache.org, general@incubator.apache.org and dev@mxnet.incubator.apache.org with the subject: [ANNOUNCE] Release Apache MXNet (incubating) version $release and a body along the lines of:
Info | ||
---|---|---|
| ||
The Apache MXNet (incubating) community is happy to announce Apache MXNet (incubating) version ##.#.#! ( ## if applicable ## ) Release blog post: ( ## -- if applicable -- ## ) Apache MXNet (incubating) is a deep learning framework designed for both efficiency and flexibility. It allows you to mix symbolic and imperative programming to maximize efficiency and productivity. ( ## for patch releases ## ) 1.3.1 is a maintenance release incorporating important bug fixes and important performance improvements. ( ## -- for patch releases -- ## ) A full list of the changes in this release can be found in the release notes: A link to the download page can be found here: If you prefer to build from source and experiment with various compile-time configuration options, use this link to get the instructions: Or you can download and play with MXNet easily using one of the options below: 1. The Pip packages can be found here: 2. The Docker Images can be found here: Links in Maven to the published Scala packages: : https://repository.apache.org/content/repositories/releases/org/apache/mxnet/ 2. The Docker Images can be found here:
The Pip package: The release tag: MXNet Resources - Our discussion forum (https://github.com/apache/incubator-mxnet/discussions) Social Media - Apache MXNet on Twitter (https://twitter.com/apachemxnet) For more information on Apache MXNet (incubating), please see:
_______ DISCLAIMER: Apache MXNet (incubating) is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the name of Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. https://cwiki.apache.org/confluence/x/BINjB |
Once the release is complete, make sure that all PRs that were created during the release process only for the release branch (version updates, README NEWS etc) are now merged into the master branch too.
3.7. Bump up the versions on the master
Merge the PR you created for the release branch into the master
...
branch too.
3.7. Bump up the versions on the master
Merge the PR you created for the release branch into the master
Add a new PR to bump up the version for next release
- Bump up the version number into v1.x branch if the new version is branched from v1.x. Bump up the version number into master branch if the new version is branched from master. Here is a previous version update PR (https://github.com/apache/incubator-mxnet/pull/19196). Please update this wiki with latest version update PR to stay current.
- make sure no broken/non-existing links are added to the master/v1.x branch.
3.8. Clean up older releases on ASF mirrors
...