Page History

...

1. Go to https://jenkins.mxnet-ci.amazon-ml.com/job/restricted-mxnet-cd/ and make sure you have write access to the Jenkins system. You can contact Joe (joseph.evans@gmail.com) to be added.
2. We would like to trigger a mxnet-cd-pipeline job which in turn triggers a down-stream mxnet-cd-release job.
   1. The existing mxnet-cd-pipeline and mxnet-cd-pipeline-v1.x pipelines point to the master and v1.x branch of https://github.com/apache/incubator-mxnet respectively. So, what what we would want to do is duplicate one of them, depending on whether it's a 2.x or 1.x release, and point the pipeline to the correct release tag.
   2. Click "New Item" on the sidebar.
   3. Give the new pipeline a meaningful name, scroll down to the bottom of the page, and paste in "mxnet-cd-pipeline" or "mxnet-cd-pipeline-v1.x" to field "Copy from".
   4. Click "OK".
   5. Go into the new pipeline and click "Configure" on the sidebar.
   6. Under "Pipeline" find "Branches to Build." Use refs/tags/<tagName> as the branch specifier.
   7. Click "Save"
   8. Now also replicate the mxnet-cd-release-job or mxnet-cd-release-job-1.x pipeline and give it a meaningful name. Here we do not need to mess with any settings.
   9. Go back to configure the duplicated cd pipeline created in step b~g. Find the parameter "CD_RELEASE_JOB_NAME", change the default value to the release job pipeline created in step h.
   10. Click "Save"
   11. Now, click "Build with Parameters" on the side-bar. Choose the variants of MXNet to build and select the "RELEASE_BUILD" box and hit the "Build" Button.
3. Now the cd pipeline will be triggered and in turn it will spawn several jobs in the release job pipeline. Specifically, "mxnet_lib/static" will build the MXNet lib statically, and "python/pypi" will publish the pip wheels to https://dist.mxnet.io/python/.
4. If any variant of MXNet failed in either of the two jobs, the pip wheel of that variant will not be published. In this case we can try to trigger the cd pipeline again for that variant or root cause and fix the issue based on the pipeline log.
5. Once all variants succeed to publish, we can go to https://dist.mxnet.io/python/ and identify the wheels we just created by the version number, date created, and variant string. We can simply download those wheels, give them another test if needed, and upload them to PyPI.
   
6. Now we can remove the two duplicated pipelines from https://jenkins.mxnet-ci.amazon-ml.com/job/restricted-mxnet-cd/.

For mac, the following automated script can be used to build the wheel.(Publishing steps not yet included)

#!/bin/bash

#Install brew
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

#install developement tools
brew install nasm
brew install automake
brew install libtool
brew install ninja
brew install pkg-config
python -m pip install contextvars
python -m pip install numpy
python -m pip install requests

#clone MXNet
cd $HOME
git clone -b v1.x --single-branch https://github.com/apache/incubator-mxnet.git
cd $HOME/incubator-mxnet/
git submodule update --init --recursive

# set ENV variables and build statically
export IS_RELEASE=True
export mxnet_variant=cpu
echo $(git rev-parse HEAD) >> python/mxnet/COMMIT_HASH
mkdir -p $HOME/pip_build/mxnet-build
cp -r * $HOME/pip_build/mxnet-build
export CODE_DIR=$(pwd)
cd $HOME/pip_build/mxnet-build
CMAKE_STATICBUILD=1 ./tools/staticbuild/build.sh $mxnet_variant pip

# Package wheel
cd $HOME/pip_build
cp -r mxnet-build/tools/pip/* .
python setup.py bdist_wheel

# copy wheel to $HOME/pip_build
cp $HOME/pip_build/dist/mxnet* $HOME/pip_build

Please note that the release tag is usually created on a branch that's forked from v1.x and master. In that case it may lack some of the latest ci/cd fixes on the v1.x and master branch. We might need to manually port back those fixes for the cd pipeline to work.

...