...

Update CHANGES.txt in master to replace Release X.Y.0 (unreleased) with Release X.Y.0 - (YYYY-MM-DD). Commit:
Code Block
language bash
git add CHANGES.txt git commit -m "Preparing for release X.Y.0"
Create a branch for the release series (ensure origin/master is up-to-date with the ASF repo before running it):
Code Block
language bash
git checkout -b branch-X.Y origin/master
Switch back to master and add A.B.C-SNAPSHOT (unreleased changes) to CHANGES.txt.
Code Block
language bash
git checkout - (edit CHANGES.txt)
Bump the version number in the master branch:
Code Block
language bash
./gradlew setversion -Pnextversion="A.B.C-SNAPSHOT"

Commit these changes to the master and push.

Code Block

language	bash

git add CHANGES.txt (and other updated files)
git commit -m "Bumping up the version to A.B.C-SNAPSHOT"
git push upstream master (assuming that ASF repo is referred to as 'upstream')

Checkout the release branch
Code Block
language bash
git checkout branch-X.Y
Update the release branch's version information: the version number in the release branch ends in -SNAPSHOT, but we need to remove this for the release. For example, 0.8.0-SNAPSHOT needs to be changed to 0.8.0.
Code Block
language bash
./gradlew setversion -Pnextversion="X.Y.Z"

Rename docker images and repo URLs:

Code Block

language	bash

grep -R --color :trunk- * | grep yaml
grep -R --color "http://repos.bigtop.apache.org/releases/X.Y.Z" *
grep -R --color "http://repos.bigtop.apache.org/releases/x.y.z" * (where "x.y.z" is the previous version of "X.Y.Z". Just in case)
(replace the occurrences found by the above commands)

Commit these changes to the release branch and push

Code Block

language	bash

git add (updated files)
git commit -m "Changing version to X.Y.Z"
git push upstream branch-X.Y (assuming that ASF repo is referred to as 'upstream')

Also, update the default version of Bigtop (and your code signing key ID, if needed) defined in hieradata on master and the release branch. For example:

Code Block

language	bash

$ git checkout master

(edit bigtop-deploy/puppet/hieradata/bigtop/repo.yaml)

$ git diff
diff --git a/bigtop-deploy/puppet/hieradata/bigtop/repo.yaml b/bigtop-deploy/puppet/hieradata/bigtop/repo.yaml
index 48c8a4ae..0d6a9126 100644
--- a/bigtop-deploy/puppet/hieradata/bigtop/repo.yaml
+++ b/bigtop-deploy/puppet/hieradata/bigtop/repo.yaml
@@ -1,5 +1,5 @@
 bigtop::bigtop_repo_gpg_check: true
-bigtop::bigtop_repo_apt_key: "BB95B97B18226C73CB2838D1DBBF9D42B7B4BD70"
+bigtop::bigtop_repo_apt_key: "01621A73025BDCA30F4159C62922A48261524827"
 bigtop::bigtop_repo_yum_key_url: "https://downloads.apache.org/bigtop/KEYS"
-bigtop::bigtop_repo_default_version: "1.4.0"
+bigtop::bigtop_repo_default_version: "1.5.0"

$ git add bigtop-deploy/puppet/hieradata/bigtop/repo.yaml 
$ git commit -m "Update default values in hieradata for release 1.5.0"
[master 9715ca7a] Update default values in hieradata for release 1.5.0
 1 file changed, 2 insertions(+), 2 deletions(-)
$ git push upstream master

$ git checkout branch-1.5 
$ git cherry-pick -x 9715ca7a
$ git push upstream branch-1.5

Shall you need to commit additional fixes into ongoing release, the commits should go to the release branch and only then be merged into master. Doing this other way around forces git cherry-pick which leads to discrepancies in the commit hash-codes and makes the branch look untidy and hard to follow.

For a patch release, just do the following four items on the branch-X.Y, instead of doing all of the items above:

Update the release branch's version information
Rename docker images and repo URLs
Update the default version of Bigtop
Commit these changes to the release branch and push

3. Generate the Release Notes

...

Manually check this list for accuracy! I've repeatedly seen closed , since JIRA can be updated after step 1, even if it's unintentional. Also, I've repeatedly seen closed bugs that were not fixed (i.e., duplicate) marked with a fix version, so that they incorrectly show up in this list. Find those, edit them to remove the fix release (only actually fixed bugs should have a fix release) and re-run the report. A better way to deal with it is to run

...

Code Block

language	bash

git tag release-x.y.z-RC0 -m "Bigtop X.Y.Z release."
git push upstream release--tags x.y.z-RC0

5. Build and run Package and Smoke Tests

...

Create a release specific job to build images such as Docker-Puppet-x.y.z, Docker-Toolchain-x.y.z and Docker-Toolchain-x.y.z-pull by copying existing jobs:

https://ci.bigtop.apache.org/view/Docker/job

...

Make sure there is enough free space on the disk of Jenkins worker nodes. Remove workspaces and Docker images for old releases if you need to reclaim the free space.

Make sure all the built images are uploaded to Dockerhub after running the job.

...

Make sure all the built images are uploaded to Dockerhub

https://hub.docker.com/r/bigtop/puppet/tags/

https://hub.docker.com/r/bigtop/slaves/tags/

Make sure all the built images are downloaded on dockeron docker-slave-06 and docker-slave-07 https://ci.bigtop.apache.org/view/Docker/job/(by Docker-Toolchain-1x.4y.0z-pull/).

5.2. Build RPM/DEB packages

...

Code Block

language	bash

cd ~/
# Change the image for AARCH64 or PPC64LE
docker run -ti --rm -v $PWD:/tmpwork bigtop/puppet:1.4.0-centos-7 bash

...

Code Block

language	bash

cd /tmpwork
OS=centos-7
 
# Sign all RPM packages (This step required to input passphrase, so don't copy and paste the entire script here)
rpm --addsign `find ${OS} -name \*rpm`
 
# Recreate the metadata for repository
createrepo ${OS}
 
# Armor the metadata
gpg --detach-sign --armor ${OS}/repodata/repomd.xml

...

Code Block

language	bash

cd ~/
# Change the image for AARCH64 or PPC64LE
docker run -ti --rm -v $PWD:/tmpwork bigtop/puppet:1.4.0-debian-9 bash

...

Code Block

language	bash

cd /tmpwork
VERSION=1.4.0
OS=debian-9
ARCH=amd64
#ARCH=arm64
#ARCH=ppc64el
SIGN_KEY=B7B4BD70
export GPG_TTY=$(tty)
 
# Sign DEB packages (This step required to input passphrase, so don't copy and paste the entire script here)
dpkg-sig --cache-passphrase --sign builder --sign-changes force_full `find ${OS}/ -name \*deb`
 
# Build signed apt repository
mkdir -p conf
cat > conf/distributions <<__EOT__
Origin: Bigtop
Label: Bigtop
Suite: stable
Codename: bigtop
Version: ${VERSION}
Architectures: ${ARCH} source
Components: contrib
Description: Apache Bigtop
SignWith: ${SIGN_KEY}
__EOT__
 
cat > conf/options <<__EOT__
verbose
ask-passphrase
__EOT__

rm -rf ${OS}/apt
# Note that this command creates the apt repository (db, dists, pool)
# in the current directory, so you can't ran it for multiple distros in parallel.
reprepro --ask-passphrase -Vb . includedeb bigtop `find ${OS}/ -name \*deb`
rm -rf ./${OS}/*
mv conf db dists pool ${OS}

...

For YUM repos, upload files /tmpwork/centos-7/* into into repos.bigtop.apache.org/releases/1.4.0/centos/7/x86_64/. For example:

...

For APT repos, upload files /tmpwork/debian-9/* into into repos.bigtop.apache.org/releases/1.4.0/debian/9/amd64/. For example:

...

Code Block

language	bash

cd ~/
docker run -ti -u jenkins -v $PWD:/tmpwork bigtop/slaves:trunk-ubuntu-18.04 bash -l
# The subsequence commands should be executed inside the docker container
cp -r /tmpwork/.gnupg ~/

Create a maven settings file ~/.m2/settings.xml with the following content:

...

Code Block

language	bash

mvn deploy -Prelease -f pom.xml
mvn deploy -Prelease -f bigtop-test-framework/pom.xml
mvn deploy -Prelease -f bigtop-tests/test-artifacts/pom.xml
.xml
HADOOP_CONF_DIR=/etc/hadoop/conf HADOOP_HOME=/usr/lib/hadoop mvn deploy -Prelease -f bigtop-tests/test-execution/pom.xml -DskipITs

...

Code Block

language	bash

VERSION=X.Y.Z
RC_NUMBER=n

cd /tmpwork
git clone https://gitbox.apache.org/repos/asf/bigtop.git -b release-${VERSION}-RC${RC_NUMBER} --depth 1
wget --no-parent --recursive --reject='index.html*' https://dist.apache.org/repos/dist/dev/bigtop/bigtop-${VERSION}-RC${RC_NUMBER}/

# make sure that the source code tree and the release tarball are identical except for .git/CI/site releated files
tar xf dist.apache.org/repos/dist/dev/bigtop/bigtop-${VERSION}-RC${RC_NUMBER}/bigtop-${VERSION}-project.tar.gz
diff -r bigtop bigtop-${VERSION}

# make sure that the signature and checksums are correct
cd dist.apache.org/repos/dist/dev/bigtop/bigtop-${VERSION}-RC${RC_NUMBER}
gpg --verify bigtop-${VERSION}-project.tar.gz.asc bigtop-${VERSION}-project.tar.gz
diff <(sha256sum bigtop-${VERSION}-project.tar.gz) bigtop-${VERSION}-project.tar.gz.sha256 
diff <(sha512sum bigtop-${VERSION}-project.tar.gz) bigtop-${VERSION}-project.tar.gz.sha512 

# make sure that all files exist and their signatures are correct
for i in $(find repos -not -name '*.asc' -type f); do echo ${i}; gpg --verify ${i}.asc ${i}; done

...

Code Block

language	bash

git checkout release-x.y.z
git tag rel/x.y.z -u <signing key ID>
git push --tags

Wait 24 Hours

~~It takes up to 24 hours for all the mirrors to sync, so don't announce the new release just yet.~~

We had to wait 24 hours before, but it's not true now since ASF adopted CDN instead of mirror sites. According to the FAQs of the ASF infra:

Apache uses a global content distribution network (CDN) which collects new releases almost as soon as you post them.
The files therefore become available for download almost immediately. You probably don't need to wait more than fifteen minutes before announcing a release.

Build and Deploy Site

Update files under src/site by referring to past commits such as https://github.com/apache/bigtop/commit/678375eb75f7367f9114a3ad8087ba40f21444b6 or https://github.com/apache/bigtop/commit/d5c45a2879f8e3a8d29a36386ad8904a4e71a274. Commit into master and push them.

...

Page tree

Versions Compared

Old Version 143

New Version Current

Key

3. Generate the Release Notes

5. Build and run Package and Smoke Tests

5.2. Build RPM/DEB packages

Wait 24 Hours

Build and Deploy Site

Page tree

Page History

Versions Compared

Old Version 143

New Version Current

Key

3. Generate the Release Notes

5. Build and run Package and Smoke Tests

5.2. Build RPM/DEB packages

Wait 24 Hours

Build and Deploy Site