...
Update CHANGES.txt in master to replace
Release X.Y.0
(unreleased)Release X.Y.0 - (YYYY-MM-DD
). Commit:Code Block language bash git add CHANGES.txt git commit -m "Preparing for release X.Y.0"
Create a branch for the release series (ensure origin/master is up-to-date with the ASF repo before running it):
Code Block language bash git checkout -b branch-X.Y origin/master
Switch back to master and add
A.B.C-SNAPSHOT (unreleased changes)
to CHANGES.txt.Code Block language bash git checkout - (edit CHANGES.txt)
Bump the version number in the master branch:
Code Block language bash ./gradlew setversion -Pnextversion="A.B.C-SNAPSHOT"
Commit these changes to the master and push.
Code Block language bash git add CHANGES.txt (and other updated files) git commit -m "Bumping up the version to A.B.C-SNAPSHOT" git push upstream master (assuming that ASF repo is referred to as 'upstream')
Checkout the release branch
Code Block language bash git checkout branch-X.Y
Update the release branch's version information: the version number in the release branch ends in
-SNAPSHOT
, but we need to remove this for the release. For example,0.8.0-SNAPSHOT
needs to be changed to0.8.0
.Code Block language bash ./gradlew setversion -Pnextversion="X.Y.Z"
Rename docker images and repo URLs:
Code Block language bash grep -R --color :trunk- * | grep yaml grep -R --color "http://repos.bigtop.apache.org/releases/X.Y.Z" * grep -R --color "http://repos.bigtop.apache.org/releases/x.y.z" * (where "x.y.z" is the previous version of "X.Y.Z". Just in case) (replace the occurrences found by the above commands)
Commit these changes to the release branch and push
Code Block language bash git add (updated files) git commit -m "Changing version to X.Y.Z" git push upstream branch-X.Y (assuming that ASF repo is referred to as 'upstream')
Also, update the default version of Bigtop (and your code signing key ID, if needed) defined in hieradata on master and the release branch. For example:
Code Block language bash $ git checkout master (edit bigtop-deploy/puppet/hieradata/bigtop/repo.yaml) $ git diff diff --git a/bigtop-deploy/puppet/hieradata/bigtop/repo.yaml b/bigtop-deploy/puppet/hieradata/bigtop/repo.yaml index 48c8a4ae..0d6a9126 100644 --- a/bigtop-deploy/puppet/hieradata/bigtop/repo.yaml +++ b/bigtop-deploy/puppet/hieradata/bigtop/repo.yaml @@ -1,5 +1,5 @@ bigtop::bigtop_repo_gpg_check: true -bigtop::bigtop_repo_apt_key: "BB95B97B18226C73CB2838D1DBBF9D42B7B4BD70" +bigtop::bigtop_repo_apt_key: "01621A73025BDCA30F4159C62922A48261524827" bigtop::bigtop_repo_yum_key_url: "https://downloads.apache.org/bigtop/KEYS" -bigtop::bigtop_repo_default_version: "1.4.0" +bigtop::bigtop_repo_default_version: "1.5.0" $ git add bigtop-deploy/puppet/hieradata/bigtop/repo.yaml $ git commit -m "Update default values in hieradata for release 1.5.0" [master 9715ca7a] Update default values in hieradata for release 1.5.0 1 file changed, 2 insertions(+), 2 deletions(-) $ git push upstream master $ git checkout branch-1.5 $ git cherry-pick -x 9715ca7a $ git push upstream branch-1.5
- Shall you need to commit additional fixes into ongoing release, the commits should go to the release branch and only then be merged into master. Doing this other way around forces
git cherry-pick
which leads to discrepancies in the commit hash-codes and makes the branch look untidy and hard to follow.
For a patch release, just do the following four items on the branch-X.Y, instead of doing all of the items above:
- Update the release branch's version information
- Rename docker images and repo URLs
- Update the default version of Bigtop
- Commit these changes to the release branch and push
3. Generate the Release Notes
...
Manually check this list for accuracy! I've repeatedly seen closed , since JIRA can be updated after step 1, even if it's unintentional. Also, I've repeatedly seen closed bugs that were not fixed (i.e., duplicate) marked with a fix version, so that they incorrectly show up in this list. Find those, edit them to remove the fix release (only actually fixed bugs should have a fix release) and re-run the report. A better way to deal with it is to run
...
Code Block | ||
---|---|---|
| ||
git tag release-x.y.z-RC0 -m "Bigtop X.Y.Z release." git push upstream release--tags x.y.z-RC0 |
5. Build and run Package and Smoke Tests
5.1. Build bigtop/slaves Docker imagesDocker images
Create a release specific job to build images such as Docker-Puppet-x.y.z, Docker-Toolchain-x.y.z and Docker-Toolchain-x.y.z-pull by copying existing jobs:
- https://ci.bigtop.apache.org/view/Docker/job/Docker-Puppet-1.4.0/
- https://ci.bigtop.apache.org/view/Docker/job/Docker-Toolchain-1.4.0/
- https://ci.bigtop.apache.org/view/Docker/job/Docker-Toolchain-1.4.0-pull/
Make sure there is enough free space on the disk of Jenkins worker nodes. Remove workspaces and Docker images for old releases if you need to reclaim the free space.
Make sure all the built images are uploaded to Dockerhub after running the job.Create a release specific job to build images:
...
...
...
...
...
Make sure all the built images are uploaded to Dockerhub
...
Make sure all the built images are downloaded on dockeron docker-slave-06 and docker-slave-07 https://ci.bigtop.apache.org/view/Docker/job/(by Docker-Toolchain-1x.4y.0z-pull/).
5.2. Build RPM/DEB packages
...
Code Block | ||
---|---|---|
| ||
cd ~/ # Change the image for AARCH64 or PPC64LE docker run -ti --rm -v $PWD:/tmpwork bigtop/puppet:1.4.0-centos-7 bash |
...
Code Block | ||
---|---|---|
| ||
cd /tmpwork OS=centos-7 # Sign all RPM packages (This step required to input passphrase, so don't copy and paste the entire script here) rpm --addsign `find ${OS} -name \*rpm` # Recreate the metadata for repository createrepo ${OS} # Armor the metadata gpg --detach-sign --armor ${OS}/repodata/repomd.xml |
...
Code Block | ||
---|---|---|
| ||
cd ~/ # Change the image for AARCH64 or PPC64LE docker run -ti --rm -v $PWD:/tmpwork bigtop/puppet:1.4.0-debian-9 bash |
...
Code Block | ||
---|---|---|
| ||
cd /tmpwork VERSION=1.4.0 OS=debian-9 ARCH=amd64 #ARCH=arm64 #ARCH=ppc64el SIGN_KEY=B7B4BD70 export GPG_TTY=$(tty) # Sign DEB packages (This step required to input passphrase, so don't copy and paste the entire script here) dpkg-sig --cache-passphrase --sign builder --sign-changes force_full `find ${OS}/ -name \*deb` # Build signed apt repository mkdir -p conf cat > conf/distributions <<__EOT__ Origin: Bigtop Label: Bigtop Suite: stable Codename: bigtop Version: ${VERSION} Architectures: ${ARCH} source Components: contrib Description: Apache Bigtop SignWith: ${SIGN_KEY} __EOT__ cat > conf/options <<__EOT__ verbose ask-passphrase __EOT__ rm -rf ${OS}/apt # Note that this command creates the apt repository (db, dists, pool) # in the current directory, so you can't ran it for multiple distros in parallel. reprepro --ask-passphrase -Vb . includedeb bigtop `find ${OS}/ -name \*deb` rm -rf ./${OS}/* mv conf db dists pool ${OS} |
...
For YUM repos, upload files /tmpwork/centos-7/* into into repos.bigtop.apache.org/releases/1.4.0/centos/7/x86_64/. For example:
...
For APT repos, upload files /tmpwork/debian-9/* into into repos.bigtop.apache.org/releases/1.4.0/debian/9//. For example:
...
Create one for each of our Distro. Following are examples for YUM and APT:
Code Block | ||
---|---|---|
| ||
VERSION=1.4.0 RC_NUMBER=0 DIR=bigtop-${VERSION}-RC${RC_NUMBER}/repos mkdir -p ${DIR} cd ${DIR} # YUM for OS in centos-7 fedora-26 opensuse-42.3; do cat > bigtop.repo <<__EOT__ [bigtop] name=Bigtop enabled=1 gpgcheck=1 baseurl=http://repos.bigtop.apache.org/releases/${VERSION}/${OS/-/\/}/\$basearch gpgkey=https://dist.apache.org/repos/dist/release/bigtop/KEYS __EOT__ gpg --detach-sign --armor bigtop.repo mkdir "${OS//\/}" mv bigtop.repo* "${OS//\/}" done # APT for OS in debian-9 ubuntu-16.04; do cat > bigtop.list <<__EOT__ deb http://repos.bigtop.apache.org/releases/${VERSION}/${OS/-/\/}/\$(ARCH) bigtop contrib __EOT__ gpg --detach-sign --armor bigtop.list mkdir "${OS//\/}" mv bigtop.list* "${OS//\/}" done |
Add your signed armored GPG key to repos
directory to ease the key import for the users
Code Block | ||
---|---|---|
| ||
gpg --armor --export <your name> >> GPG-KEY-bigtop gpg --detach-sign --armor GPG-KEY-bigtop |
Result looks like below:
No Formatcode |
---|
GPG-KEY-bigtop GPG-KEY-bigtop.asc centos7/ debian9/ fedora26/ opensuse42.3/ ubuntu16.04/ |
...
First we need to prepare a build environment. Mac OS X is unlikely supported because we'll run through some tests that depends on the OS. Here I'm running on Ubuntu-18.04 (There's an issue for running on 16.04 traced by BIGTOP-2830).
Code Block | |
---|---|
| |
| |
cd ~/ docker run -ti -u jenkins -v $PWD:/tmpwork bigtop/slaves:trunk-ubuntu-18.04 bash -l # The subsequence commands should be executed inside the docker container cp -r /tmpwork/.gnupg ~/ |
Create a maven settings file ~/.m2/settings.xml with the following content:
No Formatcode |
---|
<settings> <servers> <server> <id>apache.snapshots.https</id> <username>APACHE-ID</username> <password>APACHE-PASSWORD</password> </server> <server> <id>apache.staging.https</id> <username>APACHE-ID</username> <password>APACHE-PASSWORD</password> </server> <server> <id>apache.releases.https</id> <username>APACHE-ID</username> <password>APACHE-PASSWORD</password> </server> </servers> <profiles> <profile> <id>gpg</id> <properties> <gpg.executable>gpg</gpg.executable> <gpg.passphrase>GPG-PASSWORD</gpg.passphrase> </properties> </profile> </profiles> <activeProfiles> <activeProfile>gpg</activeProfile> </activeProfiles> </settings> |
Build the artifacts:
Code Block | ||
---|---|---|
| ||
mvn clean site mvn -Prelease package assembly:single |
The following command deploys the binary release artifacts for iTest, tests and other helper files, checksums, and signatures (you will need to enter a GPG passphrase. If you got a "gpg: signing failed: Inappropriate ioctl for device
" error, try `export GPG_TTY=$(tty)
`) to the Apache Staging repo.
Code Block | ||
---|---|---|
| ||
mvn deploy -Prelease -f pom.xml
mvn deploy -Prelease -f bigtop-test-framework/pom.xml
mvn deploy -Prelease -f bigtop-tests/test-artifacts/pom.xml
HADOOP_CONF_DIR=/etc/hadoop/conf HADOOP_HOME=/usr/lib/hadoop mvn deploy -Prelease -f bigtop-tests/test-execution/pom.xml -DskipITs |
...
If you have multiple keys, the build process seems to pick up the first one w/o asking. Make sure you're using the CODE SIGNING KEY by explicitly specifying it. For example:
Code Block | |
---|---|
| |
| |
mvn deploy -Prelease -f pom.xml -Dgpg.keyname="Evans Ye (CODE SIGNING KEY) <evansye@apache.org>" mvn deploy -Prelease -f bigtop-test-framework/pom.xml -Dgpg.keyname="Evans Ye (CODE SIGNING KEY) <evansye@apache.org>" mvn deploy -Prelease -f bigtop-tests/test-artifacts/pom.xml -Dgpg.keyname="Evans Ye (CODE SIGNING KEY) <evansye@apache.org>" mvn deploy -Prelease -f bigtop-tests/test-execution/pom.xml -Dgpg.keyname="Evans Ye (CODE SIGNING KEY) <evansye@apache.org>" |
...
The artifacts that end up in the distribution directory are the source distributions (along with their checksums and signatures), so they need to be copied from the Maven repo to a release candidate directory on apache dist, so the vote can begin:
Code Block | ||
---|---|---|
| ||
VERSION=X.Y.Z RC_NUMBER=n REPOSITORY_ID=xxxx # Should be a number, for example 1013. Find the number in NEXUS staging repository: orgapachebigtop-1013 svn checkout https://dist.apache.org/repos/dist/dev/bigtop bigtop-dist-dev cd bigtop-dist-dev mkdir bigtop-${VERSION}-RC${RC_NUMBER} cd bigtop-${VERSION}-RC${RC_NUMBER} # md5 and sha1 are out-of-date and should not be used, see: http://www.apache.org/dev/release-distribution#sigs-and-sums wget --no-check-certificate https://repository.apache.org/content/repositories/orgapachebigtop-${REPOSITORY_ID}/org/apache/bigtop/bigtop/${VERSION}/bigtop-${VERSION}-project.tar.gz wget --no-check-certificate https://repository.apache.org/content/repositories/orgapachebigtop-${REPOSITORY_ID}/org/apache/bigtop/bigtop/${VERSION}/bigtop-${VERSION}-project.tar.gz.asc # manually generate sha256 and sha512 sha256sum bigtop-${VERSION}-project.tar.gz > bigtop-${VERSION}-project.tar.gz.sha256 sha512sum bigtop-${VERSION}-project.tar.gz > bigtop-${VERSION}-project.tar.gz.sha512 cd .. svn add bigtop-${VERSION}-RC${RC_NUMBER} svn ci -m "Apache Bigtop ${VERSION}-RC${RC_NUMBER}" |
8. Sanity Check
Code Block | |
---|---|
| |
| |
VERSION=X.Y.Z RC_NUMBER=n cd /tmpwork git clone https://gitbox.apache.org/repos/asf/bigtop.git -b release-${VERSION}-RC${RC_NUMBER} --depth 1 wget --no-parent --recursive --reject='index.html*' https://dist.apache.org/repos/dist/dev/bigtop/bigtop-${VERSION}-RC${RC_NUMBER}/ # make sure that the source code tree and the release tarball are identical except for .git/CI/site releated files tar xf dist.apache.org/repos/dist/dev/bigtop/bigtop-${VERSION}-RC${RC_NUMBER}/bigtop-${VERSION}-project.tar.gz diff -r bigtop bigtop-${VERSION} # make sure that the signature and checksums are correct cd dist.apache.org/repos/dist/dev/bigtop/bigtop-${VERSION}-RC${RC_NUMBER} gpg --verify bigtop-${VERSION}-project.tar.gz.asc bigtop-${VERSION}-project.tar.gz diff <(sha256sum bigtop-${VERSION}-project.tar.gz) bigtop-${VERSION}-project.tar.gz.sha256 diff <(sha512sum bigtop-${VERSION}-project.tar.gz) bigtop-${VERSION}-project.tar.gz.sha512 # make sure that all files exist and their signatures are correct for i in $(find repos -not -name '*.asc' -type f); do echo ${i}; gpg --verify ${i}.asc ${i}; done |
...
This step makes the artifacts available on the mirrors..
Code Block | ||
---|---|---|
| ||
VERSION=X.Y.Z #Example: 1.3.0 CANDIDATE=C #Example: RC2 svn co https://dist.apache.org/repos/dist/release/bigtop cd bigtop svn mv https://dist.apache.org/repos/dist/dev/bigtop/bigtop-$VERSION-$CANDIDATE https://dist.apache.org/repos/dist/release/bigtop/bigtop-$VERSION rm stable ln -s bigtop-$VERSION stable svn commit |
...
Create permanent release tag under rel/
Code Block | ||
---|---|---|
| ||
git checkout release-x.y.z git tag rel/x.y.z -u <signing key ID> git push --tags |
Wait 24 Hours
It takes up to 24 hours for all the mirrors to sync, so don't announce the new release just yet.
We had to wait 24 hours before, but it's not true now since ASF adopted CDN instead of mirror sites. According to the FAQs of the ASF infra:
Apache uses a global content distribution network (CDN) which collects new releases almost as soon as you post them.
The files therefore become available for download almost immediately. You probably don't need to wait more than fifteen minutes before announcing a release.
Build and Deploy Site
Update files under src/site by referring to past commits such as https://github.com/apache/bigtop/commit/678375eb75f7367f9114a3ad8087ba40f21444b6 or https://github.com/apache/bigtop/commit/d5c45a2879f8e3a8d29a36386ad8904a4e71a274. Commit into master and push them.
...
As said in
Jira | ||||||
---|---|---|---|---|---|---|
|
Code Block | |
---|---|
| |
| |
VERSION=X.Y.Z #Example: 1.2.1 svn co https://dist.apache.org/repos/dist/release/bigtop cd bigtop svn delete bigtop-$VERSION svn commit |
...