...
See http://stackoverflow.com/questions/3618330/what-is-the-format-of-svn-settings-xml-for-use-with-maven-scm-plugin how to change your scm:svn: url and provide a password.
Done with the above? Proceed with the rest of the steps:
...
Update CHANGES.txt in master to replace
Release X.Y.0
(unreleased changes)Release X.Y.0 - (YYYY-MM-DD
). Commit:Code Block noformatlanguage bash git add CHANGES.txt git commit -m "Preparing for release X.Y.Z0"
Create a branch for the release series :(ensure origin/master is up-to-date with the ASF repo before running it):
Code Block noformatlanguage bash git checkout -b branch-X.Y.Z origin/master
- Add
Switch back to master and add
changesA.B.C-SNAPSHOT (unreleased
in master.)
to CHANGES.txt.
Code Block language bash git checkout - (edit CHANGES.txt)
Bump the version number in the master branch:
Code Block noformatlanguage bash ./gradlew setversion -Pnextversion="A.B.C-SNAPSHOT"
Commit these changes to the master and push.
Code Block language bash git add CHANGES.txt (and other updated files) git commit -m "Bumping up the version to A.B.C-SNAPSHOT" git push upstream master (assuming that ASF repo is referred to as 'upstream')
Checkout the release branch
Code Block language bash Checkout the release branch
No Format git checkout branch-X.Y.Z
Update the release branch's version information: the version number in the release branch ends in
-SNAPSHOT
, but we need to remove this for the release. For example,0.8.0-SNAPSHOT
needs to be changed to0.8.0
.No Formatcode language bash ./gradlew setversion -Pnextversion="X.Y.Z"
Rename docker images and repo URLs:
Code Block language bash grep -R --color :trunk- * | grep yaml grep -R --color "http://repos.bigtop.apache.org/releases/
Commit these changes to the release branch and push
No Format git commit -m "Changing version to X.Y.Z"
- Shall you need to commit additional fixes into ongoing release, the commits should go to the release branch and only then be merged into master. Doing this other way around forces
git cherry-pick
which leads to discrepancies in the commit hash-codes and makes the branch look untidy and hard to follow.
3. Generate the Release Notes
JIRA has the ability to generate release notes automatically (this is why it is so important to keep the fix version number accurate).
...
* grep -R --color "http://repos.bigtop.apache.org/
...
Manually check this list for accuracy! I've repeatedly seen closed bugs that were not fixed (i.e., duplicate) marked with a fix version, so that they incorrectly show up in this list. Find those, edit them to remove the fix release (only actually fixed bugs should have a fix release) and re-run the report. A better way to deal with it is to run
...
releases/x.y.z" * (where "x.y.z" is the previous version of "X.Y.Z". Just in case) (replace the occurrences found by the above commands)
Commit these changes to the release branch and push
Code Block language bash git add (updated files) git commit -m "Changing version to X.Y.Z" git
...
push
...
upstream branch-X.Y (assuming that ASF repo is referred to as 'upstream')
Also, update the default version of Bigtop (and your code signing key ID, if needed) defined in hieradata on master and the release branch. For example:
Code Block language bash $ git checkout master (edit bigtop-deploy/puppet/hieradata/bigtop/repo.yaml) $ git diff diff --git a/bigtop-deploy/puppet/hieradata/bigtop/repo.yaml b/bigtop-deploy/puppet/hieradata/bigtop/repo.yaml index 48c8a4ae..0d6a9126 100644 --- a/bigtop-deploy/puppet/hieradata/bigtop/repo.yaml +++ b/bigtop-deploy/puppet/hieradata/bigtop/repo.yaml @@ -1,5 +1,5 @@ bigtop::bigtop_repo_gpg_check: true -bigtop::bigtop_repo_apt_key: "BB95B97B18226C73CB2838D1DBBF9D42B7B4BD70" +bigtop::bigtop_repo_apt_key: "01621A73025BDCA30F4159C62922A48261524827" bigtop::bigtop_repo_yum_key_url: "https://downloads.apache.org/bigtop/KEYS" -bigtop::bigtop_repo_default_version: "1.4.0" +bigtop::bigtop_repo_default_version: "1.5.0" $ git add bigtop-deploy/puppet/hieradata/bigtop/repo.yaml $ git commit -m "Update default values in hieradata for release 1.5.0" [master 9715ca7a] Update default values in hieradata for release 1.5.0 1 file changed, 2 insertions(+), 2 deletions(-) $ git push upstream master $ git checkout branch-1.5 $ git cherry-pick -x 9715ca7a $ git push upstream branch-1.5
- Shall you need to commit additional fixes into ongoing release, the commits should go to the release branch and only then be merged into master. Doing this other way around forces
git cherry-pick
which leads to discrepancies in the commit hash-codes and makes the branch look untidy and hard to follow.
For a patch release, just do the following four items on the branch-X.Y, instead of doing all of the items above:
- Update the release branch's version information
- Rename docker images and repo URLs
- Update the default version of Bigtop
- Commit these changes to the release branch and push
3. Generate the Release Notes
JIRA has the ability to generate release notes automatically (this is why it is so important to keep the fix version number accurate).
https://issues.apache.org/jira/secure/ConfigureReleaseNote.jspa?projectId=12311420
Manually check this list for accuracy, since JIRA can be updated after step 1, even if it's unintentional. Also, I've repeatedly seen closed bugs that were not fixed (i.e., duplicate) marked with a fix version, so that they incorrectly show up in this list. Find those, edit them to remove the fix release (only actually fixed bugs should have a fix release) and re-run the report. A better way to deal with it is to run
Code Block |
---|
fixVersion = X.Y.Z AND project = BIGTOP AND resolution = Duplicate ORDER by priority DESC |
Select the correct version. Ask for both plain text and HTML formats.
Paste the plain text notes to the top of CHANGES.txt. Paste the HTML version into src/site/xdoc/release-notes.xml. Check this into master and the branch.
Wrap the title ("Release Notes - Bigtop - Version X.Y.Z") inside an <h3> element.
4. Commit and Tag
Tag, and push the changes and the tag to git:
Code Block | ||
---|---|---|
| ||
git tag release-x.y.z-RC0 -m "Bigtop X.Y.Z release."
git push upstream release-x.y.z-RC0 |
5. Build and run Package and Smoke Tests
5.1. Build bigtop/slaves Docker images
Create a release specific job to build images such as Docker-Puppet-x.y.z, Docker-Toolchain-x.y.z and Docker-Toolchain-x.y.z-pull by copying existing jobs:
- https://ci.bigtop.apache.org/view/Docker/job/Docker-Puppet-1.4.0/
- https://ci.bigtop.apache.org/view/Docker/job/Docker-Toolchain-1.4.0/
- https://ci.bigtop.apache.org/view/Docker/job/Docker-Toolchain-1.4.0-pull/
Make sure there is enough free space on the disk of Jenkins worker nodes. Remove workspaces and Docker images for old releases if you need to reclaim the free space.
Make sure all the built images are uploaded to Dockerhub after running the job.
Make sure all the built images are downloaded on docker-slave-06 and docker-slave-07 (by Docker-Toolchain-x.y.z-pull).
5.2. Build RPM/DEB packages
Create a release specific job to build packages:
https://ci.bigtop.apache.org/view/Releases/job/Bigtop-1.4.0/
Download all built packages via archive download feature provided by Jenkins on a machine that you want to proceed the signing:
Code Block | ||
---|---|---|
| ||
PLATFORM=amd64
#PLATFORM=aarch64 # enable when running on the ARM platform
#PLATFORM=ppc64le # enable when running on the PowerPC platform
# Download artifacts for all OS supported in Bigtop.
# Note that you should run this on each platform (x86_64, ARM, and PowerPC), because
# you will sign the artifacts and create repository in Docker container later,
# which are dependent on the platform.
for DISTRO in centos-7 fedora-26 opensuse-42.3 debian-9 ubuntu-16.04; do
curl -L https://ci.bigtop.apache.org/view/Releases/job/Bigtop-1.4.0/DISTRO=${DISTRO},PLATFORM=${PLATFORM}-slave/lastSuccessfulBuild/artifact/*zip*/archive.zip -o ${DISTRO}.zip
done
# Extract the downloaded archive. If you come across the
# "invalid zip file with overlapped components (possible zip bomb)"
# error, use other tools than unzip, e.g., `jar xf "${i}.zip"`
for i in centos-7 fedora-26 opensuse-42.3 debian-9 ubuntu-16.04; do
unzip "${i}.zip"
mkdir "${i}"
mv archive/output/* "${i}"
rm -rf archive
done |
5.3. Sign RPM packages and yum repos
Ref:
Jira | ||||||
---|---|---|---|---|---|---|
|
Startup a docker images that is RPM based system:
Code Block | ||
---|---|---|
| ||
cd ~/
# Change the image for AARCH64 or PPC64LE
docker run -ti --rm -v $PWD:/work bigtop/puppet:1.4.0-centos-7 bash |
Prepare the environment for signing:
Code Block | ||
---|---|---|
| ||
yum install -y rpm-sign createrepo pinentry
gpg --import YOUR_CODE_SIGNING_SECRET_KEY
echo "%_gpg_name YOUR_CODE_SIGNING_KEY_ID" > ~/.rpmmacros
|
Signing:
Code Block | ||
---|---|---|
| ||
cd /work
OS=centos-7
# Sign all RPM packages (This step required to input passphrase, so don't copy and paste the entire script here)
rpm --addsign `find ${OS} -name \*rpm`
# Recreate the metadata for repository
createrepo ${OS}
# Armor the metadata
gpg --detach-sign --armor ${OS}/repodata/repomd.xml |
[OPENSUSE ONLY]
Code Block | ||
---|---|---|
| ||
gpg --armor --export evansye@apache.org > opensuse-42.3/repodata/repomd.xml.key
for i in `find opensuse-42.3/repodata -name *.xml.gz` opensuse-42.3/repodata/repomd.xml.key ; do gpg --detach-sign --armor $i ; done |
5.4. Sign DEB packages and apt repos
Ref:
Jira | ||||||
---|---|---|---|---|---|---|
|
Startup a docker images that is DEB based system:
Code Block | ||
---|---|---|
| ||
cd ~/
# Change the image for AARCH64 or PPC64LE
docker run -ti --rm -v $PWD:/work bigtop/puppet:1.4.0-debian-9 bash |
Prepare the environment for signing:
Code Block | ||
---|---|---|
| ||
apt-get update
apt-get install -y gpg libterm-readkey-perl dpkg-sig reprepro
gpg --import YOUR_CODE_SIGNING_SECRET_KEY |
Signing:
Code Block | ||
---|---|---|
| ||
cd /work
VERSION=1.4.0
OS=debian-9
ARCH=amd64
#ARCH=arm64
#ARCH=ppc64el
SIGN_KEY=B7B4BD70
export GPG_TTY=$(tty)
# Sign DEB packages (This step required to input passphrase, so don't copy and paste the entire script here)
dpkg-sig --cache-passphrase --sign builder --sign-changes force_full `find ${OS}/ -name \*deb`
# Build signed apt repository
mkdir -p conf
cat > conf/distributions <<__EOT__
Origin: Bigtop
Label: Bigtop
Suite: stable
Codename: bigtop
Version: ${VERSION}
Architectures: ${ARCH} source
Components: contrib
Description: Apache Bigtop
SignWith: ${SIGN_KEY}
__EOT__
cat > conf/options <<__EOT__
verbose
ask-passphrase
__EOT__
rm -rf ${OS}/apt
# Note that this command creates the apt repository (db, dists, pool)
# in the current directory, so you can't ran it for multiple distros in parallel.
reprepro --ask-passphrase -Vb . includedeb bigtop `find ${OS}/ -name \*deb`
rm -rf ./${OS}/*
mv conf db dists pool ${OS} |
5.5. Upload to S3
The easiest way to upload artifacts to S3 is via your own AWS account. Add your account(email) to bigtop's bucket in the section "Access for other AWS accounts":
Once permission granted, you're able to use your account's access key and secret key with aws s3 sync command for upload, like:
Code Block | ||
---|---|---|
| ||
# In case of CentOS 7
aws s3 sync --acl public-read ./centos-7/ s3://repos.bigtop.apache.org/releases/1.4.0/centos/7/x86_64/ # for x86_64/amd64
aws s3 sync --acl public-read ./centos-7/ s3://repos.bigtop.apache.org/releases/1.4.0/centos/7/aarch64/ # for aarch64/arm64
aws s3 sync --acl public-read ./centos-7/ s3://repos.bigtop.apache.org/releases/1.4.0/centos/7/ppc64le/ # for ppc64le/ppc64el
# In case of Ubuntu 16.04
aws s3 sync --acl public-read ./ubuntu-16.04/ s3://repos.bigtop.apache.org/releases/1.4.0/ubuntu/16.04/amd64/ # for x86_64/amd64
aws s3 sync --acl public-read ./ubuntu-16.04/ s3://repos.bigtop.apache.org/releases/1.4.0/ubuntu/16.04/arm64/ # for aarch64/arm64
aws s3 sync --acl public-read ./ubuntu-16.04/ s3://repos.bigtop.apache.org/releases/1.4.0/ubuntu/16.04/ppc64el/ # for ppc64le/ppc64el |
The directory layouts on S3 bucket looked like the following:
Code Block |
---|
repos.bigtop.apache.org/releases/1.4.0/centos/7/x86_64
repos.bigtop.apache.org/releases/1.4.0/fedora/26/x86_64
repos.bigtop.apache.org/releases/1.4.0/opensuse/42.3/x86_64
repos.bigtop.apache.org/releases/1.4.0/debian/9/amd64
repos.bigtop.apache.org/releases/1.4.0/ubuntu/16.04/amd64 |
For YUM repos, upload files /work/centos-7/* into repos.bigtop.apache.org/releases/1.4.0/centos/7/x86_64/. For example:
Code Block |
---|
repos.bigtop.apache.org/releases/1.4.0/centos/7/x86_64/alluxio
repos.bigtop.apache.org/releases/1.4.0/centos/7/x86_64/ambari
...
repos.bigtop.apache.org/releases/1.4.0/centos/7/x86_64/repodata
... |
For APT repos, upload files /work/debian-9/* into repos.bigtop.apache.org/releases/1.4.0/debian/9//. For example:
Code Block |
---|
repos.bigtop.apache.org/releases/1.4.0/debian/9/amd64/conf
repos.bigtop.apache.org/releases/1.4.0/debian/9/amd64/db
repos.bigtop.apache.org/releases/1.4.0/debian/9/amd64/dists
repos.bigtop.apache.org/releases/1.4.0/debian/9/amd64/pool |
5.6. Create repo files
Create one for each of our Distro. Following are examples for YUM and APT:
Code Block | ||
---|---|---|
| ||
VERSION=1.4.0
RC_NUMBER=0
DIR=bigtop-${VERSION}-RC${RC_NUMBER}/repos
mkdir -p ${DIR}
cd ${DIR}
# YUM
for OS in centos-7 fedora-26 opensuse-42.3; do
cat > bigtop.repo <<__EOT__
[bigtop]
name=Bigtop
enabled=1
gpgcheck=1
baseurl=http://repos.bigtop.apache.org/releases/${VERSION}/${OS/-/\/}/\$basearch
gpgkey=https://dist.apache.org/repos/dist/release/bigtop/KEYS
__EOT__
gpg --detach-sign --armor bigtop.repo
mkdir "${OS//\/}"
mv bigtop.repo* "${OS//\/}"
done
# APT
for OS in debian-9 ubuntu-16.04; do
cat > bigtop.list <<__EOT__
deb http://repos.bigtop.apache.org/releases/${VERSION}/${OS/-/\/}/\$(ARCH) bigtop contrib
__EOT__
gpg --detach-sign --armor bigtop.list
mkdir "${OS//\/}"
mv bigtop.list* "${OS//\/}"
done |
Add your signed armored GPG key to repos
directory to ease the key import for the users
Code Block | ||
---|---|---|
| ||
gpg --armor --export <your name> >> GPG-KEY-bigtop
gpg --detach-sign --armor GPG-KEY-bigtop |
Result looks like below:
Code Block |
---|
GPG-KEY-bigtop
GPG-KEY-bigtop.asc
centos7/
debian9/
fedora26/
opensuse42.3/
ubuntu16.04/ |
5.7. Commit repo files into https://dist.apache.org/repos/dist/dev/bigtop/bigtop-X.Y.Z-RCn/repos/
5.8. Run smoke tests with the uploaded packages
Create a release specific job to deploy and run smoke test with the uploaded packages.
You may have to rerun the job several times to stabilize the result due to download failure. Leverage the "Matrix Reloaded" plugin so as to rebuild only unstable and failed combinations.
https://ci.bigtop.apache.org/view/Test/job/Bigtop-trunk-smoke-tests-1.4.0/
Summarize its result on the Wiki.
https://cwiki.apache.org/confluence/display/BIGTOP/Overview+of+Bigtop+1.4.0+Support+Matrix
6. Build and Deploy Artifacts
First we need to prepare a build environment. Mac OS X is unlikely supported because we'll run through some tests that depends on the OS. Here I'm running on Ubuntu-18.04 (There's an issue for running on 16.04 traced by BIGTOP-2830).
Code Block | ||
---|---|---|
| ||
cd ~/
docker run -ti -u jenkins -v $PWD:/work bigtop/slaves:trunk-ubuntu-18.04 bash -l
# The subsequence commands should be executed inside the docker container
cp -r /work/.gnupg ~/ |
Create a maven settings file ~/.m2/settings.xml with the following content:
Code Block |
---|
<settings>
<servers>
<server>
<id>apache.snapshots.https</id>
<username>APACHE-ID</username>
<password>APACHE-PASSWORD</password>
</server>
<server>
<id>apache.staging.https</id>
<username>APACHE-ID</username>
<password>APACHE-PASSWORD</password>
</server>
<server>
<id>apache.releases.https</id>
<username>APACHE-ID</username>
<password>APACHE-PASSWORD</password>
</server>
</servers>
<profiles>
<profile>
<id>gpg</id>
<properties>
<gpg.executable>gpg</gpg.executable>
<gpg.passphrase>GPG-PASSWORD</gpg.passphrase>
</properties>
</profile>
</profiles>
<activeProfiles>
<activeProfile>gpg</activeProfile>
</activeProfiles>
</settings>
|
Build the artifacts:
Code Block | ||
---|---|---|
| ||
mvn clean site
mvn -Prelease package assembly:single
|
The following command deploys the binary release artifacts for iTest, tests and other helper files, checksums, and signatures (you will need to enter a GPG passphrase. If you got a "gpg: signing failed: Inappropriate ioctl for device
" error, try `export GPG_TTY=$(tty)
`) to the Apache Staging repo.
Code Block | ||
---|---|---|
| ||
mvn deploy -Prelease -f pom.xml
mvn deploy -Prelease -f bigtop-test-framework/pom.xml
mvn deploy -Prelease -f bigtop-tests/test-artifacts/pom.xml
HADOOP_CONF_DIR=/etc/hadoop/conf HADOOP_HOME=/usr/lib/hadoop mvn deploy -Prelease -f bigtop-tests/test-execution/pom.xml -DskipITs |
For the last step to succeed you'd need to export HADOOP_HOME and HADOOP_CONF_DIR (BIGTOP-1464): the values don't matter - it is just a workaround. Also, the -DskipITs option for the last step should be removed in the future by resolving BIGTOP-3461.
If the deployment step fails with an Access denied error check that you have the required permissions on Nexus.
If you have multiple keys, the build process seems to pick up the first one w/o asking. Make sure you're using the CODE SIGNING KEY by explicitly specifying it. For example:
Code Block | ||
---|---|---|
| ||
mvn deploy -Prelease -f pom.xml -Dgpg.keyname="Evans Ye (CODE SIGNING KEY) <evansye@apache.org>"
mvn deploy -Prelease -f bigtop-test-framework/pom.xml -Dgpg.keyname="Evans Ye (CODE SIGNING KEY) <evansye@apache.org>"
mvn deploy -Prelease -f bigtop-tests/test-artifacts/pom.xml -Dgpg.keyname="Evans Ye (CODE SIGNING KEY) <evansye@apache.org>"
mvn deploy -Prelease -f bigtop-tests/test-execution/pom.xml -Dgpg.keyname="Evans Ye (CODE SIGNING KEY) <evansye@apache.org>" |
Login to https://repository.apache.org using your Apache SVN credentials. Click on Staging on the left. Then click on orgapachebigtop in the list of repositories. In the panel below you should see an open repository that is linked to your username and IP. Select this repository and click Close. This will close the repository from future deployments and make it available for others to view.
BIGTOP-1463 is open to track the improvements of the release process.
7. Copy Release Artifacts
The artifacts that end up in the distribution directory are the source distributions (along with their checksums and signatures), so they need to be copied from the Maven repo to a release candidate directory on apache dist, so the vote can begin:
Code Block | ||
---|---|---|
| ||
VERSION=X.Y.Z
RC_NUMBER=n
REPOSITORY_ID=xxxx # Should be a number, for example 1013. Find the number in NEXUS staging repository: orgapachebigtop-1013
svn checkout https://dist.apache.org/repos/dist/dev/bigtop bigtop-dist-dev
cd bigtop-dist-dev
mkdir bigtop-${VERSION}-RC${RC_NUMBER}
cd bigtop-${VERSION}-RC${RC_NUMBER}
# md5 and sha1 are out-of-date and should not be used, see: http://www.apache.org/dev/release-distribution#sigs-and-sums
wget --no-check-certificate https://repository.apache.org/content/repositories/orgapachebigtop-${REPOSITORY_ID}/org/apache/bigtop/bigtop/${VERSION}/bigtop-${VERSION}-project.tar.gz
wget --no-check-certificate https://repository.apache.org/content/repositories/orgapachebigtop-${REPOSITORY_ID}/org/apache/bigtop/bigtop/${VERSION}/bigtop-${VERSION}-project.tar.gz.asc
# manually generate sha256 and sha512
sha256sum bigtop-${VERSION}-project.tar.gz > bigtop-${VERSION}-project.tar.gz.sha256
sha512sum bigtop-${VERSION}-project.tar.gz > bigtop-${VERSION}-project.tar.gz.sha512
cd ..
svn add bigtop-${VERSION}-RC${RC_NUMBER}
svn ci -m "Apache Bigtop ${VERSION}-RC${RC_NUMBER}" |
8. Sanity Check
Code Block | ||
---|---|---|
| ||
VERSION=X.Y.Z
RC_NUMBER=n
cd /work
git clone https://gitbox.apache.org/repos/asf/bigtop.git -b release-${VERSION}-RC${RC_NUMBER} --depth 1
wget --no-parent --recursive --reject='index.html*' https://dist.apache.org/repos/dist/dev/bigtop/bigtop-${VERSION}-RC${RC_NUMBER}/
# make sure that the source code tree and the release tarball are identical except for .git/CI/site releated files
tar xf dist.apache.org/repos/dist/dev/bigtop/bigtop-${VERSION}-RC${RC_NUMBER}/bigtop-${VERSION}-project.tar.gz
diff -r bigtop bigtop-${VERSION}
# make sure that the signature and checksums are correct
cd dist.apache.org/repos/dist/dev/bigtop/bigtop-${VERSION}-RC${RC_NUMBER}
gpg --verify bigtop-${VERSION}-project.tar.gz.asc bigtop-${VERSION}-project.tar.gz
diff <(sha256sum bigtop-${VERSION}-project.tar.gz) bigtop-${VERSION}-project.tar.gz.sha256
diff <(sha512sum bigtop-${VERSION}-project.tar.gz) bigtop-${VERSION}-project.tar.gz.sha512
# make sure that all files exist and their signatures are correct
for i in $(find repos -not -name '*.asc' -type f); do echo ${i}; gpg --verify ${i}.asc ${i}; done |
Select the correct version. Ask for both plain text and HTML formats.
Paste the plain text notes to the top of CHANGES.txt. Paste the HTML version into src/site/xdoc/release-notes.xml. Check this into master and the branch.
Wrap the title ("Release Notes - Bigtop - Version X.Y.Z") inside an <h3> element.
4. Commit and Tag
Tag, and push the changes and the tag to git:
No Format |
---|
git tag release-x.y.z -m "Bigtop X.Y.Z release."
git push --tags |
5. Build and run Package and Smoke Tests
5.1. Build bigtop/slaves Docker images
Create a release specific job to build images:
https://ci.bigtop.apache.org/view/Docker/job/Docker-Toolchain-1.2.1/
Make sure all the built images are uploaded to Dockerhub
https://hub.docker.com/r/bigtop/slaves/tags/
No Format |
---|
# Go to bigtop CI build slave: docker-slave-06 and docker-slave-07
# See which image is built on the host
docker images |head
# Only push images successfully built by CI to Dockerhub
for os in centos-6 centos-7 fedora-25 opensuse-42.1 debian-8 ubuntu-16.04; do docker push bigtop/slaves:1.2.1-$os; done |
Make sure all the built images are downloaded on docker-slave-06 and docker-slave-07
No Format |
---|
for os in centos-6 centos-7 fedora-25 opensuse-42.1 debian-8 ubuntu-16.04; do docker pull bigtop/slaves:1.2.1-$os; done |
5.2. Build RPM/DEB packages
Create a release specific job to build packages:
https://ci.bigtop.apache.org/job/Bigtop-1.2.1/
5.3. Sign RPM packages and yum repos
Ref:
Jira | ||||||
---|---|---|---|---|---|---|
|
Clone RPMs built by CI job into a temp dir. You can find built packges under jenkins job's workspace. For example:
No Format |
---|
# Go to the build slave, docker-slave-06 or docker-slave-07, depends on which built the packages
cp /home/jenkins/workspace/Bigtop-1.2.1/OS/centos-7/output ~/centos-7 |
Startup a docker images that is RPM based system:
No Format |
---|
cd ~/
docker run -ti --rm -v $PWD:/tmp bigtop/puppet:centos-7 bash |
Prepare the environment for signing:
No Format |
---|
gpg --import YOUR_CODE_SIGNING_SECRET_KEY
echo > ~/.rpmmacros <<__EOT__
%_gpg_name 8C10EE0A
__EOT__
yum install -y rpm-sign createrepo |
Signing:
No Format |
---|
cd /tmp
# Sign all RPM packages
rpm --addsign `find centos-7 -name \*rpm`
# Recreate the metadata for repository
createrepo centos-7
# Armor the metadata
gpg --detach-sign --armor centos-7/repodata/repomd.xml |
[OPENSUSE ONLY]
No Format |
---|
gpg --armor --export evansye@apache.org > opensuse-42.1/repodata/repomd.xml.key
for i in `find opensuse-42.1/repodata -name *.xml.gz` opensuse-42.1/repodata/repomd.xml.key ; do gpg --detach-sign --armor $i ; done |
5.4. Sign DEB packages and apt repos
Ref:
Jira | ||||||
---|---|---|---|---|---|---|
|
Clone DEBs built by CI job into a temp dir. You can find built packges under jenkins job's workspace. For example:
No Format |
---|
# Go to the build slave, docker-slave-06 or docker-slave-07, depends on which built the packages
cp /home/jenkins/workspace/Bigtop-1.2.1/OS/debian-8/output ~/debian-8 |
Startup a docker images that is DEB based system:
No Format |
---|
cd ~/
docker run -ti --rm -v $PWD:/tmp bigtop/puppet:debian-8 bash |
Prepare the environment for signing:
No Format |
---|
gpg --import YOUR_CODE_SIGNING_SECRET_KEY
apt-get update
apt-get install -y dpkg-sig
apt-get install -y libterm-readkey-perl
apt-get install -y reprepro |
Signing:
No Format |
---|
cd /tmp
# Sign DEB packages
dpkg-sig --cache-passphrase --sign builder `find debian-8/ -name \*deb`
# Build signed apt repository
mkdir -p apt/conf
cat > apt/conf/distributions <<__EOT__
Origin: Bigtop
Label: Bigtop
Suite: stable
Codename: bigtop
Version: 1.2.1
Architectures: amd64 source
Components: contrib
Description: Apache Bigtop
SignWith: 8C10EE0A
__EOT__
cat > apt/conf/options <<__EOT__
verbose
ask-passphrase
__EOT__
reprepro --ask-passphrase -Vb . includedeb bigtop `find debian-8/ -name \*deb` |
5.5. Create repo files
Create one for each of our Distro. Following are examples for YUM and APT:
No Format |
---|
# YUM
cat > bigtop.repo <<__EOT__
[bigtop]
name=Bigtop
enabled=1
gpgcheck=1
type=NONE
baseurl=http://repos.bigtop.apache.org/releases/1.2.1/centos/7/x86_64
gpgkey=https://dist.apache.org/repos/dist/release/bigtop/KEYS
__EOT__
gpg --detach-sign --armor bigtop.repo
# APT
cat > bigtop.list <<__EOT__
deb http://repos.bigtop.apache.org/releases/1.2.1/debian/8/x86_64 bigtop contrib
__EOT__
gpg --detach-sign --armor bigtop.list |
5.6. Upload to S3
Create directory layouts on S3 bucket looked like the following:
No Format |
---|
repos.bigtop.apache.org/releases/1.2.1/centos/6/x86_64
repos.bigtop.apache.org/releases/1.2.1/centos/7/x86_64
repos.bigtop.apache.org/releases/1.2.1/fedora/25/x86_64
repos.bigtop.apache.org/releases/1.2.1/opensuse/42.1/x86_64
repos.bigtop.apache.org/releases/1.2.1/debian/8/x86_64
repos.bigtop.apache.org/releases/1.2.1/ubuntu/16.04/x86_64 |
For YUM repos, upload files /tmp/centos-7/* into repos.bigtop.apache.org/releases/1.2.1/centos/7/x86_64/. For example:
No Format |
---|
repos.bigtop.apache.org/releases/1.2.1/centos/7/x86_64/alluxio
repos.bigtop.apache.org/releases/1.2.1/centos/7/x86_64/ambari
...
repos.bigtop.apache.org/releases/1.2.1/centos/7/x86_64/repodata
... |
For APT repos, upload files /tmp/debian-8/* into repos.bigtop.apache.org/releases/1.2.1/debian/8/x86_64/. For example:
No Format |
---|
repos.bigtop.apache.org/releases/1.2.1/debian/8/x86_64/conf
repos.bigtop.apache.org/releases/1.2.1/debian/8/x86_64/db
repos.bigtop.apache.org/releases/1.2.1/debian/8/x86_64/dists
repos.bigtop.apache.org/releases/1.2.1/debian/8/x86_64/pool |
5.7. Commit repo files into https://dist.apache.org/repos/dist/dev/bigtop/repos
6. Build and Deploy Artifacts
First we need to prepare a build environment. Mac OS X is unlikely supported because we'll run through some tests that depends on the OS. Here I'm running on Ubuntu-14.04 (There's an issue for running on 16.04 traced by BIGTOP-2830).
No Format |
---|
cd ~/
docker run -ti -u jenkins -v $PWD:/tmp bigtop/slaves:trunk-ubuntu-14.04 bash -l
# The subsequence commands should be executed inside the docker container
cp -r /tmp/.gnupg ~/ |
Create a maven settings file ~/.m2/settings.xml with the following content:
No Format |
---|
<settings>
<servers>
<server>
<id>apache.snapshots.https</id>
<username>APACHE-ID</username>
<password>APACHE-PASSWORD</password>
</server>
<server>
<id>apache.staging.https</id>
<username>APACHE-ID</username>
<password>APACHE-PASSWORD</password>
</server>
<server>
<id>apache.releases.https</id>
<username>APACHE-ID</username>
<password>APACHE-PASSWORD</password>
</server>
</servers>
</settings>
|
Build the artifacts:
No Format |
---|
mvn site
mvn -Prelease package assembly:assembly
|
The following command deploys the binary release artifacts for iTest, tests and other helper files, checksums, and signatures (you will need to enter a GPG passphrase) to the Apache Staging repo.
No Format |
---|
mvn deploy -Prelease -f pom.xml
mvn deploy -Prelease -f bigtop-test-framework/pom.xml
mvn deploy -Prelease -f bigtop-tests/test-artifacts/pom.xml
mvn deploy -Prelease -f bigtop-tests/test-execution/pom.xml |
For the last step to succeed you'd need to export HADOOP_HOME and HADOOP_CONF_DIR (BIGTOP-1464): the values don't matter - it is just a workaround. If this step fails with an Access denied error check that you have the required permissions on Nexus.
If the deployment step fails with an Access denied error check that you have the required permissions on Nexus.
If you have multiple keys, the build process seems to pick up the first one w/o asking. Make sure you're using the CODE SIGNING KEY by explicitly specifying it. For example:
No Format |
---|
mvn deploy -Prelease -f pom.xml -Dgpg.keyname="Evans Ye (CODE SIGNING KEY) <evansye@apache.org>"
mvn deploy -Prelease -f bigtop-test-framework/pom.xml -Dgpg.keyname="Evans Ye (CODE SIGNING KEY) <evansye@apache.org>"
mvn deploy -Prelease -f bigtop-tests/test-artifacts/pom.xml -Dgpg.keyname="Evans Ye (CODE SIGNING KEY) <evansye@apache.org>"
mvn deploy -Prelease -f bigtop-tests/test-execution/pom.xml -Dgpg.keyname="Evans Ye (CODE SIGNING KEY) <evansye@apache.org>" |
Login to https://repository.apache.org using your Apache SVN credentials. Click on Staging on the left. Then click on orgapachebigtop in the list of repositories. In the panel below you should see an open repository that is linked to your username and IP. Select this repository and click Close. This will close the repository from future deployments and make it available for others to view.
BIGTOP-1463 is open to track the improvements of the release process.
7. Copy Release Artifacts
The artifacts that end up in the distribution directory are the source distributions (along with their checksums and signatures), so they need to be copied from the Maven repo to a release candidate directory on apache dist, so the vote can begin:
No Format |
---|
VERSION=X.Y.Z
REPOSITORY_ID=xxxx # Should be a number, for example 1013. Find the number in NEXUS staging repository: orgapachebigtop-1013
svn checkout https://dist.apache.org/repos/dist/dev/bigtop bigtop-dist-dev
cd bigtop-dist-dev
mkdir bigtop-$VERSION-RC1
cd bigtop-$VERSION-RC1
for ext in "" .asc .md5 .sha1; do
wget --no-check-certificate https://repository.apache.org/content/repositories/orgapachebigtop-$REPOSITORY_ID/org/apache/bigtop/bigtop/$VERSION/bigtop-$VERSION-project.tar.gz$ext
done
cd ..
svn add bigtop-$VERSION-RC1
svn ci -m "Apache Bigtop $VERSION-RC1" |
8. Sanity Check
TODO
...
9. Run the Vote
Run the vote on the dev@bigtop.apache.org
...
This step makes the artifacts available on the mirrors.
Code Block | |
---|---|
| |
| |
VERSION=X.Y.Z #Example: 1.3.0 CANDIDATE=C #Example: RC2 svn co https://dist.apache.org/repos/dist/release/bigtop cd bigtop svn mv https://dist.apache.org/repos/dist/dev/bigtop/bigtop-$VERSION-$CANDIDATE https://dist.apache.org/repos/dist/release/bigtop/bigtop-$VERSION rm stable ln -s bigtop-$VERSION stable svn commit |
The last line is to remove the previous version, since only the most recent version on a particular branch should be in the dist directory (older versions are archived automatically, see http://archive.apache.org/dist/bigtop/ and http://www.apache.org/dev/mirrors.html).
Log in to https://repository.apache.org, click on Staging on the left. Select the repository that you closed earlier, and click Release, using a description like "Apache Bigtop X.Y.Z artifacts". This will make the artifacts publicly available.
...
Create permanent release tag under rel/
Code Block | ||
---|---|---|
| ||
git checkout release-x.y.z git tag rel/x,.y,.z -u <signing key ID> git push --tags |
Deploy binary packages to permanent location
...
On Debian systems create the signature for the top-level Release fil
Code Block |
---|
gpg -u <you key ID> -ab o Release.gpg Release |
On RPM-bases distros create ~/.rpmmacors files (due to a bug, RPM can not use DSA keys longer than 1024 bytes, so you better use a buggy RSA for that)
Code Block |
---|
% cat > ~/.rpmmacros <<_EOF
%_signature gpg
%_gpg_name <you signing key ID>
_EOF
% rpmsign --resign [fedora,centos,opensuse,sles]/*/x86_64/**/**/*rpm
# recreate the repos
% for r in fedora centos opensuse sles; do pushd; cd $r; createrepo -v *; popd; done |
Wait 24 Hours
It takes up to 24 hours for all the mirrors to sync, so don't announce the new release just yet.
We had to wait 24 hours before, but it's not true now since ASF adopted CDN instead of mirror sites. According to the FAQs of the ASF infra:
Apache uses a global content distribution network (CDN) which collects new releases almost as soon as you post them.
The files therefore become available for download almost immediately. You probably don't need to wait more than fifteen minutes before announcing a release.
Build and Deploy Site
Update files under src/site by referring to past commits such as https://github.com/apache/bigtop/commit/678375eb75f7367f9114a3ad8087ba40f21444b6 or https://github.com/apache/bigtop/commit/d5c45a2879f8e3a8d29a36386ad8904a4e71a274. Commit into master and push them.
Login to the Jenkins server provided by ASF Infra. Open our website publishing pipeline and run it from the "Build Now" menu. It checks out the master branch with your update from our git repository, runs mvn site
, copies generated webpages into the asf-site branch, and commits them. It automatically updates our website thanks to .asf.yaml.
(If the above process doesn't seem to work correctly just like
Jira | ||||||||
---|---|---|---|---|---|---|---|---|
|
As said in
Jira | ||||||
---|---|---|---|---|---|---|
|
Code Block | ||
---|---|---|
| ||
VERSION=X.Y.Z #Example: 1.2.1
svn co https://dist.apache.org/repos/dist/release/bigtop
cd bigtop
svn delete bigtop-$VERSION
svn commit |
11. Announce the Release
TODO: Add a news section to the website.
Send an email to announce@apache.org (the from: address must be @apache.org). E.g.
No Format |
---|
To: announce@apache.org, user@bigtop.apache.org, dev@bigtop.apache.org
Subject: [ANNOUNCE] Apache Bigtop X.Y.Z released
The release is available here:
https://bigtop.apache.org/download.html#releases
A few highlights of this release include:
<LIST-OF-ITEMS-TO-BE-HIGHLIGHTED>
With Bigtop X.Y.Z the community continues to deliver the most advanced big data stack to date. More details about X.Y.Z release are here:
https://bigtop.apache.org/release-notes.html
Deploying Bigtop is easy: grab the repo/list file for your favorite Linux distribution:
https://www.apache.org/dyn/closer.lua/bigtop/bigtop-X.Y.Z/repos/
and you'll be running your very own bigdata cluster in no time!
We welcome your help and feedback. For more information on how to report problems, and to get involved, visit the project website at:
https://bigtop.apache.org
|
Please be noted that for download page it should be updated with following compliance:
- It should contain links to all current and archived releases along with links to KEYS, checksums, and signatures for all releases.
- The links on the page need to be links to release artifacts, not links to directories.
- The links to the asc and sha files should refer to archives only for non-current releases. The current release links should be to apache.org/dist, not archive.apache.org/dist.
12. Add a release page on Bigtop Wiki
- Copy a new page from previous release page: Ex: Bigtop 1.4.0 Release
- Edit to update the content. Basically just copy & paste from existing materials.
- Update links:
13
Sign *.repo and *list files
Code Block |
---|
% gpg --sign -u <you key ID> -a [fedora,redhat,opensuse,sles]/*/x86_64/**/bigtop.[repo,list] |
...
Sync signed packages to repos.bigtop.apache.org (ask a previous release manager about the details and credentials). Be aware: just sync apt/ directory for *.deb distributions, otherwise you'll copy two versions of the packages.
...
Commit the repo files to the svn next to the X.Y.Z release sources (refer to the last release for the structure of the repos
directory)
...
Add your signed armored GPG key to repos
directory to ease the key import for the users
Wait 24 Hours
It takes up to 24 hours for all the mirrors to sync, so don't announce the new release just yet.
Build and Deploy Site
Full version of this procedure is described and maintained in Deploying Bigtop's Apache Website
No Format |
---|
mvn site-deploy
|
This will deploy the generated website to /www/bigtop.apache.org/ on people.apache.org. You can SSH there to check that things look OK.Once the deployment is done you need to publish new website via https://cms.apache.org/bigtop/: login and click "Publish bigtop site" link. Otherwise the content won't get propagated.
It will take an hour or so for the website to become live, but you can inspect the website by temporarily setting your browser proxy to see the new, unmirrored content (I used FoxyProxy in Firefox) as described in http://www.apache.org/dev/project-site.html.
11. Announce the Release
TODO: Add a news section to the website.
Send an email to announce@apache.org (the from: address must be @apache.org). E.g.
No Format |
---|
To: announce@apache.org, user@bigtop.apache.org, dev@bigtop.apache.org
Subject: [ANNOUNCE] Apache Bigtop X.Y.Z released
The Apache Bigtop team is pleased to announce the release of Bigtop X.Y.Z.
The release is available here:
http://www.apache.org/dyn/closer.cgi/bigtop/
The full change log is available here:
https://issues.apache.org/jira/browse/BIGTOP/fixforversion/12315111
We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
http://bigtop.apache.org/
The Apache Bigtop Team
|
...
. Add the Next Release to JIRA
Add the next version number (e.g. 0.2.0 after 0.1.0) to JIRA using this
link: https://issues.apache.org/jira/plugins/secureservlet/project/ManageVersions.jspa?pid=12311420.-config/BIGTOP/administer-versions?status=unreleased
In JIRA mark the released version as "released" on the "manage versions" page. Be sure to fill in a date if not already specified.
...