Page History

...

• Update CHANGES.txt in master to replace  Release X.Y.0 (unreleased) with Release X.Y.0 - YYYY-MM-DD. Commit:

  noformat

  Code Block
  language bash
  git add CHANGES.txt git commit -m "Preparing for release X.Y.0"

  
  

• Create a branch for the release series:

  noformat

  Code Block
  language
  bash
  git checkout -b branch-X.Y origin/master

  
  

• Switch back to master and add A.B.C-SNAPSHOT (unreleased changes) to CHANGES.txt.

  noformat

  Code Block
  language
  bash
  git checkout - (edit CHANGES.txt)

  
  

• Bump the version number in the master branch:

  noformat

  Code Block
  language
  bash
  ./gradlew setversion -Pnextversion="A.B.C-SNAPSHOT"

  
  

• Commit these changes to the master and push.

  noformat

  Code Block
  language
  bash
  git add CHANGES.txt (and other updated files) git commit -m "Bumping up the version to A.B.C-SNAPSHOT" git push upstream master (assuming that ASF repo is referred to as 'upstream')

  
  

• Checkout the release branch

  noformat

  Code Block
  language
  bash
  git checkout branch-X.Y

  
  

• Update the release branch's version information: the version number in the release branch ends in -SNAPSHOT, but we need to remove this for the release. For example, 0.8.0-SNAPSHOT needs to be changed to 0.8.0.

  noformat

  Code Block
  language
  bash
  ./gradlew setversion -Pnextversion="X.Y.Z"

  
  

• Rename docker images and repo URLs:

  noformat

  Code Block
  language
  bash
  grep -R --color :trunk- * | grep yaml grep -R --color "http://repos.bigtop.apache.org/releases/X.Y.Z" * grep -R --color "http://repos.bigtop.apache.org/releases/x.y.z" * (where "x.y.z" is the previous version of "X.Y.Z". Just in case) (replace the occurrences found by the above commands)

  
  

• Commit these changes to the release branch and push

  noformat

  Code Block
  language
  bash
  git add (updated files) git commit -m "Changing version to X.Y.Z" git push upstream branch-X.Y (assuming that ASF repo is referred to as 'upstream')

  
  

• Also, update the default version of Bigtop (and your code signing key ID, if needed) defined in hieradata on master and the release branch. For example:

  noformat

  Code Block
  language
  bash
  $ git checkout master (edit bigtop-deploy/puppet/hieradata/bigtop/repo.yaml) $ git diff diff --git a/bigtop-deploy/puppet/hieradata/bigtop/repo.yaml b/bigtop-deploy/puppet/hieradata/bigtop/repo.yaml index 48c8a4ae..0d6a9126 100644 --- a/bigtop-deploy/puppet/hieradata/bigtop/repo.yaml +++ b/bigtop-deploy/puppet/hieradata/bigtop/repo.yaml @@ -1,5 +1,5 @@ bigtop::bigtop_repo_gpg_check: true -bigtop::bigtop_repo_apt_key: "BB95B97B18226C73CB2838D1DBBF9D42B7B4BD70" +bigtop::bigtop_repo_apt_key: "01621A73025BDCA30F4159C62922A48261524827" bigtop::bigtop_repo_yum_key_url: "https://downloads.apache.org/bigtop/KEYS" -bigtop::bigtop_repo_default_version: "1.4.0" +bigtop::bigtop_repo_default_version: "1.5.0" $ git add bigtop-deploy/puppet/hieradata/bigtop/repo.yaml $ git commit -m "Update default values in hieradata for release 1.5.0" [master 9715ca7a] Update default values in hieradata for release 1.5.0 1 file changed, 2 insertions(+), 2 deletions(-) $ git push upstream master $ git checkout branch-1.5 $ git cherry-pick -x 9715ca7a $ git push upstream branch-1.5

  
  

• Shall you need to commit additional fixes into ongoing release, the commits should go to the release branch and only then be merged into master. Doing this other way around forces git cherry-pick which leads to discrepancies in the commit hash-codes and makes the branch look untidy and hard to follow.

...

Tag, and push the changes and the tag to git:

git tag release-x.y.z -m "Bigtop X.Y.Z release."
git push --tags 

...

Download all built packages via archive download feature provided by Jenkins on a machine that you want to proceed the signing:

PLATFORM=amd64
#PLATFORM=aarch64  # forenable allwhen OSrunning andon archthe supportedARM in Bigtop
for i in centos-7 fedora-26 opensuse-42.3 debian-9 ubuntu-16.04; do \
  for j in amd64-slave aarch64-slave ppc64le-slave; do \
    wget platform
#PLATFORM=ppc64le  # enable when running on the PowerPC platform

# Download artifacts for all OS supported in Bigtop.
# Note that you should run this on each platform (x86_64, ARM, and PowerPC), because
# you will sign the artifacts and create repository in Docker container later,
# which are dependent on the platform.
for DISTRO in centos-7 fedora-26 opensuse-42.3 debian-9 ubuntu-16.04; do
  curl -L https://ci.bigtop.apache.org/view/Releases/job/Bigtop-1.4.0/DISTRO=${iDISTRO},PLATFORM=${jPLATFORM}-slave/lastSuccessfulBuild/artifact/*zip*/archive.zip
 -o ${DISTRO}.zip
done
done

5.3.  Sign RPM packages and yum repos

Ref: 

Startup a docker images that is RPM based system:

cd ~/
# Change the image for AARCH64 or PPC64LE
docker run -ti --rm -v $PWD:/tmp bigtop/puppet:1.4.0-centos-7 bash

Prepare the environment for signing:

gpg --import YOUR_CODE_SIGNING_SECRET_KEY
echo "%_gpg_name YOUR_CODE_SIGNING_KEY_ID" > ~/.rpmmacros
yum install -y rpm-sign createrepo

Signing:

cd /tmp
OS=centos-7
 
# Sign all RPM packages (This step required to input passphrase, so don't copy and paste the entire script here)
rpm --addsign `find ${OS} -name \*rpm`
 
# Recreate the metadata for repository
createrepo ${OS}
 
# Armor the metadata
gpg --detach-sign --armor ${OS}/repodata/repomd.xml

[OPENSUSE ONLY] 

# Extract the downloaded archive. If you come across the 
# "invalid zip file with overlapped components (possible zip bomb)"
# error, use other tools than unzip, e.g., `jar xf "${i}.zip"`
for i in centos-7 fedora-26 opensuse-42.3 debian-9 ubuntu-16.04; do
  unzip "${i}.zip"
  mkdir "${i}"
  mv archive/output/* "${i}"
  rm -rf archive
done

5.3.  Sign RPM packages and yum repos

Ref: 

Startup a docker images that is RPM based system:

cd ~/
# Change the image for AARCH64 or PPC64LE
docker run -ti --rm -v $PWD:/tmp bigtop/puppet:1.4.0-centos-7 bash

Prepare the environment for signing:

yum install -y rpm-sign createrepo pinentry
gpg --import YOUR_CODE_SIGNING_SECRET_KEY
echo "%_gpg_name YOUR_CODE_SIGNING_KEY_ID" > ~/.rpmmacros

Signing:

cd /tmp
OS=centos-7
 
# Sign all RPM packages (This step required to input passphrase, so don't copy and paste the entire script here)
rpm --addsign `find ${OS} -name \*rpm`
 
# Recreate the metadata for repository
createrepo ${OS}
 
# Armor the metadata
gpg 

gpg --armor --export evansye@apache.org > opensuse-42.3/repodata/repomd.xml.key
for i in `find opensuse-42.3/repodata -name *.xml.gz` opensuse-42.3/repodata/repomd.xml.key ; do gpg --detach-sign --armor $i ; done

5.4.  Sign DEB packages and apt repos

Ref: 

Startup a docker images that is DEB based system:

cd ~/
docker run -ti --rm -v $PWD:/tmp bigtop/puppet:1.4.0-debian-9 bash

Prepare the environment for signing:

${OS}/repodata/repomd.xml

[OPENSUSE ONLY] 

gpg --armor --export evansye@apache.org > opensuse-42.3/repodata/repomd.xml.key
for i in `find opensuse-42.3/repodata -name *.xml.gz` opensuse-42.3/repodata/repomd.xml.key ; do gpg --detach-sign --armor $i ; done

5.4.  Sign DEB packages and apt repos

Ref: 

Startup a docker images that is DEB based system:

cd ~/
# Change the image for AARCH64 or PPC64LE
docker run -ti --rm -v $PWD:/tmp bigtop/puppet:1.4.0-debian-9 bash

Prepare the environment for signing:

apt-get update

apt-get update
apt-get install -y gpg
apt-get install -y libterm-readkey-perl
apt-get install -y dpkg-sig
apt-get install -y gpg libterm-readkey-perl dpkg-sig reprepro
gpg --import YOUR_CODE_SIGNING_SECRET_KEY

Signing:

cd /tmp
VERSION=1.4.0
OS=debian-9
ARCH=amd64
#ARCH=arm64
#ARCH=ppc64el
SIGN_KEY=B7B4BD70
export GPG_TTY=$(tty)
 
# Sign DEB packages (This step required to input passphrase, so don't copy and paste the entire script here)
dpkg-sig --cache-passphrase --sign builder --sign-changes force_full `find ${OS}/ -name \*deb`
 
# Build signed apt repository
mkdir -p conf
cat > conf/distributions <<__EOT__
Origin: Bigtop
Label: Bigtop
Suite: stable
Codename: bigtop
Version: ${VERSION}
Architectures: ${ARCH} source
Components: contrib
Description: Apache Bigtop
SignWith: ${SIGN_KEY}
__EOT__
 
cat > conf/options <<__EOT__
verbose
ask-passphrase
__EOT__

rm -rf ${OS}/apt
reprepro --ask-passphrase -Vb . includedeb bigtop `find # Note that this command creates the apt repository (db, dists, pool)
# in the current directory, so you can't ran it for multiple distros in parallel.
reprepro --ask-passphrase -Vb . includedeb bigtop `find ${OS}/ -name \*deb`
rm -rf ./${OS}/*
mv conf db dists pool ${OS}

5.5. Upload to S3

The easiest way to upload artifacts to S3 is via your own AWS account. Add your account(email) to bigtop's bucket in the section "Access for other AWS accounts":

...

Once permission granted, you're able to use your account's access key and secret key with aws s3 sync command for upload, like:

# In case of CentOS 7
aws s3 sync --acl public-read ./centos-7/ s3://repos.bigtop.apache.org/releases/1.4.0/centos/7/x86_64/   # for x86_64/amd64
aws s3 sync --acl public-read ./centos-7/ s3://repos.bigtop.apache.org/releases/1.4.0/centos/7/aarch64/  # for aarch64/arm64
aws s3 sync --acl public-read ./centos-7/ s3://repos.bigtop.apache.org/releases/1.4.0/centos/7/ppc64le/  # for ppc64le/ppc64el

# In case of Ubuntu 16.04
aws s3 sync --acl public-read ./ubuntu-16.04/ s3://repos.bigtop.apache.org/releases/1.4.0/ubuntu/16.04/amd64/    # for x86_64/amd64
aws s3 sync --acl public-read ./ubuntu-16.04/ s3://repos.bigtop.apache.org/releases/1.4.0/ubuntu/16.04/arm64/    # for aarch64/arm64

aws s3 sync --acl public-read ./ubuntu-16.04/ s3://repos.bigtop.apache.org/releases/1.4.0/ubuntu/16.04/amd64//ppc64el/  # for ppc64le/ppc64el

The directory layouts on S3 bucket looked like the following:

repos.bigtop.apache.org/releases/1.4.0/centos/7/x86_64
repos.bigtop.apache.org/releases/1.4.0/fedora/26/x86_64
repos.bigtop.apache.org/releases/1.4.0/opensuse/42.3/x86_64
repos.bigtop.apache.org/releases/1.4.0/debian/9/amd64
repos.bigtop.apache.org/releases/1.4.0/ubuntu/16.04/amd64

For YUM repos, upload files /tmp/centos-7/* into repos.bigtop.apache.org/releases/1.4.0/centos/7/x86_64/. For example:

repos.bigtop.apache.org/releases/1.4.0/centos/7/x86_64/alluxio
repos.bigtop.apache.org/releases/1.4.0/centos/7/x86_64/ambari
...
repos.bigtop.apache.org/releases/1.4.0/centos/7/x86_64/repodata
...

For APT repos, upload files /tmp/debian-9/* into repos.bigtop.apache.org/releases/1.4.0/debian/9/amd64/. For example:

repos.bigtop.apache.org/releases/1.4.0/debian/9/amd64/conf
repos.bigtop.apache.org/releases/1.4.0/debian/9/amd64/db
repos.bigtop.apache.org/releases/1.4.0/debian/9/amd64/dists
repos.bigtop.apache.org/releases/1.4.0/debian/9/amd64/pool

...

Create one for each of our Distro. Following are examples for YUM and APT:

VERSION=1.4.0
RC_NUMBER=0
DIR=bigtop-${VERSION}-RC${RC_NUMBER}/repos
mkdir -p ${DIR}
cd ${DIR}

# YUM
for OS in centos-7 fedora-26 opensuse-42.3; do
  

# YUM
cat > bigtop.repo <<__EOT__
[bigtop]
name=Bigtop
enabled=1
gpgcheck=1
baseurl=http://repos.bigtop.apache.org/releases/1.4.0/centos/7/x86_64/${VERSION}/${OS/-/\/}/\$basearch
gpgkey=https://dist.apache.org/repos/dist/release/bigtop/KEYS
__EOT__
 
gpg --detach-sign --armor bigtop.repo
 
# APT

  gpg --detach-sign --armor bigtop.repo
  mkdir "${OS//\/}"
  mv bigtop.repo* "${OS//\/}"
done

# APT
for OS in debian-9 ubuntu-16.04; do
  cat > bigtop.list <<__EOT__
deb http://repos.bigtop.apache.org/releases/1.4.0/debian/9/amd64	/${VERSION}/${OS/-/\/}/\$(ARCH) bigtop contrib
__EOT__
 
  gpg --detach-sign --armor bigtop.list
  mkdir "${OS//\/}"
  mv bigtop.list* "${OS//\/}"
done

Add your signed armored GPG key to repos directory to ease the key import for the users

...