THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
https://ci.bigtop.apache.org/view/Releases/job/Bigtop-1.4.0/
5.3. Sign RPM packages and yum repos
Ref:
| Jira | ||||||
|---|---|---|---|---|---|---|
|
Clone RPMs built by CI job into a temp dir. You can find built packges under jenkins job's workspace. For example:
| No Format |
|---|
# Go to the build slave, docker-slave-06 or docker-slave-07, depends on which built the packages
VERSION=1.4.0
SLAVE=amd64-slave # or aarch64-slave, or ...
for i in centos-7 fedora-26 opensuse-42.3 debian-9 ubuntu-16.04; do cp -r /home/jenkins/workspace/Bigtop-${VERSION}/DISTRO/$i/PLATFORM/${SLAVE}/output ~/$i; done |
Startup a docker images that is RPM based system:
| No Format |
|---|
cd ~/
# Change the image for AARCH64 or PPC64LE
docker run -ti --rm -v $PWD:/tmp bigtop/puppet:1.4.0-centos-7 bash |
Prepare the environment for signing:
| No Format |
|---|
gpg --import YOUR_CODE_SIGNING_SECRET_KEY
echo "%_gpg_name YOUR_CODE_SIGNING_KEY_ID" > ~/.rpmmacros
yum install -y rpm-sign createrepo |
Signing:
| No Format |
|---|
cd /tmp
OS=centos-7
# Sign all RPM packages (This step required to input passphrase, so don't copy and paste the entire script here)
rpm --addsign `find ${OS} -name \*rpm`
# Recreate the metadata for repository
createrepo ${OS}
# Armor the metadata
gpg --detach-sign --armor ${OS}/repodata/repomd.xml |
[OPENSUSE ONLY]
Download all built packages via archive download feature provided by Jenkins on a machine that you want to proceed the signing:
| Code Block | ||
|---|---|---|
| ||
# for all OS and arch supported in Bigtop
wget https://ci.bigtop.apache.org/view/Releases/job/Bigtop-1.4.0/DISTRO=centos-7,PLATFORM=amd64-slave/lastSuccessfulBuild/artifact/*zip*/archive.zip |
5.3. Sign RPM packages and yum repos
Ref:
| Jira | ||||||
|---|---|---|---|---|---|---|
|
Clone RPMs built by CI job into a temp dir. You can find built packges under jenkins job's workspace. For example:
| No Format |
|---|
# Go to the build slave, docker-slave-06 or docker-slave-07, depends on which built the packages
VERSION=1.4.0
SLAVE=amd64-slave # or aarch64-slave, or ...
for i in centos-7 fedora-26 opensuse-42.3 debian-9 ubuntu-16.04; do cp -r /home/jenkins/workspace/Bigtop-${VERSION}/DISTRO/$i/PLATFORM/${SLAVE}/output ~/$i; done |
Startup a docker images that is RPM based system:
| No Format |
|---|
cd ~/
# Change the image for AARCH64 or PPC64LE
docker run -ti --rm -v $PWD:/tmp bigtop/puppet:1.4.0-centos-7 bash |
Prepare the environment for signing:
| No Format |
|---|
gpg --import YOUR_CODE_SIGNING_SECRET_KEY
echo "%_gpg_name YOUR_CODE_SIGNING_KEY_ID" > ~/.rpmmacros
yum install -y rpm-sign createrepo |
Signing:
| No Format |
|---|
cd /tmp
OS=centos-7
# Sign all RPM packages (This step required to input passphrase, so don't copy and paste the entire script here)
rpm --addsign `find ${OS} -name \*rpm`
# Recreate the metadata for repository
createrepo ${OS}
# Armor the metadata
|
| No Format |
gpg --armor --export evansye@apache.org > opensuse-42.3/repodata/repomd.xml.key for i in `find opensuse-42.3/repodata -name *.xml.gz` opensuse-42.3/repodata/repomd.xml.key ; do gpg --detach-sign --armor $i ; done |
5.4. Sign DEB packages and apt repos
Ref:
| Jira | ||||||
|---|---|---|---|---|---|---|
|
Clone DEBs built by CI job into a temp dir. You can find built packges under jenkins job's workspace. For example:
| No Format |
|---|
# Go to the build slave, docker-slave-06 or docker-slave-07, depends on which built the packages
VERSION=1.4.0
OS=debian-9
SLAVE=amd64-slave # or aarch64-slave, or ...
for i in centos-7 fedora-26 opensuse-42.3 debian-9 ubuntu-16.04; do cp -r /home/jenkins/workspace/Bigtop-${VERSION}/DISTRO/$i/PLATFORM/${SLAVE}/output ~/$i; done |
Startup a docker images that is DEB based system:
| No Format |
|---|
cd ~/
docker run -ti --rm -v $PWD:/tmp bigtop/puppet:1.4.0-debian-9 bash |
Prepare the environment for signing:
| No Format |
|---|
apt-get update
apt-get install -y gpg
apt-get install -y libterm-readkey-perl
apt-get install -y dpkg-sig
apt-get install -y reprepro
gpg --import YOUR_CODE_SIGNING_SECRET_KEY |
Signing:
${OS}/repodata/repomd.xml |
[OPENSUSE ONLY]
| No Format |
|---|
gpg --armor --export evansye@apache.org > opensuse-42.3/repodata/repomd.xml.key
for i in `find opensuse-42.3/repodata -name *.xml.gz` opensuse-42.3/repodata/repomd.xml.key ; do gpg --detach-sign --armor $i ; done |
5.4. Sign DEB packages and apt repos
Ref:
| Jira | ||||||
|---|---|---|---|---|---|---|
|
Clone DEBs built by CI job into a temp dir. You can find built packges under jenkins job's workspace. For example:
| No Format |
|---|
# Go to the build slave, docker-slave-06 or docker-slave-07, depends on which built the packages
VERSION=1.4.0
OS=debian-9
SLAVE=amd64-slave # or aarch64-slave, or ...
for i in centos-7 fedora-26 opensuse-42.3 debian-9 ubuntu-16.04; do cp -r /home/jenkins/workspace/Bigtop-${VERSION}/DISTRO/$i/PLATFORM/${SLAVE}/output ~/$i; done |
Startup a docker images that is DEB based system:
| No Format |
|---|
cd ~/
docker run -ti --rm -v $PWD:/tmp bigtop/puppet:1.4.0-debian-9 bash |
Prepare the environment for signing:
| No Format |
|---|
apt-get update
apt-get install -y gpg
apt-get install -y libterm-readkey-perl
apt-get install -y dpkg-sig
apt-get install -y reprepro
gpg --import YOUR_CODE_SIGNING_SECRET_KEY |
Signing:
| No Format |
|---|
cd /tmp
VERSION=1.4.0
OS=debian-9
ARCH=amd64
SIGN_KEY=B7B4BD70
export GPG_TTY=$(tty)
# Sign DEB packages (This step required to input passphrase, so don't copy and paste the entire script here)
dpkg-sig --cache-passphrase --sign builder `find ${OS}/ -name \*deb`
# Build signed apt repository
mkdir -p conf
|
| No Format |
cd /tmp
VERSION=1.4.0
OS=debian-9
ARCH=amd64
SIGN_KEY=B7B4BD70
export GPG_TTY=$(tty)
# Sign DEB packages (This step required to input passphrase, so don't copy and paste the entire script here)
dpkg-sig --cache-passphrase --sign builder `find ${OS}/ -name \*deb`
# Build signed apt repository
mkdir -p conf
cat > conf/distributions <<__EOT__
Origin: Bigtop
Label: Bigtop
Suite: stable
Codename: bigtop
Version: ${VERSION}
Architectures: ${ARCH} source
Components: contrib
Description: Apache Bigtop
SignWith: ${SIGN_KEY}
__EOT__
cat > conf/options <<__EOT__
verbose
ask-passphrase
__EOT__
rm -rf ${OS}/apt
reprepro --ask-passphrase -Vb . includedeb bigtop `find ${OS}/ -name \*deb` |
5.5.
...
Upload to S3
The easiest way to upload artifacts to S3 is via your own AWS account. Add your account(email) to bigtop's bucket in the section "Access for other AWS accounts":
Once permission granted, you're able to use your account's access key and secret key with aws s3 sync command for upload:
| No Format |
|---|
aws s3 sync --acl public-read ./ubuntu-16.04/ s3 |
Create one for each of our Distro. Following are examples for YUM and APT:| No Format |
|---|
# YUM cat > bigtop.repo <<__EOT__ [bigtop] name=Bigtop enabled=1 gpgcheck=1 type=NONE baseurl=http://repos.bigtop.apache.org/releases/1.24.10/centos/7/x86_64 gpgkey=https://dist.apache.org/repos/dist/release/bigtop/KEYS __EOT__ gpg --detach-sign --armor bigtop.repo # APT cat > bigtop.list <<__EOT__ deb http://ubuntu/16.04/amd64/ |
The directory layouts on S3 bucket looked like the following:
| No Format |
|---|
repos.bigtop.apache.org/releases/1.2.1/centos/6/x86_64 repos.bigtop.apache.org/releases/1.2.1/debiancentos/87/x86_64 bigtop contrib __EOT__ gpg --detach-sign --armor bigtop.list |
5.6. Upload to S3
The easiest way to upload artifacts to S3 is via your own AWS account. Add your account(email) to bigtop's bucket in the section "Access for other AWS accounts":
Once permission granted, you're able to use your account's access key and secret key with aws s3 sync command for upload:
...
repos.bigtop.apache.org/releases/1.2.1/fedora/25/x86_64
repos.bigtop.apache.org/releases/1.2.1/opensuse/42.1/x86_64
repos.bigtop.apache.org/releases/1.2.1/debian/8/x86_64
repos.bigtop.apache.org/releases/1.2.1/ubuntu/16.04/x86_64 |
For YUM repos, upload files /tmp/centos-7/* into repos.bigtop.apache.org/releases/1.
...
...
...
...
The directory layouts on S3 bucket looked like the following7/x86_64/. For example:
| No Format |
|---|
repos.bigtop.apache.org/releases/1.2.1/centos/67/x86_64/alluxio repos.bigtop.apache.org/releases/1.2.1/centos/7/x86_64/ambari ... repos.bigtop.apache.org/releases/1.2.1/fedoracentos/257/x86_64/repodata ... |
For APT repos, upload files /tmp/debian-8/* into repos.bigtop.apache.org/releases/1.2.1/
...
...
8/x86_64/. For example:
| No Format |
|---|
repos.bigtop.apache.org/releases/1.2.1/debian/8/x86_64/conf repos.bigtop.apache.org/releases/1.2.1/ubuntudebian/16.048/x86_64 |
...
/db repos.bigtop.apache.org/releases/1.2.1/ |
...
debian/ |
...
8/x86_64/ |
...
| No Format |
|---|
dists repos.bigtop.apache.org/releases/1.2.1/centosdebian/78/x86_64/alluxio repos.bigtop.apache.org/releases/1.2.1/centos/7/x86_64/ambari ... repos.bigtop.apache.org/releases/1.2.1/centos/7/x86_64/repodata ... |
...
pool |
5.6. Create repo files
Create one for each of our Distro. Following are examples for YUM and APT:| No Format |
|---|
# YUM cat > bigtop.repo <<__EOT__ [bigtop] name=Bigtop enabled=1 gpgcheck=1 type=NONE baseurl=http://repos.bigtop.apache.org/releases/1.2.1/ |
...
centos/ |
...
7/x86_64 gpgkey=https:/ |
...
| No Format |
|---|
repos.bigtop./dist.apache.org/releases/1.2.1/debian/8/x86_64/conf repos/dist/release/bigtop/KEYS __EOT__ gpg --detach-sign --armor bigtop.repo # APT cat > bigtop.list <<__EOT__ deb http://repos.bigtop.apache.org/releases/1.2.1/debian/8/x86_64/db repos.bigtop.apache.org/releases/1.2.1/debian/8/x86_64/dists repos.bigtop.apache.org/releases/1.2.1/debian/8/x86_64/pool bigtop contrib __EOT__ gpg --detach-sign --armor bigtop.list |
5.7. Commit repo files into https://dist.apache.org/repos/dist/dev/bigtop/repos
...