This document is for you to quickly setup CI jobs to ensure the quality of your own customized Bigtop distribution.
Notice that, the document assume you already know well how to use Jenkins, hence the instructions are brief.
If you have questions, feel free to ask on Bigtop mailing list(user@bigtop.apache.org).
Setup a Jenkins master. /etc/profile.d/bigtop.sh;
If you already have a managed Jenkins/Hudson, or you prefer your all installation, skip and go to Setup Bigtop packages build matrix
To setup a Jenkins master by leveraging Docker, do the following:
# create jenkins user on host machine with uid=1000 to map the jenkins uid inside jenkins image sudo adduser jenkins -u 1000 sudo yum install -y docker git sudo su - jenkins -c "git config --global user.email \"jenkins@bigtop.apache.org\"" sudo su - jenkins -c "git config --global user.name \"jenkins\"" sudo usermod -a -G docker jenkins sudo service docker start sudo su - jenkins -c "docker run -d --name jenkins-master -p 80:8080 -v `pwd`:/var/jenkins_home jenkins"
And the needed Plugin(s):
- git plugin
Setup Bigtop packages build matrix
Create a new job:
- New Item -> Multi-configuration project
In Source Code Management section, fill in your repo and the branch, for example:
Repository URL: https://git-wip-us.apache.org/repos/asf/bigtop.git Branch Specifier: master
In the Configuration Matrix section, add a user defined axis with following name and values:
Name: BUILD_ENVIRONMENTS Values: centos-6 centos-7 fedora-25 ubuntu-16.04 debian-8 opensuse-42.1
Add another user defined axis:
Name: COMPONENTS Values: bigtop-groovy bigtop-jsvc bigtop-tomcat bigtop-utils crunch datafu flume giraph hadoop hama hbase hive hue ignite-hadoop kafka kite mahout oozie phoenix pig solr spark sqoop sqoop2 tachyon tez ycsb zookeeper
Finally, add a shell build step with the following script:
docker run --rm -v `pwd`:/ws --workdir /ws bigtop/slaves:trunk-$BUILD_ENVIRONMENTS \ bash -c '. /etc/profile.d/bigtop.sh; ./gradlew allclean $COMPONENTS-pkg'
The result will be looked like this:
http://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/
However, do aware that full matrix build of packages is time consuming, and they need roughly 50GB disk space for each build.
Setup Bigtop deployments build matrix
To setup a deployments build matrix, prepare a job like above in packages build matrix and add these defined axis:
Name: OS Values: centos6 centos7 debian8 ubuntu_xenial
See the sample configs in bigtop/provisioner/docker.
Name: COMPONENTS Values: alluxio apex crunch flink flume giraph ignite_hadoop hbase hcat hive httpfs hue mahout mapred-app oozie pig qfs solrcloud spark sqoop sqoop2 tez yarn zookeeper ycsb gpdb ambari
See the available components for deployment in bigtop-deploy/puppet/hieradata/site.yaml.
Use the following shell script for builds:
# destroy previous cluster ./gradlew docker-provisioner-destroy # setup configuration file CONFIG="config_${OS}_Bigtop-trunk-deployments.yaml" sed "s/components.*/components: [hdfs, yarn, mapreduce, ${COMPONENTS}]/g" \ provisioner/docker/config_${OS}.yaml > provisioner/docker/${CONFIG} # provision # Since the puppet deployment will always return 0, # we need to save the log and grep error to determine status ./gradlew -Pconfig=${CONFIG} -Pnum_instances=1 docker-provisioner > tmp.log 2>&1 cat tmp.log # destroy provisioned cluster ./gradlew docker-provisioner-destroy # Fail the build if Errors occur in tmp.log grep Error tmp.log && exit 1 exit 0
The result will be looked like this:
https://ci.bigtop.apache.org/view/Deployment/job/Bigtop-trunk-deployments
You can replace the repo by your own repo by adding sed commend so that your own packages can be tested using the same deployment recipes.
The components is also configurable, choose what you'd like to deploy and test as what you want.
Advanced part: Setup a SSL secured Jenkins master
First you need a SSL certificate.
Generating a SSL cert
Here we will use a certificate created by the http://letsencrypt.org service.
Most convient way to run it is as a docker container, building a container:
# git clone https://github.com/letsencrypt/letsencrypt # cd letsencrypt # docker build -t letsencrypt .
Now use this container to sign into letsencrypt. You have to make sure that no other service is running on port 80 and 443. This starts an intermediate server an handles a challenge response handshake in order to prove that we actually are running http://ci.bigtop.apache.org
# docker run --rm -i -t -p 80:80 -p 443:443 -v "/etc/letsencrypt:/etc/letsencrypt" letsencrypt certonly --standalone --email dev@bigtop.apache.org -d ci.bigtop.apache.org
Please note that this writes into /etc/letsencrypt as root on the host.
Generating first cert
The command does not generate a certificate on first run. It generates new certificate (renewals) each new run. Please be sure that any httpd server is stopped
# docker run --rm -i -t -p 80:80 -p 443:443 -v "/etc/letsencrypt:/etc/letsencrypt" letsencrypt certonly --standalone --email dev@bigtop.apache.org -d ci.bigtop.apache.org
Please note it created /etc/letsencrypt/live/ci.bigtop.apache.org containing cert and key.
Enabling of SSL
Since the lifetime of the letsencrypt certs is quite short on needs a flexible infrastructure to use the certs as is.
The basic design here is to use a reverse proxy terminating the SSL at the proxy. I.e. All handling of https:// is done by the reverse proxy and requests are proxied to the web application and sent back to the client.
A quick setup on a RPM based system is:
# yum install httpd mod_ssl
in /etc/http/conf.d/ssl.conf add this block after the line
<VirtualHost _default_:443>
(see jenkins documentation about deeper insights into this glibberish)
<Proxy *> Order deny,allow Allow from all </Proxy> ProxyRequests Off # Because of JENKINS-22539 ProxyPreserveHost On Header edit Location ^http://ci.bigtop.apache.org/ https://ci.bigtop.apache.org/ ProxyPass / http://localhost:8080/ nocanon ProxyPassReverse / http://localhost:8080/ ProxyRequests Off AllowEncodedSlashes NoDecode <Proxy http://localhost:8080/*> Order deny,allow Allow from all </Proxy>
This enables the reverse proxy mode of port 433 to port 8080 and setting Jenkins specific parameters.
And the location of the certificates have to be aligned with letsencrypt in /etc/http/conf.d/ssl.conf
# Server Certificate: # Point SSLCertificateFile at a PEM encoded certificate. If # the certificate is encrypted, then you will be prompted for a # pass phrase. Note that a kill -HUP will prompt again. A new # certificate can be generated using the genkey(1) command. SSLCertificateFile /etc/letsencrypt/live/ci.bigtop.apache.org/cert.pem # Server Private Key: # If the key is not combined with the certificate, use this # directive to point at the key file. Keep in mind that if # you've both a RSA and a DSA private key you can configure # both in parallel (to also allow the use of DSA ciphers, etc.) SSLCertificateKeyFile /etc/letsencrypt/live/ci.bigtop.apache.org/privkey.pem # Server Certificate Chain: # Point SSLCertificateChainFile at a file containing the # concatenation of PEM encoded CA certificates which form the # certificate chain for the server certificate. Alternatively # the referenced file can be the same as SSLCertificateFile # when the CA certificates are directly appended to the server # certificate for convinience. SSLCertificateChainFile /etc/letsencrypt/live/ci.bigtop.apache.org/chain.pem
The last thing is to change jenkins to port 8080 and start apache httpd
# docker run -d --name jenkins-master-8080 -p 8080:8080 -v /home/jenkins:/var/jenkins_home jenkins # systemctl start httpd
In order to redirect the browser from http://ci.bigtop.apache.org to https://ci.bigtop.apache.org place a file into /var/www/html/index.html
<META HTTP-EQUIV="refresh" CONTENT="1; URL=https://ci.bigtop.apache.org">
Renewing the cert
The command does not generate a certificate on first run. It generates new certificate (renewals) each new run. Please be sure that any httpd server is stopped
# /etc/init.d/httpd stop # docker run --rm -i -t -p 80 -p 443:443 -v "/etc/letsencrypt:/etc/letsencrypt" quay.io/letsencrypt/letsencrypt:latest renew # (obsolete) docker run --rm -i -t -p 80 -p 443:443 -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" quay.io/letsencrypt/letsencrypt:latest certonly --standalone --email dev@bigtop.apache.org -d ci.bigtop.apache.org # /etc/init.d/httpd start