...

Setup Bigtop deployments build matrix

To setup a deployments build matrix, prepare a job like above in packages build matrix and add these defined axis:

Name: OS
 
Values: 
centos6
centos7
debian8
ubuntu_xenial

See the sample configs in bigtop/provisioner/docker.

Name: COMPONENTS
 
Values: 
alluxio
apex
crunch
flink
flume
giraph
ignite_hadoop
hbase
hcat
hive
httpfs
hue
mahout
mapred-app
oozie
pig
qfs
solrcloud
spark
sqoop
sqoop2
tez
yarn
zookeeper
ycsb
gpdb
ambari

See the available components for deployment in bigtop-deploy/puppet/hieradata/site.yaml.

Use the following shell script for builds deployment for centos 6 OS, use following shell script:

# destroy previous cluster
./gradlew docker-provisioner-destroy
 
# setup configuration file
CONFIG="config_${OS}_Bigtop-trunk-deployments.yaml"
sed "s/components.*/components: [hdfs, yarn, mapreduce, ${COMPONENTS}]/g" \
  provisioner/docker/config_${OS}.yaml > provisioner/docker/${CONFIG}


# provision
# Since the puppet deployment will always return 0, 
# we need to save the log and grep error to determine status
./gradlew -Pconfig=${CONFIG} -Pnum_instances=1 docker-provisioner > tmp.log 2>&1
cat tmp.log


# destroy provisioned cluster
./gradlew docker-provisioner-destroy


# Fail the build if Errors occur in tmp.log
grep Error tmp.log && exit 1
exit 0

...

You can replace the repo by your own repo by adding sed commend so that your own packages can be tested using Bigtop the same deployment recipes.

The components is also configurable, choose what you'd like to deploy and test as what you want.

Advanced part: Setup a SSL secured

...

Jenkins master 

First you need a SSL certificate.

...