THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
The components is also configurable, choose what you'd like to deploy and test as what you want.
Setup Bigtop smoke tests build matrix
Create a new job:
- New Item -> Multi-configuration project
In Source Code Management section, fill in your repo and the branch, for example:
| No Format |
|---|
Repository URL: https://gitbox.apache.org/repos/asf/bigtop.git
Branch Specifier: master |
In the Configuration Matrix section, add axises with following name and values:
User-defined axis:
| No Format |
|---|
Name: COMPONENTS
Values:
hdfs.alluxio@alluxio
ambari.bigtop-utils@ambari
hdfs.yarn.apex@apex
flume@flume
hdfs.hbase@hbase
hdfs.yarn@hdfs.hcfs
hdfs.yarn.hive@hive
hdfs.yarn.ignite_hadoop@ignite-hadoop
hdfs.yarn.mahout@mahout
hdfs.yarn@mapreduce
hdfs.yarn.qfs@qfs
hdfs.zookeeper.solrcloud@solr
hdfs.yarn.spark@spark
sqoop@sqoop
hdfs.yarn@yarn
zookeeper@zookeeper |
For each value, the format is defined as DEPLOY_COMPONENTS@TEST_COMPONENETS, where DEPLOY_COMPONENTS cam be period separated string.
For example,
hdfs.alluxio@alluxioComponent hdfs and alluxio will be deployed, and alluxio will be tested.
Note: Why use period instead of comma as separator? Because Jenkins does not allow comma to be used in user defined axis.
Slaves aixs:
You got check boxes to check. To add options, go to Manage Jenkins → Manage Nodes and add labels for your slaves. The labels will be available here.
The following labels are bigtop's configuration for your reference.
| No Format |
|---|
Name: OS
Values:
centos-7-x86_64-deploy
fedora-26-x86_64-deploy
debian-9-amd64-deploy
ubuntu-16.04-amd64-deploy
debian-9-arm64-deploy
ubuntu-16.04-arm64-deploy
fedora-26-aarch64-deploy |
In build section, add execute shell:
| No Format |
|---|
#!/bin/bash -x
# Setup configuration file
CONFIG="config_${OS}_Bigtop-smoke-tests.yaml"
DEPLOY_COMPONENTS=`echo ${COMPONENTS} | cut -d '@' -f 1 | sed "s/\\./,/g"`
TEST_COMPONENTS=`echo ${COMPONENTS} | cut -d '@' -f 2 | sed "s/\\./,/g"`
OS_FOR_URL=`echo ${OS} | awk -F"-" '{ print $1"-"$2"-"$3}' | sed "s@-@/@g"`
# Adjust mem based on test target
if [[ $DEPLOY_COMPONENTS == *"mahout"* ]]; then
MEM=6g
fi
# The combination of DISTRO X ARCH is complicated. Resolved in the following logic
OS_WO_ARCH=`echo ${OS} | awk -F"-" '{ print $1"-"$2}'`
ARCH=`echo ${OS} | awk -F"-" '{ print $3}'`
case "${ARCH}" in
aarch64|arm64)
ARCH_SUFFIX="-aarch64"
;;
ppc64le|ppc64el)
ARCH_SUFFIX="-ppc64le"
;;
x86_64|amd64)
ARCH_SUFFIX=""
;;
*)
echo "Unsupported arch [${ARCH}]"
exit 1
;;
esac
# Workaround that files written inside containers are own by root
docker run --rm -v ${PWD}:/bigtop-home "bigtop/puppet:trunk-${OS_WO_ARCH}${ARCH_SUFFIX}" bash -c "chown -R ${UID}:${UID} /bigtop-home"
# Determine distro for config.yaml
case "${OS}" in
centos-*|fedora-*|opensuse-*)
DISTRO=centos
;;
debian-*|ubuntu-*)
DISTRO=debian
;;
*)
echo "Unsupported distro [${OS}]"
exit 1
;;
esac
cat > provisioner/docker/${CONFIG} <<-__EOT__
docker:
memory_limit: "${MEM:-4g}"
image: "bigtop/puppet:trunk-${OS_WO_ARCH}"
# Package CI is currently broken
# repo: https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-repos/OS=${OS},label=docker-slave/ws/output
repo: http://repos.bigtop.apache.org/releases/1.3.0/${OS_FOR_URL}
distro: ${DISTRO}
components: [${DEPLOY_COMPONENTS}]
enable_local_repo: false
smoke_test_components: [${TEST_COMPONENTS}]
__EOT__
cat provisioner/docker/${CONFIG}
# Provision
(cd provisioner/docker && ./docker-hadoop.sh -d -C ${CONFIG} -c 3 -s)
# Destroy provisinoed cluster
(cd provisioner/docker && ./docker-hadoop.sh -d)
# Workaround that files written inside containers are own by root
docker run --rm -v ${PWD}:/bigtop-home "bigtop/puppet:trunk-${OS_WO_ARCH}${ARCH_SUFFIX}" bash -c "chown -R ${UID}:${UID} /bigtop-home" |
The result will be looked like this:
https://ci.bigtop.apache.org/view/Test/job/Bigtop-trunk-smoke-tests_BIGTOP-3122/
Advanced part: Setup a SSL secured Jenkins master
...