Versions Compared

Old Version 6

changes.mady.by.user Brock Noland

Saved on

compared with

New Version 7

changes.mady.by.user Szehon Ho

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Background

Hive builds include Hive PreCommit Testing, and regular builds of trunk or other branches.  They are run on the Hive Ptest2 Infrastructure, which is a EC2 cluster currently sponsored by Cloudera, with each job running tests across all the resources of the cluster to get the results quickly.

Location

The infrastructure master is at

FAQ

What the hosts, paths, etc, should I be aware of?

...

ec2-174-129-184-35.compute-1.amazonaws.com

...

.  Committers can be granted access to this host by request, see HIVE-4739.

Processes

The infrastructure master hosts two processes:

  1. Jenkins

...

  1. : http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/
    • This is the admin for Hive builds.  In the dashboard, user can see state of build queues, test results of run jobs, etc.  
    • Jobs are submitted either manually in the dashboard UI, automatically by patch upload (See Hive PreCommit Testing), or by schedule (regular builds).
  2. Hive PTest WebServer
    • Exposes a REST API to take test request from the Jenkins jobs.  Each request specifies a profile property-file, which contains a set of properties such as what branch to build, what tests to run.
    • On a test request, it creates EC2 slaves, which are spot-instances.
    • Makes SSH and EC2 calls to distribute the testing to the EC2 slaves, and gather the results.  If no further test request come in 30 minutes, the slaves are shutdown.

Jenkins Configuration/Debug

  • Jenkins logs:  /var/log/jenkins/jenkins.log
  • Jenkins config: /etc/sysconfig/jenkins
  • Jenkins home: /var/lib/jenkins  (job info is located at /var/lib/jenkins/jobs)

Hive PTest Configuration/Debug

  • Logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/  (/Paths:
  • Profile properties files: /usr/local/hiveptest/etc/public/
    • Test Logs: /data/hive-ptest/logs/)
  • Profile properties files: /usr/local/hiveptest/etc/public/
  • Webservice logs: /home/hiveptest/apache-tomcat-7.0.41/logs/
  • Working Directory: /data/hive-ptest/working/ (where libs, source, etc are stored)
  • Scripts: /usr/local/hiveptest/bin/
  • Relevant JIRAS: HIVE-4675 and HIVE-4739

FAQ:

How do I restart Jenkins?

sudo service jenkins restart

How do I setup a new branch?

In /usr/etclocal/hiveptest/public/ you should create a file called branch-mr1.properties and branch-mr1.properties. Please copy trunk-mr1.properties or trunk-mr2.properties. The file should be updated accordinglycan copy existing profile properties file, and update accordingly.

How can I read or make changes to Hive PTest Infra code?

Code is located in hive git repo.  It is under /hive/testutils/ptest2.  As the Hive Ptest2 Webserver is a running process, if changes are made, it needs to restart.

How do I stop, start, restart the Hive PTest2 server, or

...

have it use the latest test infra code

...

?

No Format

$ sudo /usr/local/hiveptest/bin/stop-server.sh 
$ sudo /usr/local/hiveptest/bin/start-server.sh 
$ sudo /usr/local/hiveptest/bin/restart-server.sh 
$ sudo /usr/local/hiveptest/bin/update.sh

/usr/local/hiveptest/bin/update.sh

Details

The ptest2 webservice runs on ec2-174-129-184-35.compute-1.amazonaws.com and builds are executed from there. When a build runs on apache jenkins it makes a webservice call to the master and then tails the log file of the test in progress. Committers can be granted access to this host by request, see HIVE-4739.

...