Versions Compared

Old Version 8

changes.mady.by.user Szehon Ho

Saved on

compared with

New Version 9

changes.mady.by.user Szehon Ho

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Background

Hive build jobs include Hive PreCommit Testing, and scheduled builds of various branches with various options..  They are run on the Hive Ptest2 Infrastructure, which is a EC2 cluster currently sponsored by Cloudera, which runs each build sequentially, but splits the testing across all the resources of the cluster to achieve much higher throughput.

...

  1. Jenkins: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/
    • This is the admin for Hive builds.  In the dashboard, user can see state of build queues, test results of run jobs, etc.  
    • Jobs are submitted either manually in the dashboard UI, automatically by patch upload (See Hive PreCommit Testing), or by schedule (regular branch builds).
  2. Hive PTest WebServer
    • Exposes a REST API to take test request from the Jenkins jobs.  Each request specifies a profile property-file, which contains a set of properties such as what branch to build, what tests to run.
    • On a test request, it creates EC2 slaves, which are spot-instances.
    • Makes SSH and EC2 calls to distribute the testing to the EC2 slaves, and gather the results.  If no further test request come in 30 minutes, the slaves are shutdown.

...