Page History

...

It would be perfect to get all changes to the apache hive trunk run a subset of Facebook workload on a test cluster (inside Facebook) using the deployment model
in Facebook (including all the custom hooks, and the configurations) before being committed. This would give us the confidence that none of these changes are breaking
any internal Facebook deployment. However, if any test breaks, and the patch is being contributed by an outside Facebook contributor, it is impossible for the
outside committer to debug these issues, since there is no access to this test environment. The work-around is to mimic this workload, and make it available in open
source, so that all committers can run/debug these tests. So, we are trying to create a set of tests, which will run (via jenkins) on the internal Facebook test cluster
for every patch, and the patch should not be committed if these tests break. The assumption is that these tests can be run outside also, assuming you have a mysql and
hadoop installation. Eventually, these tests can be run outside Facebook, but for that, someone needs to host a hadoop cluster for running these. I dont want to sign up
for that right now, since that will increase the scope of this exercise.

The following assumptions are being made here:

• The issues caused by the hive patches are independent of hadoop. So, if the jenkins test fails (while running a version of Facebook Hadoop), it will also fail in any version of Hadoop supported by Hive.

This document lists the steps in detail of how to get there:

• Create a set of tests which are run in parallel, on a real hadoop cluster, using apache hive trunk and the version of hadoop used by facebook

Run the apache hive trunk

• .
• Make the hooks available used by Facebook be used in these tests.
• Use the same configuration options as are used internally by Facebook. (For eg. local metastore, statistics collection etc.)

Having said that. there are some hooks in facebook which are very specific to Facebook. For eg., there is a hook which transparently redirects traffic to
a production cluster. Those hooks do not make any sense for anyone outside. So, instead of getting all the hooks, the plan is to only get those hooks in open source,
which are generic. We can slowly convert all the hooks to become more generic.