This page briefly describes how the Sqoop pre-commit hook works and how is everything executed.

Script test-patch.py

All internals of downloading a patch from JIRA, applying it, running tests and finally updating JIRA with results is contained in test-patch.py script in Sqoop repository. You can run it easily locally to get feeling how it works:

./dev-support/test-patch.py \
--defect SQOOP-666 \
--output patch-process/ \
--run-tests \
--post-results \
--username $jiraUsername \
--password $jiraPasword

Please note that if you omit the --post-results parameter the script won't update JIRA with run details. This is great for trying out the script on your local machine.

When creating the pre-commit infrastructure, Jarcec copied the script from flume project that by itself copied it from somewhere else. Our version has been quite enhance since then by adding additional capabilities that are specific to Sqoop, so it might be easier to look at the original file first.

Jenkins job PreCommit-SQOOP-Build

The aforementioned script test-patch.py is executed from Jenkins job PreCommit-SQOOP-Build that is parametrized and can be started by anyone who have privileges to start jobs on Jenkins. The parameters are (at the time of writing this wiki):

  • ISSUE_NUM: Number of JIRA that we need to test, we're expecting only a number not the project name, so for example to test SQOOP-666, one should enter only "666". Please not that this is different from the --defect parameter that is subsequently used.
  • ATTACHMENT_ID: JIRA's attachment ID for patch that should be tested. There is no good way how to get it other then taking a look at the associate JIRA and it's attachments and checking the URL for patch that should be tested (you should see something like http://.../jira/secure/attachment/12552136/bugSQOOP-666.patch).

Jenkins job PreCommit-Admin

The final piece in the puzzle is a Jenkins job PreCommit-Admin that periodically scans all registered projects for open JIRAs in "Patch Available" state and new attachments and automatically kicks off the PreCommit-SQOOP-Build job with parameters received from JIRA. Please note that this JOB is maintained by Hadoop project and does not belong to Sqoop. Contact person that helped Jarcec to register Sqoop here was Todd Lipcon, but any Hadoop committer should be able to help here (or at least provide pointer who can help). This is usually the best place to start looking when pre-commit hook is not executed.

 

  • No labels