Goal: provide additional quality checks on top of the Hadoop Yetus checks, including all docker based acceptance test suite.

Goal: make it easier to contribute for newbies and external contributors (but use the existing apache)

Non-goal: avoid/remove Yetus checks

Note: everything here is just experimental which can help to decide which method can be the best for the future. This wiki page summarizes the current state/experiences to be base of a discussion.

Earlier we had Jenkins experiments see the history of this page if you are interested about the problems with Jenkins.


All the checks are committed to hadoop-ozone/dev-support/checks. You can run it locally or on any server. As of now I am running them on a kubernetes cluster with the help of argo workflow.

TLDR;

  1.  You can create pull request to apache hadoop repository
  2.  You need to add the ozone label to check it with this CI (if you have no permission to do it add a comment with the text /label ozone).
  3.  It will be checked by a cluster of kubernetes nodes.
  4.  The result will be published under https://github.com/elek/ozone-ci-q4
  5.  The checks are additional checks, not replacement of the Yetus checks.

Note: you can request new test with adding a comment with the following text: /retest

For security reason this build is triggered only for the contributors who are on a whitelist: https://github.com/elek/argo-ozone/blob/master/klepif.yaml It will be modified soon to include this list in the repository (need repo separation)

Reference repositories

  1. https://github.com/elek/argo-ozone
    1. the argo (=kubernetes workflow engine) definition
    2. Kubernetes job definition
    3. Configuration of the pull request poller
  2. https://gitub.com/elek/klepif
    1. This script polls the gihub api a checks the new commits and comments (handle the /label and /retest commands and submit new argo jobs) prow
  3. https://github.com/elek/ozone-ci-1t
    1. Contains all the test results (pr builds and nightly runs)

Future directions

  • Github actions are under evaluation. With github action we can remove the requirements of personal repositories
    • the glue code + build definition to execute the checker scripts can be committed to the main repo.
    • We don't need separated repository to store the build data
  • No labels