Gate requirements

For our development we now use a hybrid of review-board and "github pull requests with offline pulling". The processes around these tolls are elaborate and not inviting nor a satisfying degree of guarantee to the quality of the resulting applied patches. Here we gather requirements for a new patch submission process that will facilitate both the quality and the hospitality principles.

In the following MoSCoW prioritization, the prefixes 'H:' and 'Q:' to indicate that either of the mentioned principles are targeted;

• H: = Hospitality, facilitate contribution
• Q: = Quality, code, contribution quality.

must

• H: the process must be inviting to both committers and contributors.
  • for contributors this will be the only means of submitting patches. For them it must be easy to enter and to update patches and to start a semi - or fully automated test and validation suite on the submitted patch. (see further requirements for formalization of this)
  • for committers it must be inviting enough so they will choose to use this process. Committers must feel the advantages over directly pushing commits to the apache git server. Commiters must be invited as they should always be allowed to commit outside this process due to for instance undisclosed security work.
• H: Incremental Patch Set:
  • Reviewer: It must be able to add patches as suggestions for points of improvement as part of code review.
  • Submitter: It must be easy to enter and to update patches, either by adding commits or by replacing the commit stack altogether.
• H: Ability to Comment: It must have a discuss facility to allow interaction between submitter and reviewers.
• Q: Comments on patch during review process must be copied to dev mailing list
• Q: Testability:
• Automated testing of patch against the specified branch by the submitter is mandatory 
• Test results should be readily available for review. The patch test status should be passed/failed
• H: it must be possible for a committer to push a patch from the submission system.

• -Q: it must be possible (for a committer) to either circumvent the gates, or to have security embargoes on patches for working on undisclosed vulnerabilities

should

• H: Replicate-able test design: it should be possible to add a test environment to the system with a list of tags indicating which tests can be run on that system.
  • Q: such a system must deliver reports in a well defined form back to the submission system.
  • Tests should be tagged with a tag defining their type and requirements set in terms of hard- and software.
• H: It should be possible to merge test reports from different test systems into one.
• Q: submitted patches should be marked as such and become immutable.
• H: It should be possible to track review results of votes on submission
• H: The system should provide notification on aging of pending reviews

• Q: Test-ability:
  • H: Reviewers should also have the capability to test submitted patches against master and any branch.
    Q: Enforcement: The system should automatically check for the presence of unit testing and integration (marvin) tests. Implementation possibility: The patch should either specify existing test case which would validate the patch or in absence of an existing test case a new test case should be provided. There may be scenarios where an automated test case may not be practical (race condition, specific deployment scenario) in those cases manual test steps, to verify the fix should be provided. How do we implement this?

could

• H: We could add squash functionality to merge patches from a submission into a single commit.
• H: Aged Pending Reviews
  • H: Automated retest on pending reviews to call out stale patches

would

• H: it would be nice for contributors to be able to push a commit after it meets a complete set of requirement.
• unit and integration/functional tests
• regression passed
• reviewed