Status

Current state: Under Discussion

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

As the Kafka community grows, so does the number of committers and the overall rate of change to our source code. In order to facilitate a high velocity of change while maintaining safety and correctness, this KIP proposes to serialize all changes made to Kafka into a single build queue. This will ensure that no commit is able to break our mainline branches and that concurrent changes do not have bad interactions with each other. The mechanism for implementing this serial build queue is GitHub Merge Queues.

Public Interfaces

No public interfaces are affected by this KIP. This is a development process change.

Proposed Changes

Current Workflow

Our current change management workflow is that contributors open change requests known as Pull Requests (PRs). These changes are then tested, approved by a committer, and finally squash merged into the base branch. When we test our PRs, we use a special "merge" reference that is provided by GitHub. This reference is equivalent to the base branch with the PR merged into it. This means the test results of our PRs should reflect what the results would be if the PR was merged.

...

A separate class of problem exists where an author makes a small innocuous change after a PR has been tested and approved. Technically, any change should be fully retested, but there have been circumstances when this has been skipped (intentionally or otherwise).

Merge Queue

Rather than chasing down problems in our mainline branches after the fact, this KIP proposes to leverage the Merge Queue feature of GitHub.

...

This controlled and sequential workflow will allow us to ensure that no change made to Kafka will break our mainline branches.

GitHub Details

When enabled, the merge queue replaces the "Squash and Merge" button that we use today. We can still do a squash merge, but the merge button becomes "Merge when ready". Clicking this button is tantamount to merging the PR to trunk. Similar to today, only committers are authorized to click the button, and only PRs which have been approved should be merged.

...

The "Only merge non-failing pull requests" option can help mitigate the risks of batching PRs into one merge group assuming a build concurrency of greater than 1.

Rollout Plan

The extent of validation we can perform in the merge queue job is governed by the rate of PRs we expect to merge. Taking data from https://github.com/apache/kafka/graphs/commit-activity, we can see that over the last year we have merged around 15 commits per day. Our "Compile and Check Java" CI step is very consistently taking 11-12 minutes on trunk (with no caching). Our full test suite runs in around 2 hours on trunk (again, no caching). With the current rate of change in Kafka, we cannot reasonably run the full test suite for each PR in the merge queue. However, it would quite reasonable to perform compilation and static checks on each PR.

...

Following this rollout, we can experiment with the merge group batch size and concurrency. Eventually, we may opt to run some or all tests as part of the merge queue. That remains out of scope of this KIP.

Rejected Alternatives

The only other solution to the problems listed above is to require PRs to be up-to-date before merging (a configuration option in GitHub). However, since our test suite runs for around 2 hours, this would lead to a never-ending cycle of testing and updating from the base branch. Enabling the up-to-date requirement for PRs would be a burden on contributors and our CI infrastructure.