Status

Current state:   [One of "Under Discussion", "Accepted", "Rejected"]

Discussion thread: here [Change the link from the KIP proposal email archive to your own email threadNot started]

JIRA: here [Change the link from KAFKA-1 to your own ticket]

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Describe the problems you are trying to solve.

Public Interfaces

Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.

A public interface is any change to the following:

• Binary log format

• The network protocol and api behavior

• Any class in the public packages under clientsConfiguration, especially client configuration

  • org/apache/kafka/common/serialization

  • org/apache/kafka/common

  • org/apache/kafka/common/errors

  • org/apache/kafka/clients/producer

  • org/apache/kafka/clients/consumer (eventually, once stable)

• Monitoring

• Command line tools and arguments

• Anything else that will likely break existing users in some way when they upgrade

Proposed Changes

Describe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users?
• If we are changing behavior how will we phase out the older behavior?
• If we need special migration tools, describe them here.
• When will we remove the existing behavior?

Test Plan

Describe in few sentences how the KIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?

Rejected Alternatives

Motivation

This KIP aims to introduce Docker Official Image (DOI) for Apache Kafka, ensuring the Kafka Docker images adhere to Docker's quality standards.

Several other Apache projects, like Flink, Spark, Solr, have already released Docker Official Images, with download figures ranging from 50 million to over 1 billion. These numbers highlight the significant demand among users. Also the Docker Official Images are always the top 1 search result, irrespective of the number of downloads. This visibility guarantees that over time, a larger number of users will opt for Docker images provided by the Apache Kafka community. The support from these Apache projects indicates that we can achieve similar success with Apache Kafka.

Post-support, the Dockerfile for Apache Kafka will continue to be managed by the Apache Kafka community and undergo review by Docker. The Docker community will oversee the images' maintenance, aligning them with Docker's quality standards. Adopting this approach, means that the build and release process for the official Docker image will be managed by Docker Hub, eliminating any extra maintenance burden for the Apache Kafka community.

Public Interfaces

• There will be a new additional artifact i.e. Official Kafka docker image for every Apache Kafka release.

• Add public documentation in Official Docker Image Repo.

Proposed Changes

• There will be a docker official image as an additional artifact for every Apache Kafka release. The docker files for the same will be maintained by the Apache Kafka community, but the images will be maintained by the Docker community.

• This docker official image, like the existing OSS sponsored images, will consist of JVM based Kafka and will have support for Linux based AMD and ARM architectures. 

• Extend the Apache release process to publish the new docker image to official DockerHub.

Compatibility, Deprecation, and Migration Plan

• No impact

Making the Dockerfile Compliant

• We will need to make sure to carry out the necessary modifications to ensure our Dockerfile aligns with the guidelines and criteria established for Official Docker Images, and ensure that it adheres to Docker's best practices. We will take steps to ensure our image aligns with the below pointers.

  • The New Image checklist - NEW-IMAGE-CHECKLIST.md

  • Best practices for Dockerfile instructions - Best practices for Dockerfile instructions

  • General best practices for writing Dockerfiles - General best practices for writing Dockerfiles

  • Review Guidelines: https://github.com/docker-library/official-images/blob/master/README.md#review-guidelines

• This step is necessary because the Dockerfiles will be subject to review by the Dockerhub team, and will require back-and-forth communication with them.

Proposed Workflow for Release Process

1. Initial Setup:

   • Establish a new directory, docker_official_images, within trunk to centralize Docker Official Image Dockerfiles and scripts.

   • Ensure dockerfile is compliant as per Docker Official Images guidelines.

2. Release Candidate (RC) Preparation:

   • RM would have generated and pushed Apache Kafka's Release Candidate artifacts to apache sftp server hosted in home.apache.org by release.py script.

   • For all subsequent steps - the above Release Candidate tarball URL will be used.

   • Upon confirming readiness, proceed to the Docker image preparation phase.

3. Docker Image Preparation via GitHub Actions:

   • Implement an automated GitHub Actions workflow to:

     • Generate a version-specific folder, following the major_version.minor_version.patch_version-rc naming scheme, under the docker_official_images directory, for housing version-specific Dockerfiles and scripts.

     • For each new RC within the same major_version.minor_version sequence, we will update the existing directory by overwriting its contents. This ensures that only the most recent RC Dockerfiles and scripts are maintained for a particular major_version.minor_version.

     • Convert build arguments (ARGs) to fixed environment variables (ENVs) to adhere to Docker best practices.

     • The github actions workflow uploads this folder as an artifact.

     • The RM checks if there is already a folder with the same name in trunk. If yes, he deletes this, and put this folder under the docker_official_images directory in his fork of the kafka repo, and merges this change into the official kafka repo by raising a PR. To make this step more automated, we can do the same via a script that takes in the local path of the downloaded artifact as an input.

     • The pull request (PR) is merged into trunk.

4. Review and Integration:

   • Upon PR approval and merge, cherry-pick the changes into the designated RC branch.

5. Docker Build and Release Testing:

   • Perform Docker build tests to ensure image integrity. The dockerfiles for the same will be used from the specific folder in the docker_official_images directory.

   • Start the Voting for RC, which will include the Docker image as well as docker sanity tests report.

   • In case any docker image specific issue is detected, that will be evaluated by the community, if it’s a release blocker or not.

   • Once the vote passes, run the Docker release pipeline to publish RC images to the Docker Hub open-source software (OSS) sponsored images. The image will be pushed to OSS sponsored apache/kafka with the version as tag.

   • Steps for the Docker image release will be included in the Release Process doc of Apache Kafka

6. Iterative RC Updates:

   • Repeat steps 2 through 5 for subsequent RCs, ensuring the repository only retains the latest patch version and RC.

7. Final Release Promotion:

   • Upon final release approval, run the Docker promote pipeline from the RC branch to the OSS Docker Hub. The latest major_version.minor_version.patch_version image will now be available on the OSS Docker Hub.

8. Post-Release Maintenance:

   • Return to the trunk to perform maintenance tasks:

     • Remove unsupported version folders from the docker_official_images directory to maintain repository cleanliness.

     • Remove the -rc identifier from the major_version.minor_version.patch_version-rcfolder. This serves as the new source of truth for the DOI. Ensure that the folder of the version being released does not have the -rc tag at the end.

     • Commit these changes → this new commit hash will now be provided to the DOI repo as the new images.

   • Run a script to automate the creation of a PR template for the Docker Hub official images library submission.

9. Docker Hub Submission:

   • Submit the final PR to Docker Hub’s official images repository. Following PR approval, the Kafka Docker image becomes officially available.

Directory Structure

A new directory named docker_official_images will be added to the /docker repository. This directory will contain all the Official Docker Images related code.

kafka/
    - docker/
        - docker_official_images/ #Contains resources needed to create the official docker images.
            - 3.7.2-rc/           #Contains resources needed to create the docker images for the latest RC tag for the 3.7.2 version. 
              - jvm/              #Contains Dockerfile for the JVM-based Apache Kafka Docker image.
                - Dockerfile + necessary scripts 
            - 3.7.1/              #Contains resources needed to create the official docker images for the 3.7.1 version.
              - jvm/              #Contains Dockerfile for the JVM-based Apache Kafka Official Docker image.
                - Dockerfile + necessary scripts
            - 3.7.0.              #Contains resources needed to create the official docker images for the 3.7.0 version.
              - jvm/              #Contains Dockerfile for the JVM-based Apache Kafka Official Docker image.
                - Dockerfile + necessary scripts
            - 3.6.5/
              - jvm/
                - Dockerfile + necessary scripts
        - jvm/
            - Dockerfile          #Dockerfile for the JVM-based Apache Kafka Docker image.
        - resources/              #Contains resources needed to create the Docker image.
        - test/                   #Contains sanity tests for the Docker image.
        - docker_build_test.py    #Python script for building and testing the Docker image.
        - docker_release.py       #Python script for building the Docker image and pushing it to Docker Hub.
        - docker_build_latest_artifact.py #Python script for building the artifact for the current RC release.

Test Plan

Existing tests for docker image should pass.

Rejected Alternatives

Creating a separate repository named apache-docker within the Apache GitHub organization.This strategy mirrors the practices of other Apache projects, such as Spark and Flink, where Docker-related files and scripts reside in a distinct repository. However, this move would involve transferring existing Docker resources to the new repository too, without providing any added advantage. This approach does not provide any benefit in the requirement for creating specific folders with hardcoded values for each Docker image releaseIf there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.