...

Current state:  Under Discussion

Discussion thread: here

Voting thread: here

JIRA: KAFKA-15444

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

Native binaries operate independently and do not require specific packages to run. Consequently, opting for the most minimal base images will enable us to produce compact Docker images.
We propose to make use of alpine image as the base image.

Image Naming

Image naming should:

1. Transparently communicate the packaged Kafka version.

2. Maintain the above point in the event of CVEs/bugs requiring a dedicated Docker release.

While Alpine images offer a lightweight solution, contributing to a smaller Docker image size, there are certain considerations to bear in mind

• Alpine uses musl libc, but for native image compatibility, we require glibc. To address this, we'll need to install gcompat.
• Alpine uses an older shell instead of bash, necessitating the installation of bash to run our helper scripts.
• Alpine employs the apk package manager, which, being relatively less popular, may pose challenges in the future. There's a potential risk that certain libraries we might need could lack support from apk.

Alpine vs Ubuntu Docker Base Image

The next best option I explored is the Ubuntu Docker image( https://hub.docker.com/_/ubuntu/tags) which is a more complete image.

• Size: It has a size of 70MB compared to the 15MB of the Alpine image (post-installation of glibc and bash), resulting in a difference of 55MB.
• Performance: I executed produce/consume performance scripts on the Kafka native Docker image using both Alpine and Ubuntu, and the results indicated comparable performance between the two.

Image Naming

Image naming should:

1. Transparently communicate the packaged Kafka version.

2. Maintain the above point in the event of CVEs/bugs requiring a dedicated Docker release.

Adhering to the outlined constraints, Adhering to the outlined constraints, image tagging can follow this format
<image-name>:<kafka-version>

• kafka-native:3.7.0

  • Name of the image: kafka-native
    For example, for 3.7.0 version of kafka, the image name with tagging would be apache/kafka-native:3.7.0

  • native indicates that the image consists of the native binary.

NOTE: The JVM based Apache Kafka docker image will be named as apache/kafka:<version>

Directory Structure

A new directory named docker will be added to the repository. This directory will contain all the Docker related code.
Directory Structure:

...

GraalVM based Apache Kafka Image is an experimental docker image for local development and testing usage. GraalVM Native-Image tool is still in maturing stage, hence the usage of this image for production can’t be recommended.
Testing of the Docker Image: Sanity Tests for the P0 functionalities like Image coming up, topics creation, producing, consuming, restart etc will be added. We will also try to run the existing system tests on the built Apache Kafka native executable.

Build, Test and Scanning Pipeline

...

We intend to setup a nightly cron job using GitHub Actions and leverage an open-source vulnerability scanning tool like trivy (https://github.com/aquasecurity/trivy), to get vulnerability reports on all supported images. This tool offers a straightforward way to integrate vulnerability checks directly into our GitHub Actions workflow. 

Release Process

Following is the plan to release the Docker image:

1. RM would have generated and pushed Apache Kafka's Release Candidate artifacts to apache sftp server hosted in Image Modifiedhome.apache.org by release.py script
2. Run the automation to build the docker image(using the above Release Candidate tarball URL) and test the image.

3. The docker image needs to be pushed to some Dockerhub repo(eg. Release Manager's) for the evaluation of RC Docker image.

4. Start the Voting for RC, which will include the Docker image as well as docker sanity tests report.

5. In case any docker image specific issue is detected, that will be evaluated by the community, if it’s a release blocker or not.

6. Once the vote passes, the image will be pushed to apache/kafka-native with the version as tag.

7. Steps for the Docker image release will be included in the Release Process doc of Apache Kafka

8. eg. for AK release 3.7.0 and image released will be apache/kafka-native:3.7.0 (=> image contains AK 3.7.0)

Ownership of the Docker Images' Release

...