Status

Current state:  Under Discussion

Discussion thread: here

JIRA: KAFKA-15444

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Existing Java-based Kafka brokers take a few seconds to startup. Although it is not affecting the criticality of long-running brokers in production workloads, it annoys developers instantiating hundreds of brokers for unit testing their applications during local development.

This KIP aims to deliver an experimental Apache Kafka docker image that can launch brokers with sub-second startup time and minimal memory footprint by leveraging a GraalVM based native Kafka binary. 

Thanks to Ozan Gunalp's work on kafka-native which inspired to formalise native Kafka artifact through this KIP.

Broker Startup Time, CPU Usage and Memory Footprint with GraalVM based Native Kafka Broker

GraalVM has provisions to build native standalone executables through ahead-of-time compilation of Java applications. These native executables generally have a smaller memory footprint and start faster than their JVM counterparts.
GraalVM based Kafka broker showed significantly faster startup time and less RAM consumption in benchmarking against JVM based broker.  Details can be found below.

Avg Kafka Broker Startup Time GraalVM versus JVM

• Avg Kafka Server Startup Time Using JVM(default configs): ~1150 ms - 1200ms
• Following is the table of startup times of GraalVM based Native Kafka Broker depending on GC and PGO:

Kafka Broker CPU and Memory usage for GraalVM versus JVM

                              Kafka Broker using JVM

                         Native-Binary-serialGC                                                                    Native-Binary-G1GC

Observations:

• The startup time of the GraalVM broker is ~1/9th of the startup time of the JVM broker.
• No CPU spike is observed for GraalVM broker but JVM broker had sudden spikes both when the broker started and also when the load testing started.
• Considerable difference in memory usage is observed as well. For JVM broker the max memory usage went up to ~1GB as compared to the ~500MB in case of GraalVM broker with G1GC and ~250MB in case of GraalVM broker with serial GC.

Use-cases Tested for the benchmarking: Created a topic, produced to and consumed from it using default configurations.

Machine configuration used for benchmarking:

• Machine: m6i.2xlarge
• Architecture: x86_64
• OS: ubuntu
• Java version: openjdk 17.0.8 2023-07-18
• GraalVM native-image version: 
• native-image 17.0.8 2023-07-18
• GraalVM Runtime Environment Oracle GraalVM 17.0.8+9.1 (build 17.0.8+9-LTS-jvmci-23.0-b14)

Public Interfaces

• There will be a new additional artifact i.e. Apache Kafka docker image for every Apache Kafka release.

• Sample docker-compose.yml

• Quick start examples

• Add public documentation

Proposed Changes

• There will be a docker image as an additional artifact for every Apache Kafka release. 
• This docker image will consist of GraalVM native-image based kafka binary and will have support for Linux based AMD and ARM architectures.
• Add sanity tests to run against the new docker image for each tag. 
• There will be a new build system for generating Kafka native executables for both ARM and AMD architectures. This new build system is in addition to the existing Java and Scala-based Kafka build systems.
• Add release script to extend the Apache release prcocess to publish a new docker image to public DockerHub.

Handle Multiple Entry-points in the Native Kafka Broker

While building the native image we need to provide the entry point of our application and when the native image will be run, it will start from that entry point.
Example: native-image [options] -jar jarfile -cp kafka.Kafka
Apache kafka provides multiple scripts for multiple use cases. They internally trigger the java classes which makes use of JVM. Consider a scenario:
i. We provide a docker image for Apache Kafka.
ii. User provides the clusterId value.
iii. We need to format the Kafka log directories using kafka-storage.sh format -t <clusterid> -c config/kraft/server.properties.
iv. We would need to support the storage format using native image as well, otherwise it will require JVM which will defeat our purpose.

Solution:

1. Create 2 native images with different entry points? This will increase the size of the docker image. Also, this is not scalable.

2. [Preferred Approach]Make changes in the Apache Kafka codebase to add a wrapper on the entry points.

   object KafkaNativeWrapper extends Logging {
     def main(args: Array[String]): Unit = {
       if (args.length == 0) {
         throw new RuntimeException(s"Error: No operation input provided. " +
           s"Please provide a valid operation: 'storage-tool' or 'kafka'.")
       }
       val operation = args.head
       val arguments = args.tail
       operation match {
         case "storage-tool" => StorageTool.main(arguments)
         case "kafka" => Kafka.main(arguments)
         case _ =>
           throw new RuntimeException(s"Unknown operation $operation. " +
             s"Please provide a valid operation: 'storage-tool' or 'kafka'.")
       }
     }
   }

Generate Metadata Configs for Building Native-Image Kafka Broker

The native-image, on building, does static analysis and includes only the classes and methods that are reachable from the application's entry point. It doesn’t support dynamic class loading which can be an issue if our application uses reflection.
Hence, the program elements reflectively accessed at run time must be specified using a manual configuration.

Solution: GraalVM does provide an option to create these configs automatically by running the application normally with the native-image agent attached(-agentlib:native-image-agent=config-output-dir=META-INF/native-image).

• To generate configs we intend to use the existing Apache Kafka System Tests as they are quite exhaustive.
• We will run the the system tests using GraalVM JIT and attaching the native-image agent to the process
• Merge the configs of all the tests.

JAVAG="/graalvm-jdk-17.0.8+9.1/bin/java"

# Launch mode
if [ "x$DAEMON_MODE" = "xtrue" ]; then
  nohup "$JAVAG" -agentlib:native-image-agent=config-merge-dir=native-configs $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp "$CLASSPATH" $KAFKA_OPTS "$@" > "$CONSOLE_OUTPUT_FILE" 2>&1 < /dev/null &
else
  exec "$JAVAG" -agentlib:native-image-agent=config-merge-dir=native-configs $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp "$CLASSPATH" $KAFKA_OPTS "$@"
fi

Note: The above will be record only the code path which is executed. We can never be 100% sure of having an exhaustive coverage of all the reflection classes making it hard to support in production.

GraalVM Version for Building Native-Image Kafka Broker

GraalVM is available as the following distributions:

1. [Preferred Approach]GraalVM Community Edition(Java 21)

   1. This is based on OpenJDK and features like G1 garbage collector, profile guided optimisations etc are not available in GraalVM Community Edition.

   2. It includes Serial GC.

   3. Being community edition, we won’t face any licensing issues.

2. Oracle GraalVM: This is based on Oracle JDK and includes all the GraalVM features for free under GraalVM Free Terms and Conditions (GFTC) license

   1. [Alternative Approach]GraalVM for JDK 21

      1. Provides G1 GC support for both linux/amd64 and linux/aarch64 architectures.

      2. It will receive updates under the GFTC, until September 2026.

      3. Subsequent updates of GraalVM for JDK 21 will be licensed under the GraalVM OTN License Including License for Early Adopter Versions (GOTN) and production use beyond the limited free grants of the GraalVM OTN license will require a fee.

   2. GraalVM for JDK 17

      1. Does not provide G1 GC support for linux/aarch64.

      2. It will receive updates under the GFTC, until September 2024.

Note: GraalVM provides the docker images for each of the above distributions which can be leveraged for multi layer docker builds.

Docker Base Image

Native binaries operate independently and do not require specific packages to run. Consequently, opting for the most minimal base images, such as Alpine and Distroless, will enable us to produce compact Docker images.
We propose to make use of alpine image as the base image.

Image Naming

Image naming should:

1. Transparently communicate the packaged Kafka version.

2. Maintain the above point in the event of CVEs/bugs requiring a dedicated Docker release.

Adhering to the outlined constraints, image tagging can follow this format
<image-name>:<kafka-version>-<optional-suffix>

• [Preferred Approach]kafka-local:3.5.1

  • local indicates that this image is intended only for the local development use.

  • 3.5.1 is the sample kafka version.

  • kafka-local:3.5.1-1 In case of docker dedicated release, added a suffix -1

OR

• kafka-native:3.5.1

  • native indicates that the image consists of the native binary.

  • For many users, native might not make much sense.

  • kafka-native:3.5.1-1 In case of docker dedicated release, added a suffix -1

Compatibility, Deprecation, and Migration Plan

• For existing apache kafka users there will be no impact as native-image based kafka docker image will be a new feature.
• The GraalVM native-image based Apache Kafka docker image will be an experimental docker image.
• Unlike JVM, GraalVM native-image performs ahead-of-time compilation and does not support dynamic class loading. It requires extensive testing to understand the total broker functionality support and performance through GraalVM native-image. The GraalVM native-image based container is recommended only for development, and testing and not for production workloads.
• For docker image catering production workloads refer the KIP-975.

Test Plan

GraalVM based Apache Kafka Image is an experimental docker image for local development and testing usage. GraalVM Native-Image tool is still in maturing stage, hence the usage of this image for production can’t be recommended.
Testing of the Docker Image: Sanity Tests for the P0 functionalities like Image coming up, topics creation, producing, consuming, restart etc will be added

Release Process

Following are the 2 ways to introduce Docker image generation in the release process:

1. Build Docker image using RC binary

   1. Use the binary generated and pushed to RM’s apache sftp server hosted in home.apache.org by release.py script as source of kafka binary for Docker image of RC.

   2. Run the script to build and test the image locally.

   3. The docker image needs to be either pushed to some private Dockerhub repo OR we can just keep the Dockerfile along with relevant scripts in home.apache.org , for the evaluation of RC Docker image.

   4. Start the Voting for RC, which will include the Docker image as well.

   5. In case any docker image specific issue is detected, that will be evaluated by the community, if it’s a release blocker or not.

   6. Once the vote passes, the image will be pushed to apache/kafka with the version as tag.

   7. Steps for the Docker image release will be included in the Release Process doc of Apache Kafka

2. [Preferred Approach]Docker Image release without voting during AK RC

   1. The docker image is basically packaging official Apache Kafka released tarballs.

   2. The voting for the Apache Kafka released tarballs had already been successfully done.

   3. So the process of Docker image release can be completely separate and should happen post Apache Kafka release.

   4. The docker image release process will involve executing a script which will handle building and testing the image locally.

   5. Once the Docker image artifact is ready, it will get reviewed by the community and voting will be conducted, just for the Docker image release.

   6. In case any docker image specific issue is detected, that will be evaluated by the community, if it’s a release blocker or not.

   7. This image will then be pushed to apache/kafka with proper tagging to communicate kafka version.

Ownership of the Docker Images' Release

• Suggestion: The docker image release should be owned by the Release Manager.

• As per the current release process, only PMC members should be allowed to push to apache/kafka docker image.

• If the RM is not a PMC member, they’ll need to take help from a PMC member to release the image.

Rejected Alternatives

NA