Current state: Under Discussion
Discussion thread: here
Voting thread: here
JIRA: KAFKA-15444
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Existing Java-based Kafka brokers take a few seconds to startup. Although it is not affecting the criticality of long-running brokers in production workloads, it annoys developers instantiating hundreds of brokers for unit testing their applications during local development.
This KIP aims to deliver an experimental Apache Kafka docker image that can launch brokers with sub-second startup time and minimal memory footprint by leveraging a GraalVM based native Kafka binary and runs in the Kraft mode.
Thanks to Ozan Gunalp's work on kafka-native which inspired to formalise native Kafka artifact through this KIP.
GraalVM has provisions to build native standalone executables through ahead-of-time compilation of Java applications. These native executables generally have a smaller memory footprint and start faster than their JVM counterparts.
GraalVM based Kafka broker showed significantly faster startup time and less RAM consumption in benchmarking against JVM based broker. Details can be found below.
GC | PGO | Kafka Native Binary size | Avg Kafka Broker Startup Time | Max Memory |
---|---|---|---|---|
serial | disabled | 96MB | ~130ms | ~250MB |
serial | enabled | 67MB | ~110ms | ~230MB |
G1 | disabled | 128MB | ~140ms | ~520MB |
G1 | enabled | 82MB | ~135ms | ~540MB |
Kafka Broker using JVM
Native-Binary-serialGC Native-Binary-G1GC
Observations:
Use-cases Tested for the benchmarking: Created a topic, produced to and consumed from it using default configurations.
Machine configuration used for benchmarking:
Sample docker-compose.yml
Quick start examples
Add public documentation
While building the native image we need to provide the entry point of our application and when the native image will be run, it will start from that entry point.
Example: native-image [options] -jar jarfile -cp kafka.Kafka
Apache kafka provides multiple scripts for multiple use cases. They internally trigger the java classes which makes use of JVM. Consider a scenario:
i. We provide a docker image for Apache Kafka.
ii. User provides the clusterId
value.
iii. We need to format the Kafka log directories using kafka-storage.sh format -t <clusterid> -c config/kraft/server.properties
.
iv. We would need to support the storage format
using native image as well, otherwise it will require JVM which will defeat our purpose.
Solution:
[Preferred Approach]Make changes in the Apache Kafka codebase to add a wrapper on the entry points.
object KafkaNativeWrapper extends Logging { def main(args: Array[String]): Unit = { if (args.length == 0) { throw new RuntimeException(s"Error: No operation input provided. " + s"Please provide a valid operation: 'storage-tool' or 'kafka'.") } val operation = args.head val arguments = args.tail operation match { case "storage-tool" => StorageTool.main(arguments) case "kafka" => Kafka.main(arguments) case _ => throw new RuntimeException(s"Unknown operation $operation. " + s"Please provide a valid operation: 'storage-tool' or 'kafka'.") } } } |
The native-image, on building, does static analysis and includes only the classes and methods that are reachable from the application's entry point. It doesn’t support dynamic class loading which can be an issue if our application uses reflection.
Hence, the program elements reflectively accessed at run time must be specified using a manual configuration.
Solution: GraalVM does provide an option to create these configs automatically by running the application normally with the native-image agent attached(-agentlib:native-image-agent=config-output-dir=META-INF/native-image
).
JAVAG="/graalvm-jdk-17.0.8+9.1/bin/java" # Launch mode if [ "x$DAEMON_MODE" = "xtrue" ]; then nohup "$JAVAG" -agentlib:native-image-agent=config-merge-dir=native-configs $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp "$CLASSPATH" $KAFKA_OPTS "$@" > "$CONSOLE_OUTPUT_FILE" 2>&1 < /dev/null & else exec "$JAVAG" -agentlib:native-image-agent=config-merge-dir=native-configs $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp "$CLASSPATH" $KAFKA_OPTS "$@" fi |
Note: The above will be record only the code path which is executed. We can never be 100% sure of having an exhaustive coverage of all the reflection classes making it hard to support in production.
GraalVM is available as the following distributions:
[Preferred Approach]GraalVM Community Edition(Java 21)
This is based on OpenJDK and features like G1 garbage collector, profile guided optimisations etc are not available in GraalVM Community Edition.
Being community edition, we won’t face any licensing issues.
Note: GraalVM provides the docker images for the above distribution which can be leveraged for multi layer docker builds.
Native binaries operate independently and do not require specific packages to run. Consequently, opting for the most minimal base images will enable us to produce compact Docker images.
We propose to make use of alpine image as the base image.
While Alpine images offer a lightweight solution, contributing to a smaller Docker image size, there are certain considerations to bear in mind
Alpine vs Ubuntu Docker Base Image
The next best option I explored is the Ubuntu Docker image( https://hub.docker.com/_/ubuntu/tags) which is a more complete image.
Image naming should:
Transparently communicate the packaged Kafka version.
Maintain the above point in the event of CVEs/bugs requiring a dedicated Docker release.
Adhering to the outlined constraints, image tagging can follow this format
<image-name>:<kafka-version>
kafka-native:3.7.0
native
indicates that the image consists of the native binary.
NOTE: The JVM based Apache Kafka docker image will be named as apache/kafka:<version>
A new directory named docker
will be added to the repository. This directory will contain all the Docker related code.
Directory Structure:
|
NOTE: This structure is designed with the anticipation of introducing another Docker image based on the native Apache Kafka Broker (as per KIP-975). Both images will share the same resources for image building.
We offer two methods for passing the above properties to the container:
File Mounting: Users can mount a properties file to a specific path within the container (we will clearly document this path). This file will then be utilized to start up Kafka.
Using Environment Variables: Alternatively, users have the option to provide configurations via environment variables. Here's how to structure these variables:
.
with _
_
with __
(double underscore)
___
(triple underscore)
KAFKA_
Examples:
abc.def
, use KAFKA_ABC_DEF
abc-def
, use KAFKA_ABC___DEF
abc_def
, use KAFKA_ABC__DEF
This way, you have flexibility in how you pass configurations to the container, making it more adaptable to various user preferences and requirements.
NOTE:
GraalVM based Apache Kafka Image is an experimental docker image for local development and testing usage. GraalVM Native-Image tool is still in maturing stage, hence the usage of this image for production can’t be recommended.
Testing of the Docker Image: Sanity Tests for the P0 functionalities like Image coming up, topics creation, producing, consuming, restart etc will be added. We will also try to run the existing system tests on the built Apache Kafka native executable.
This section will be same as mentioned for the JVM Docker Image in KIP-975 build and test pipeline.
Prior to release, the Docker images must undergo building, testing, and vulnerability scanning. To streamline this process, we'll be setting up a GitHub Actions workflow. This workflow will generate two reports: one for test results and another for scanning results. These reports will be available for community review before voting.
We intend to setup a nightly cron job using GitHub Actions and leverage an open-source vulnerability scanning tool like trivy (https://github.com/aquasecurity/trivy), to get vulnerability reports on all supported images. This tool offers a straightforward way to integrate vulnerability checks directly into our GitHub Actions workflow.
Following is the plan to release the Docker image:
The docker image needs to be pushed to some Dockerhub repo(eg. Release Manager's) for the evaluation of RC Docker image.
Start the Voting for RC, which will include the Docker image as well as docker sanity tests report.
In case any docker image specific issue is detected, that will be evaluated by the community, if it’s a release blocker or not.
Once the vote passes, the image will be pushed to apache/kafka-native
with the version as tag.
Steps for the Docker image release will be included in the Release Process doc of Apache Kafka
apache/kafka-native:3.7.0
(=> image contains AK 3.7.0)Suggestion: The docker image release should be owned by the Release Manager.
As per the current release process, only PMC members should be allowed to push to apache/kafka docker image.
If the RM is not a PMC member, they’ll need to take help from a PMC member to release the image.
Oracle GraalVM: This is based on Oracle JDK and includes all the GraalVM features for free under GraalVM Free Terms and Conditions (GFTC) license
GraalVM for JDK 21
Provides G1 GC support for both linux/amd64 and linux/aarch64 architectures.
It will receive updates under the GFTC, until September 2026.
Subsequent updates of GraalVM for JDK 21 will be licensed under the GraalVM OTN License Including License for Early Adopter Versions (GOTN) and production use beyond the limited free grants of the GraalVM OTN license will require a fee.
GraalVM for JDK 17
Does not provide G1 GC support for linux/aarch64.
It will receive updates under the GFTC, until September 2024.
Post the free updates, the above distributions will require a fee. Therefore, the community version appears to be the most viable option for us.
The Distroless image was also evaluated as a potential base image. It required additional packages to be installed for our application to function.. Ultimately, we found that Alpine offered more flexibility, although there's no strong preference..
Another option considered for the image was kafka-local:3.5.1