Versions Compared

Old Version 8

changes.mady.by.user Unknown User (jeqo)

Saved on

compared with

New Version Current

changes.mady.by.user Anatolii Popov

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Table of Contents

Status

Current state: "Under Discussion" Accepted

Discussion thread: here

Vote Discussion thread: here

JIRA: KAFKA-19161

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).


Motivation

...

Background

The Apache Kafka protocol has become a successful base for building streaming applications, and has attracted workloads that push the Apache Kafka implementation to new limits. The Apache Kafka implementation is designed around low-durability block storage and direct replication, and provides strong consistency and high durability backed by commodity hardware.

Currently, Apache Kafka is often operated in cloud hyperscaler environments where high-reliability object storage is available and more cost-effective than block storage for equivalent workloads. The existing Tiered Storage feature (KIP-405) provides the capability to use object storage for inactive segments, and has seen widespread adoption. However, Tiered Storage does not remove the need for replication and durable storage of active segments, which is the most substantial infrastructure cost for Apache Kafka operators on hyperscalers today.

...

Even in the case where network traffic is not accounted; we believe that the operational benefits of Diskless topics are still appealing to Kafka users, for example:

  • Cluster scalability increases when less data is required to be stored on disks and rebalanced between brokers.
  • Object storages normally have better durability than local disks.

Multiple protocol-compatible alternatives to Apache Kafka now use object storage to fully replace direct replication and substantially lower the cost to operate a cluster on a hyperscaler cloud. These alternatives are finding market success and their adoption is rising, showing a general market interest in this optimization.

...

  • Eliminate inter-zone data transfer costs from replication
  • Eliminate inter-zone ingress and egress costs for data from producers and to consumersPermit multi-region active-active topics with automatic failover

Diskless topics will allow all Apache Kafka operators to:

  • Write through to object storage, avoiding local disk usageusing local disks for caching instead of durable storage
  • Reduce or eliminate broker disks on some nodes by configuring extremely low local retention
  • Separate handling of topic data (keys and values) and metadata (offsets, timestamps, etc) for more flexible operation
  • Pick pluggable commodity storage backends based on their environment
  • Balance traffic among brokers and eliminate broker hotspots with per-client granularity
  • Upgrade and scale clusters without moving active segments or electing leaders
  • Tradeoff cost optimization and latency on a per-topic basis

...

It's important to clarify what exactly "diskless" means. "Diskless" primarily refers to not using broker disk for storing user data. There are no index files stored on broker disk for diskless topics as welldisks as the primary durable storage of user data. However, diskless topics still require some broker disk usage, particularly:

  • normal topic KRaft metadata;
  • batch metadata may be stored on broker disk depending on the batch coordinator implementation (e.g. in a Kafka topic);
  • brokers may require some limited amount of disk space to perform certain operations like object compaction;
  • diskless topic user data may be stored on disk while being copied to tiered storage
  • diskless topic user data may be cached on disk to be served to consumerscaching in the read path may optionally use broker disk instead of memory.

It is also worth mentioning that Diskless topics are not meant to change the Kafka Storage API, but to have a separate request processing that takes care of the access to remote storage.

Diskless topics may be configured for extremely short on-disk retention times (10s of seconds), or configured with memory-backed storage in clusters with suitable memory available.

In short, Diskless is to “No Disks” as Serverless is to “No Servers,” the attached disks become a less important abstraction for operators but are still functionally present.

Proposed Changes

This KIP will not require any changes to the codebase or documentation upon acceptance. By accepting this KIP, we will come to a consensus on the need for this feature, and its end-user requirements, but not any specific implementation details.

At a high level, Diskless Topics are a new topic type, with a distinct ingestion engine for assigning offsets for incoming data, and durably storing those assignments. They will operate in parallel to classic topics, and copy data to Tiered Storage. Data will remain accessible to consumers both in the ingestion engines and in tiered storage.

Image Added


For details on the planned implementation, please see the integral follow-up KIPs:

KIPs without a number will be published in the following weeks.

Each of these KIPs will have its own discussion and voting. Effort should be focused on this KIP first, and only after the community has generally agreed this KIP is something we want, should the particular implementation be designed. These KIPs will influence one another, and together they constitute the minimum viable form of this feature.

These KIPs will be aligned with the values of the Kafka community, and propose a long-lasting and extensible design that composes well with existing functionality. This will involve both substantial re-use of existing code, and refactoring in order to ensure that this feature does not need substantial rework in the future.

Further Work

In addition to the minimum viable implementation described in the integral KIPs above, below are some optional follow-ups. These are features which are not critical to the core functionality, but are natural extensions, further optimizations, and new innovations which are unlocked once the core functionality is in place.

  • Topic Type Changing: Allow classic topics to be changed into Diskless and vice versa.
  • Broker Roles: Specializing brokers between produce/consume/coordination/compaction operations and permitting heterogeneous Kafka clusters
  • Parallel Produce Handling: Processing multiple Produce requests concurrently, increasing potential per-producer throughput in high latency environments.
  • Transactions on Diskless Topics: Including Diskless Topics in Exactly-Once Semantics Workloads
  • Iceberg Format: Allowing massively parallel processing of at-rest topic data. This work enables a pluggable storage interface where one can innovate in the log format layer independently
  • Dynamically Enabled Diskless: Allowing extremely easy migrations to try out & revert Diskless
  • Unification/Relationship with Tiered Storage: Identifying a long-term vision for Diskless and Tiered Storage pluginsMulti-region active-active topics with automatic failover by replicating topic metadata

These components are less defined, and currently don’t have KIPs attached. Contributions are welcome to either suggest other extensions, or design one of the above extensions. Design, discussion, and voting on these is expected to begin after the integral KIPs are complete.

Public Interfaces

Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.

A public interface is any change to the following:

  • Binary log format
  • The network protocol and api behavior
  • Any class in the public packages under clientsConfiguration, especially client configuration
  • org/apache/kafka/common/serialization
  • org/apache/kafka/common
  • org/apache/kafka/common/errors
  • org/apache/kafka/clients/producer
  • org/apache/kafka/clients/consumer (eventually, once stable)
  • Monitoring
  • Command line tools and arguments

...

Relationships to other open KIPs

Because Diskless Topics are a new type of topic, they will necessarily interact with other features which concern topics. This may improve the benefits and user experience of both features.

Public Interfaces

This KIP does not propose any new public interfaces; but its sub-KIPs will.

Compatibility, Deprecation, and Migration Plan

  • What impact (if any) will there be on existing users?
  • If we are changing behavior how will we phase out the older behavior?
  • If we need special migration tools, describe them here.

...

This will be a backwards-compatible upgrade for existing Kafka Clusters. Diskless Topics will also support all existing APIs with the same external semantics as non-Diskless topics, including:

  • Ordering
  • Idempotentcy
  • Transactions
  • Consumer Groups/Offsets
  • Queues/Share Groups
  • Tiered Storage

Broadly, Diskless topics are intended to be semantically interchangeable with non-Diskless topics, while enabling latency and cost tradeoffs.

Specific compatibility, deprecation, and migration details will be covered in follow-up KIPs.

Test Plan

...

There will be no modifications to existing tests for this KIP. Follow-up KIPs will have specific test plans.

Documentation Plan

What will the impact be on the documentation? List any affected areas/files. 

There will be no modifications to the documentation for this KIP. Follow-up KIPs will have specific plans for changes to documentation.

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.

Drop support for non-Diskless topics

...

For users, a Cluster is an administrative boundary, one with a unified resource namespace, permissions system, and physical deployment. Users within one administrative boundary may have distinct performance requirements, and wish to choose different underlying storage parameters. This mirrors the existing topic configurability (retention, segment rolling, etc.)

Resolve some but not all rack transfer costs

Currently there are external techniques for avoiding transfer costs with non-Diskless topics, such as single-rack topics. However, these force users to make durability, availability, semantic, usability, and other application-specific compromises. If Diskless topics do not eliminate all rack transfer costs, users will still need to make these compromises with Diskless topics. By eliminating these transfer costs with internal design changes, we can offer a better user experience overall.

Additionally, if there is pressure to eliminate costs broadly, a solution which only partially resolves these costs may be shortly replaced with one that does, duplicating effort in the community. By striving for a holistic solution, we can make best use of the Kafka community's limited resources.

Do Nothing

As time progresses, this will become the single most substantial missing feature from the upstream implementation. This will drive high-scale and cloud users to Apache Kafka alternatives, which will grow in total market share. This will further fragment the control that Apache Kafka has over the Kafka Protocol, and Kafka may lose its mandate over the Kafka protocol entirely. This may lead to needing to coordinate with forks for new functionality, proliferation of hard protocol forks, or re-centralization of the protocol under a standards organization. We should take steps now to avoid or delay this outcome.