...

Status

Current state: "Under Discussion" Voting

Discussion thread: here

Vote Discussion thread: here

JIRA: KAFKA-19161

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

• KIP-1163: Diskless Core
• KIP-1164: Topic Based Batch Coordinator
• KIP-1181: Metadata Rack Awareness for Diskless TopicsKIP-O: Garbage collection for Diskless objects

KIPs without a number will be published in the following weeks.

Each of these KIPs will have its own discussion and voting. Effort should be focused on this KIP first, and only after the community has generally agreed this KIP is something we want, should the particular implementation be designed. These KIPs will influence one another, and together they constitute the minimum viable form of this feature.

These KIPs will be aligned with the values of the Kafka community, and propose a long-lasting and extensible design that composes well with existing functionality. This will involve both substantial re-use of existing code, and refactoring in order to ensure that this feature does not need substantial rework in the future.

Further Work

In addition to the minimum viable implementation described in the integral KIPs above, below are some optional follow-ups. These are features which are not critical to the core functionality, but are natural extensions, further optimizations, and new innovations which are unlocked once the core functionality is in place.

...

This will be a backwards-compatible upgrade for existing Kafka Clusters. Diskless Topics will also support all existing APIs with the same external semantics as non-Diskless topics, including:

• Ordering
• Idempotentcy
• Transactions
• Consumer Groups/Offsets
• Queues/Share Groups
• Tiered Storage

Broadly, Diskless topics are intended to be semantically interchangeable with non-Diskless topics, while enabling latency and cost tradeoffs.

Specific compatibility, deprecation, and migration details will be covered in follow-up KIPs.

...

For users, a Cluster is an administrative boundary, one with a unified resource namespace, permissions system, and physical deployment. Users within one administrative boundary may have distinct performance requirements, and wish to choose different underlying storage parameters. This mirrors the existing topic configurability (retention, segment rolling, etc.)

Resolve some but not all rack transfer costs

Currently there are external techniques for avoiding transfer costs with non-Diskless topics, such as single-rack topics. However, these force users to make durability, availability, semantic, usability, and other application-specific compromises. If Diskless topics do not eliminate all rack transfer costs, users will still need to make these compromises with Diskless topics. By eliminating these transfer costs with internal design changes, we can offer a better user experience overall.

Additionally, if there is pressure to eliminate costs broadly, a solution which only partially resolves these costs may be shortly replaced with one that does, duplicating effort in the community. By striving for a holistic solution, we can make best use of the Kafka community's limited resources.

Do Nothing

As time progresses, this will become the single most substantial missing feature from the upstream implementation. This will drive high-scale and cloud users to Apache Kafka alternatives, which will grow in total market share. This will further fragment the control that Apache Kafka has over the Kafka Protocol, and Kafka may lose its mandate over the Kafka protocol entirely. This may lead to needing to coordinate with forks for new functionality, proliferation of hard protocol forks, or re-centralization of the protocol under a standards organization. We should take steps now to avoid or delay this outcome.