DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
| Table of Contents |
|---|
Status
Current state: Voting Accepted
Discussion thread: here
Vote thread: here
...
Currently, Apache Kafka is often operated in cloud hyperscaler environments where high-reliability object storage is available and more cost-effective than block storage for equivalent workloads. The existing Tiered Storage feature (KIP-405) provides the capability to use object storage for inactive segments, and has seen widespread adoption. However, Tiered Storage does not remove the need for replication and durable storage of active segments, which is the most substantial infrastructure cost for Apache Kafka operators on hyperscalers today.
...
- Write through to object storage, avoiding using local disks for caching instead of durable storage
- Reduce or eliminate broker disks on some nodes by configuring extremely low local retention
- Separate handling of topic data (keys and values) and metadata (offsets, timestamps, etc) for more flexible operation
- Pick pluggable commodity storage backends based on their environment
- Tradeoff cost optimization and latency on a per-topic basis
...
It is also worth mentioning that Diskless topics are not meant to change the Kafka Storage API, but to have a separate request processing that takes care of the access to remote storage.
Diskless topics may be configured for extremely short on-disk retention times (10s of seconds), or configured with memory-backed storage in clusters with suitable memory available.
In short, Diskless is to “No Disks” as Serverless is to “No Servers,” the attached disks become a less important abstraction for operators but are still functionally present.
...
This KIP will not require any changes to the codebase or documentation upon acceptance. By accepting this KIP, we will come to a consensus on the need for this feature, and its end-user requirements, but not any specific implementation details.
At a high level, Diskless Topics are a new topic type, with a distinct ingestion engine for assigning offsets for incoming data, and durably storing those assignments. They will operate in parallel to classic topics, and copy data to Tiered Storage. Data will remain accessible to consumers both in the ingestion engines and in tiered storage.
For details on the planned implementation, please see the integral follow-up KIPs:
...
- Topic Type Changing: Allow classic topics to be changed into Diskless and vice versa.
- Broker Roles: Specializing brokers between produce/consume/coordination/compaction operations and permitting heterogeneous Kafka clusters
- Parallel Produce Handling: Processing multiple Produce requests concurrently, increasing potential per-producer throughput in high latency environments.
- Iceberg Format: Allowing massively parallel processing of at-rest topic data. This work enables a pluggable storage interface where one can innovate in the log format layer independently
- Dynamically Enabled Diskless: Allowing extremely easy migrations to try out & revert Diskless
- Multi-region active-active topics with automatic failover by replicating topic metadata
These components are less defined, and currently don’t have KIPs attached. Contributions are welcome to either suggest other extensions, or design one of the above extensions. Design, discussion, and voting on these is expected to begin after the integral KIPs are complete.
Relationships to other open KIPs
Because Diskless Topics are a new type of topic, they will necessarily interact with other features which concern topics. This may improve the benefits and user experience of both features.
- KIP-1279: Cluster Mirroring Diskless Topics may be efficiently mirrored by referencing existing data in object storage
- KIP-1272: Support compacted topic in tiered storage Diskless Topics can be compacted by offloading to Tiered Storage and performing compaction there.
- KIP-1269: Configurable number of batches to retain in broker Diskless Topics will incur higher produce latency, and well-pipelined producers may desire to have additional requests outstanding.
- KIP-1248: Broker support for remote tiered storage fetch from consumer and KIP-1254: Kafka Consumer Support for Remote Tiered Storage Fetch Diskless Topics will be offloaded to tiered storage, and lagging consumers on diskless topics may perform remote storage fetches to reduce broker bandwidth and improve scalability.
Public Interfaces
This KIP does not propose any new public interfaces; but its sub-KIPs will.
...