Authors: Greg Harris, Ivan Yurchenko, Jorge Quilcate, Giuseppe Lillo, Anatolii Popov, Juha Mynttinen, Josep Prat, Filip Yonov

Status

Current state: Under Discussion

Discussion thread: here

JIRA: KAFKA-19161

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

KIP-1150: Diskless Topics introduces the concepts of diskless topics. This KIP describes the core functionality of diskless topics, such as the produce and consume paths.

Glossary

Proposed Changes

Overview

The KIP proposes a new kind of topic in Kafka called diskless topic, which have the following properties and immediate effects:

  1. Data in diskless topics is durably stored solely in object storage, and not in segments on broker disks. 
    1. Classic concepts of log directories, segment rolling, object writing and deletion, message flushing, tiered storage, etc. and the related APIs do not apply. Effectively, brokers will not create any files related to diskless topics.
    2. Remote storage may have higher latency than local disks, increasing the latency of Kafka requests and end-to-end data latency.
  2. Kafka delegates replication of diskless topics to object storage, and does not perform replication itself.
    1. Classic concepts of replicas, replication, ISR, partition assignment and the related APIs do not apply.
    2. All operators can use efficient types of storage backends, such as ones with erasure coding.
    3. Hyperscaler operators can avoid most inter-zone data replication costs.
  3. All brokers are capable of interacting with all diskless topics, and no broker is uniquely considered the leader of a partition.
    1. Classic replica leader election is not applicable to diskless topics and all requests may use any broker as if it were the leader.
    2. Clusters are able to perform fine-grained client balancing across the cluster independently of topic/partition hot spots.
    3. Hyperscaler operators can avoid most inter-zone data ingress/egress costs.

This KIP allows certain non-fundamental functional limitations between diskless and classic topics:

  1. Diskless topics will not immediately support compaction.
  2. Diskless topics will not immediately support transactional writes.

The preliminary research suggests that these limitations are not fundamental and can be resolved. They are out of scope of this KIP, and are expected to be addressed in follow-up KIPs. See the Further Work section for more details.

In other functional aspects, diskless topics are indistinguishable from classic topics. This includes durability guarantees, ordering guarantees, non-transactional producer API, consumer API, consumer groups, share groups, data retention, and authorization mechanisms. Streaming applications can be moved from classic to diskless topics without changing client versions, application code, or experiencing correctness problems.

Because a single cluster can host diskless, classic, and tiered topics together, application developers are able to trade off latency and cost on a per-topic basis, while not incurring additional operational overhead of multiple clusters.

Data Flow

In order to satisfy the requirement of multiple brokers writing to a single partition with a total order, it is necessary to decouple the handling of batch data and batch metadata. This permits low-cost replication of batch data with fewer consistency guarantees, while processing batch metadata and the state of the partition with strong consistency guarantees.

In classic topics, the broker serving the Produce request is responsible for:

  1. Validating the incoming data.
  2. Assigning offsets and timestamps.
  3. Injecting offsets/timestamps into the batch data.
  4. Writing batches to durable storage.
  5. Waiting for replication to complete.
  6. Returning the Produce response.

For diskless topics, the Batch Coordinator is now responsible for offset/timestamp assignment (2), and the fetch broker is responsible for injecting the offsets into the batch data (3). These components provide the effect that fetched data sent to the consumer has the same structure as data appended to a classic segment.

The data being persisted to object storage has no implicit order, and ordering is delegated to the batch coordinates. Coordinates passed from each broker to the batch coordinator are locally ordered, and have an order only relative to other coordinates passed from that same broker. The batch coordinator performs the append and assigns a global order to the batch coordinates in each partition.

Data Format and Shared Log Segment Objects

The main unit of storage in Kafka is a record batch, and is the smallest unit of data which may be produced or consumed. Kafka groups these batches together into contiguous files to more efficiently manipulate them in the backing storage, while still preserving access to the individual batches. This remains true in diskless topics.

Batches for classic topics are grouped into Segments, which each contain batches from only a single partition, and batches in log segments always have offsets and timestamps applied. In contrast, data from multiple diskless topics and partitions are mixed in single objects, called shared log segments. This is necessary to keep the object storage write operation costs reasonable.

The batches within a shared log segment are ordered by their layout in the object, but do not have any inherent order relative to other shared log segments. Their logical offsets and append timestamps may be unset.

By locating data from multiple partitions together, shared log segments prevent the number of topic-partitions from influencing object read and write behavior, and reduce the total number of individual reads and writes to the backing storage.

Batches are grouped by partition into contiguous parts of the object. This improves the locality of later operations in the same way as segments do for classic topics. Shared log segment objects are given globally unique (e.g. UUID) names and uploaded without coordination or conflicts.

The shared log segment objects are immutable, similar to inactive segments in classic and tiered topics. Once an object is uploaded, its content is associated with its name for the whole lifetime of the object. This also aligns with the typical interface of object storage where random writes to an object are not supported.

Produce path

Data is produced to diskless topics in the following way:

  1. Producers send Produce requests to any broker.
  2. The broker accumulates Produce requests in a buffer until exceeding some size or time limit.
  3. When enough data accumulates or the timeout elapses, the Broker creates a shared log segment and batch coordinates for all of the buffered batches.
  4. The shared log segment is uploaded to object storage and is written durably.
  5. The broker commits the batch coordinates with the Batch Coordinator (described in details in KIP-1164).
  6. The Batch Coordinator assigns offsets to the written batches, persists the batch coordinates, and responds to the Broker.
  7. The broker sends responses to all Produce requests that are associated with the committed object.

This is illustrated on the following sequence diagram:

Each broker accumulates its buffers and uploads objects with its requested offsets independently, with no coordination. However, the Batch Coordinator ensures that the batch coordinates are committed in order and the offsets are assigned correctly. Moreover, a broker can upload multiple objects in parallel, it only needs to commit them with the Batch Coordinator in their order.

Brokers will be able to process requests containing partitions from both classic and new diskless topics in the same batch. When responding to these mixed-typed requests, the broker will be delayed until the diskless partitions are committed.

Error handling and segment object garbage collection

There may be errors during various steps of this process. In some cases, garbage objects (i.e. objects that are uploaded but not committed to the batch coordinator) may be left in the object storage. Sometimes, these objects cannot be cleaned up on the spot. Consider the following scenarios:

  1. The Broker fails to upload the object. No garbage object is created. The pending Produce requests receive responses with errors. It’s safe for the producers to retry.
  2. The Brokers succeeds in uploading the object, tries to commit it, and couldn’t receive a response from the batch coordinator. A garbage object is left on the object storage. It’s not safe to clean it on the spot without consulting the batch coordinator, which may not be available. The garbage cleanup is delegated to the separate asynchronous process. The pending Produce requests receive responses with errors. It’s safe for the producers to retry provided they use idempotent produce.

The concrete details of the asynchronous garbage collection process is out of scope of this KIP and will be described in a separate KIP.

Consume path

Data is consumed from diskless topics in the following way:

  1. The consumer sends a Fetch request to the broker.
  2. The broker queries the Batch Coordinator for the relevant batch coordinates.
  3. The broker gets the data either from the object storage and/or from the cache.
  4. The broker injects the computed offsets and timestamps into the batches.
  5. The broker constructs and sends the Fetch response to the Consumer.

Depending on the structure of the shared log segment, a single Fetch request could be satisfied by one GET request, or may need multiple requests. In order to improve the locality of requests, Diskless shared log segments will be periodically compacted in order to increase sequential spans of batches. The compaction is discussed in detail in KIP-1165: Object Compaction for Diskless.

Object storage implementations are expected to implement caching to reduce the quantity of overall requests to the underlying storage. These caches should be partitioned per-rack, to maintain good locality for requests. Caching is discussed in detail in a follow-up KIP.

Batch deletion

There are various reasons a batch may require deletion:

Since objects are immutable, logical batch deletion doesn’t change the object content. For each object, it’s necessary to track its effectively used size. This size decreases by the batch size when a batch is deleted from the object. When the used size becomes 0, the object could be actually deleted.

Object deletion will be implemented as an asynchronous operation. First, the object is marked for deletion by the batch coordinator and a broker performs the actual deletion from the object storage. This has two advantages:

  1. The Batch Coordinator doesn’t need to be concerned about the object storage, to have credentials and the code to access it.
  2. The deletion could have a grace period. The grace period is a useful way to allow potentially in-progress Fetch requests to successfully finish.

To enforce the time and size retention settings of topics, a background process will periodically check metadata of diskless topics and logically delete affected batches.

It’s quite possible that for compliance reasons, a particular batch has a deadline for physical deletion. Despite the batch being logically deleted, its data are located in an object that may be kept alive by other batches, potentially forever. Therefore, batches with physical deletion deadlines must be either moved to isolated files during merging, or additional merge passes will be necessary to physically delete the data. This problem is addressed in KIP-1165.

Public Interfaces

Plugin Interfaces

Apache Kafka will provide pluggable interfaces which are intended to abstract functionality that may require contacting an external service.

Batch Coordinator

This interface abstracts the batch order consensus mechanism needed to establish the order of batches within each partition and generally keep track of the metadata. This interface is out of scope for this KIP and discussed in KIP-1164

Storage Backend

This interface abstracts the access methods to the object storage.

public record ByteRange(long offset, long size) {}

public interface ObjectStorage extends Configurable {
    /**
     * Uploads an object to object storage.
     * <p>An exception must be thrown in case the number of bytes streamed from
     * {@code inputStream} is different from {@code length}.
     * @param key                      key of the object to upload.
     * @param inputStream              data of the object that will be uploaded.
     * @param length                   length of the data that will be uploaded.
     * @throws StorageBackendException if there are errors during the upload.
     */
    void upload(ObjectKey key, InputStream inputStream, long length)
        throws StorageBackendException;


    /**
     * Fetch a range of an object from object storage.
     * <p>This result should be cached for later access
     * @param key                      key of the object to fetch.
     * @param range                    range of bytes within the object.
     * @throws StorageBackendException if there are errors during the fetch.
     */
    InputStream fetch(ObjectKey key, ByteRange range)
        throws StorageBackendException;

    /**
     * Idempotently delete multiple objects.
     * <p>If an object doesn't exist, the operation should still succeed	
     * to be idempotent.
     * @param keys                     keys of objects to delete.
     * @throws StorageBackendException if there are other errors which prevent deletion.
     */
    void delete(Set<ObjectKey> keys)
        throws StorageBackendException;
}

Apache Kafka will not provide a production-grade implementation of this interface, because this would require Kafka depending on third party storage drivers. The list of potential storage is big and it’s not possible to provide implementations for all of them. Implementations will be provided by 3rd party developers in the Kafka ecosystem.

Broker Configurations

Topic Configurations

Client Configurations

Producer:

Monitoring

The following metrics may be useful for operators:

  1. Object upload:
    1. count and rates;
    2. object size average and percentiles;
    3. upload traffic;
    4. latency;
    5. errors.
  2. Object commit:
    1. count and rates;
    2. latency;
    3. errors.
  3. Read:
    1. count and rates;
    2. GET requests per Fetch request.

Command line tools

Existing tools to be adapted:

kafka-diskless-metadata.sh is a new tool, which does the following:

  1. Diskless topic overview:
  2. Getting object metadata, including object key, total size, used size.

Compatibility, Deprecation, and Migration Plan

Existing users upgrading to a version of Kafka with support for diskless writes will not experience any change in behavior. All broker and topic configurations will have defaults which are consistent with the existing storage model. This will be a backwards-compatible upgrade. These users may downgrade without additional steps.

Users which configure diskless brokers but no diskless topics may experience failures related to those configurations if they are invalid (e.g. a plugin is not installed, or backing service is unavailable). If the configuration passes validation and the brokers are able to start, users will experience no change in behavior. These users may downgrade without additional steps.

Users which configure diskless brokers and diskless topics will be able to produce and consume data with the same semantics and consistency model as traditional topics (save for the explicitly outlined exceptions/limitations), but with a higher latency than expected. These users will need to migrate away from and delete all diskless topics before downgrading.

Test Plan

We need to create new integration tests that would cover the following:

We need to create new systems tests that are end to end but stubbed with MinIO where:

Various failure scenarios (e.g. the Batch Coordinator or Object Storage are inaccessible or partially accessible) should be tested extensively, with the focus primarily on correctness.

The majority of the existing produce and consume tests should be adapted to diskless topics by parameterizing the topic configuration. However, some of the current tests fall in the “known limitations and exceptions”.

Documentation Plan

These are the areas where we need to add documentation:

Rejected Alternatives

Use Tiered Storage as-is

Tiered storage only affects the behavior of inactive segments, while active segments continue to use block storage and replication. It is possible to avoid inter-zone replication by setting replication.factor=1, but causes the topic to inherit the durability of block storage, which may experience correlated failures within a single rack.

Tiered Storage as-is forces a trade-off between durability and replication costs.

Using “Aggressive Tiering”, an operator may configure their cluster to roll active segments quickly, reducing the total active segment size. This has the effect of also reducing the window that data is stored non-durably, such that it might be possible to delay acks until a segment is uploaded to tiered storage. However, as tiered storage uploads individual segments, these requests can incur a large I/O cost. In many cases, this will negate the positive effects from shrinking or eliminating the block storage.

Tiered Storage as-is forces a trade-off between durability and excessive I/O overhead.

Tiered topics still have leaders, and under the Kafka protocol, producers must produce to the leader, even if the leader is in a different rack, incurring cross-rack transfer costs. To mitigate this, all producers and brokers can be placed in a single rack.

Tiered storage as-is forces a trade-off between availability and producer ingress costs.

As a summary, it can be seen that the inter-AZ traffic can be avoided but with the cost of losing availability of producers and brokers and availability and durability of topic data. The local storage requirement can be minimized with certain penalties explained earlier but not fully eliminated. The authors are not aware of any mechanism to improve tiered storage to eliminate these shortcomings.

Tiered Storage metadata being abstracted to segment granularity and segments being limited to a single partition prevents the overall cost optimization of the system, and forces operators to sacrifice very important guarantees in order to unlock cost savings. Diskless topics maintain these guarantees, while still unlocking cost savings.

File system driver

One may argue that we could keep the Kafka code unchanged or minimally changed, but instead use a sophisticated file system driver. Brokers would use the local broker disk backed by the driver and write segment files and other file types and read from them normally. The driver would be responsible for transferring the IO to the object storage. The native Kafka replication could be prevented by setting replication.factor=1. This approach comes with a number of challenges that make us reject it.

Mapping block storage operations to object storage directly is inefficient, and requires some assumptions about access patterns. For example, object storages don’t widely support random writes or appends, so objects would need to be overwritten often. This would result in poor performance and high costs in hyperscalers which have a per-request cost.

There are log-structured file systems implemented on top of object storage, but these are either proprietary or immature. And because they operate on a lower-level abstraction than Diskless Shared Log Segments, they are not able to take advantage of application-specific access behaviors and optimizations.

A filesystem driver also does not cover the rack awareness functionality, and modifying the code would still be required to implement a similar feature. Kafka may also need to be aware of some external component as low level as the file system driver to collaborate with it and have brokers which are able to access data they have not replicated. Kafka is not prepared to manage multiple writers to a single local filesystem.

This type of solution would be less accessible and portable between operating systems. Kernel-space or user-space drivers need to be implemented and supported for every operating system that Kafka supports. This also requires a specific skill set that may not be widely represented in the present Kafka community.