You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Authors: Henry Cai

Status

Current state: "Under Discussion"

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: here [Change the link from KAFKA-1 to your own ticket]

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

KIP-405 enables Kafka brokers to upload log segments to remote tiered storage, this opens the door to many new use cases, one of them is having consumers to fetch from the tiered storage directly.

Currently, when a new consumer or a fallen-off consumer requires fetching messages from a historical period, and those messages are no longer present in the Kafka broker's local storage, the broker must download the message from the remote tiered storage and subsequently transfer the data back to the consumer. This process involves two data transfers, resulting in increased network latency and transfer costs. Critically, when a broker downloads the data segment from remote storage and stores it locally, it imposes additional IOPS on the network and local disk capacity. Furthermore, this action negatively impacts the page cache established for messages closer to the end of the message queue.

With the data segment file already present on remote tiered storage, there is no reason why the consumer cannot directly fetch them from remote storage without affecting the broker performance.  To address this gap, we propose here to allow the consumers to fetch from the remote tiered storage.

Proposed Changes

Know the location of remote log segment file

In order to support fetching from the remote tiered storage, the consumer needs to be notified about the location of the remote log segment file for the fetch offset.  This can be done either by (1) the consumer subscribes the tiered storage metadata topic: __remote_log_metadata, filters to the partition it interests in, parses each metadata message and tracks the state change;  (2) have the leader broker notifies the consumer about the location of the remote log segment file during the consumer FetchRequest/Response message exchange.  We recommend the second option: have the leader broker notifies the consumer about the location.

The reason we favor the broker/consumer location exchange is because:

  1. __remote_log_metaddata topic is a congested topic in tiered storage which contains metadata for all the tiered storage enabled topic partitions and the client also needs to be aware that each metadata needs to go through several state changes.  The topic includes the metadata history of all the remote log segment from day 1 even if the segment file itself is already removed from tiered storage; There are lot of reading/filtering and caching needs to happen on the consumer side;
  2. With many programming languages to support on the consumer side and given the trend of moving complex consumer logic to the broker side, we feel it is more natural to have the broker to keep track of the remote tiered storage segment file and exchange the info with the consumer.  This is similar to the design choice (whether consumer or the broker keep tracks of preferred read replica) during KIP-392: Allow consumers to fetch from closest replica design where it also favors to have the broker exchange the preferred read replica information with the consumer.

We propose to extend the Fetch API to convey the remote log segment location information.

The consumer client will initiate the FetchRequest with an extra boolean parameter: RemoteLogSegmentLocationRequested which indicates the client is able to read from remote storage and is interested to get the location of remote log segment file on remote storage. When the leader broker receives FetchRequest from the consumer and it finds out it no longer has the requested fetch offset in the local storage, it can respond with an empty Records and with the extra information about the location of the remote log segment.  This is very similar to how KIP-392 responds with empty Records and preferred read replica in FetchResponse to the consumer.

Authorization check

The broker will still perform the standard authorization check against the fetch request.

Fetch and Read Remote Log segment file

Once the consumer is aware of the location of the remote log segment file, it can initiate the read from remote tiered storage and parses the content in log segment file and converts the message into ConsumerRecord objects;  We will address them in the follow-up KIP since this work is dependent on the programming language choice on the consumer side.

Public Interfaces

Protocol Changes

We will extend the FetchRequest API by adding a new field RemoteLogSegmentLocationRequested to indicate the client has the capability to read from remote storage and is interested in getting the location of remote log segment file on remote storage.

 


FetchRequest
{
  "validVersions": "0-18",
  "fields": [
    { "name": "RackId", ... }
    { "name": "RemoteLogSegmentLocationRequested", "type":  "bool", "versions": "18+", "default": "false", "ignorable": true,
      "about": "Indicate whether the client is able to read from remote storage and interested in getting the location of remote log segment file"}
  ]
}

 

We will extend the FetchResponse API in order to convey the location of the remote log segment file to the consumer client.


FetchResponse
{
  "validVersions": "0-18",
  "fields": [
    { "name": "Responses", "type": "[]FetchableTopicResponse", "versions": "0+",
      { "name": "Partitions", "type": "[]PartitionData", "versions": "0+",
        { "name": "PreferredReadReplica", ... },
        { "name": "RemoteLogSegmentId", "type": "uuid", "versions": "18+", "ignorable": true,
          "about": "The uuid of the remote log segment id"},
        { "name": "RemoteLogSegmentCustomMetadata", "type": "bytes", "versions": "18+", "ignorable": true,
          "about": "The byte[] of the custom metadata of the remote log segment metadata"},
      ]}
    ]},
  ]
}

Two new fields are added to FetchResponse.Partitions message structure:

These two fields should be sufficient to resolve the physical file path on remote tiered storage.


Compatibility, Deprecation, and Migration Plan

The change is backwards compatible with previous versions.  The new parameter RemoteLogSegmentLocationRequested in consumer FetchRequest is false by default to not to change the FetchResponse behavior. And If the receiving broker does not support this feature, it will resort to the old behavior of fetching old messages from remote tiered storage and sending them back to the consumer client.

Rejected Alternatives

  • Having the consumer subscribes the tiered storage metadata topic: __remote_log_metadata, filters to the partition it interests in, parses each metadata message and tracks the state change;  We don’t recommend this approach because:
    1. __remote_log_metaddata topic is a congested topic in tiered storage which contains metadata for all the tiered storage enabled topic partitions and each metadata needs to go through several state changes.  The topic includes the metadata history of all the remote log segment from day 1 even if the segment file itself is already removed from tiered storage; There are lot of reading/filtering and caching needs to happen on the consumer side;
    2. With many programming languages to support on the consumer side and given the trend of moving complex consumer logic to the broker side, we feel it is more natural to have the broker to keep track of the remote tiered storage segment file and exchange the info with the consumer.  This is similar to the design choice (whether consumer or the broker keep tracks of preferred read replica) during KIP-392: Allow consumers to fetch from closest replica design where it also favors to have the broker exchange the preferred read replica information with the consumer.


  • No labels