Page History

...

Any broker is able to serve Produce and Fetch requests for any Diskless topics. However, in many cases arbitrary selection of the request receiver by the client will not be optimal from the performance point of view. We need a way to let the clients choose brokers to serve their Produce and Fetch requests optimally. Only the partition leader can serve ShareFetch requests. Taking into account these requirements and also ones that other KIPs from the Diskless initiative (e.g. KIP-1164) may have, we propose the following.

We introduce the version 14 of of MetadataRequest and  and MetadataResponse, which will allow the clients to send brokers their rack.id and receive back more information about Diskless topics, such as whether the topic is Diskless (IsDiskless) and what is/are recommended brokers to produce to a particular Diskless partition (PreferredProduceBrokers). The definition of these is provided below in the Public Interfaces section.

Newer clients will use the new API version 14 to send and receive this information. No behavior changes expected regarding classic topics. For Diskless topics, newer clients will do the following:

1. Send Produce requests to recommended produce brokers (in order and subject to availability).
2. Send Fetch requests as for classic topics: either to the leader or to the closest replica based on the rack.
3. Send ShareFetch requests as for classic topics, to the leader.

Older clients will use previous API versions 0-13 and the broker will not be able to tell them whether the topic is Diskless and what are the recommended brokers for producing. To allow Kafka users with older clients and also with third-party clients (which may have their own schedule of adding support for new features) to benefit from Diskless, we will allow them to add “,diskless_rack_id=<rack_id>” to their client ID. The Client ID is always transferred to the broker and the broker will take this information into account. It will respond with the matching version of the MetadataResponse, however, it will modify the true partition metadata for Diskless partitions. If the partition has replicas in the matching rack, the LeaderId will be replaced with one of them. Thus, this client will be able to produce and consume data within the rack and avoid inter-rack network traffic costs. This, however, will prevent ShareFetch requests from being served, because only the real leader can serve them. The way out of this is to specify diskless_rack_id for producers and normal consumers and don’t specify for share consumers. The latter must be separate instances in the code.

...