Page History

...

1. When the replica is not in-sync, i.e. there's a long gap between the log end in the Diskless Coordinator and the local log.
2. When the requested offset is older than the earliest offset available locally.

A direct read from remote storage involves:

1. The broker determines a fetch for a partition cannot be served from local segments
2. The broker queries the Diskless Coordinator for the relevant batch coordinates.
3. The broker gets the data either from the object storage.
4. The broker injects the computed offsets and timestamps into the batches in-memory.
5. The broker constructs and sends the Fetch response to the Consumer.

Depending on the structure of the shared log segment, a single Fetch request could be satisfied by one GET request, or may need multiple requests.

Image Modified

Queues

In contrast to normal Fetch requests, ShareFetch requests can only be served by the share-partition leader, which is co-located with the partition leader and manages internal state, such as record state.

...

This type of solution would be less accessible and portable between operating systems. Kernel-space or user-space drivers need to be implemented and supported for every operating system that Kafka supports. This also requires a specific skill set that may not be widely represented in the present Kafka community.

Coordinator-less approaches

It seems possible to partially or even fully eliminate inter-zone traffic without introducing the centralized Diskless coordinator for batch management. A number of proposals were made, such as:

1. Make partition leaders coordinators of their own partitions (proposed here). Any broker can handle Produce requests. Partition leaders write batch metadata to the log instead of data. Followers replicate the metadata log as usual. Data is consumed from remote storage.
2. Produce requests are handled by partition leaders, but remote storage is the replication medium (proposed in KIP-1176).
3. Produce requests are handled by any broker and remote storage is the replication medium (proposed here).

These approaches fully or partially achieve the goal of inter-zone traffic elimination set by KIP-1150. They lean more towards the classic topic design, while the current KIP takes a more "revolutionary" approach, which brings additional benefits. In this KIP, stateful (coordinator) and stateless (replicas) components are explicitly separated. This allows better flexibility and scalability for current and potential future tasks. For example, stateless replicas can be added and removed from the cluster easily without the need to rebalance data stored on local disks, because data is stored on object storage. Object storages are also normally have better durability than local disks. The potential of the new design could be developed further, e.g. towards using serverless compute available in cloud environments for various cluster tasks. This, however, remains out of scope of this KIP.