Page History

...

Fetch requests follow a long poll model so they can be made to block for a period of time if sufficient data is not immediately available.

One thing to note is that the fetch API requires specifying the partition to consume from. The question is how should a consumer know what partitions to consume from? In particular how can you balance the partitions over a set of consumers acting as a group so that each consumer gets a subset of partitions. We have done this assignment dynamically using zookeeper for the scala and java client. The downside of this approach is that it requires a fairly fat client and a zookeeper connection. We haven't yet created a Kafka API to allow this functionality to be moved to the server side and accessed more conveniently. A simple consumer client can be implemented by simply requiring that the partitions be specified in config, though this will not allow dynamic reassignment of partitions should that consumer fail. We hope to address this gap in the next major release.

FetchRequest => ReplicaId MaxWaitTime MinBytes [TopicFetchRequest]


ReplicaId => int32
MaxWaitTime => int32
MinBytes => int32
TopicFetchRequest => TopicName [PartitionFetchRequest]
TopicName => string
PartitionFetchResponse => Partition FetchOffset MaxBytes
FetchOffset => int64


FetchResponse => [TopicData]
TopicData => Partition [PartitionData]
PartitionData => ErrorCode FetchedOffset HighwaterMarkOffset MessageSetSize MessageSet
ErrorCode => int16
FetchedOffset => uint64
HighwaterMarkOffset => int64
MessageSetSize => int32

...