Page History

...

1. For each topic present in the subscription for at least one consumer, determine the current lag on each partition for the consumer group
   1. Lag is computed by subtracting the latest offset from the last committed offset for each partition
   2. If the consumer group has not yet committed an offset for a partition, determine the lag based on the auto.offset.reset consumer property, as follows:
      1. If auto.offset.reset=latest, the current lag for the partition is 0
      2. If auto.offset.reset=earliest (or any other value) we assume a current lag equal to the total number of records currently available in that partition (i.e. latest offset - beginning offset)
2. For each topic present in the subscription:
   1. Sort partitions in order of decreasing lag
   2. While there are still unassigned partitions:
      1. Select the unassigned partition with the maximum lag
      2. Assign this partition to the consumer with the minimum total number of assigned partitions
      3. If all consumers have an equal number of partitions assigned, assign the partition to the consumer with the minimum total assigned lag

Obtaining

...

Consumer Group Offsets

A means of obtaining the earliest, latest and last committed offset for each topic partition is required.

...

The prototype targeted Kafka 0.10.2.0.  In Kafka 0.11.0.0 it may be more appropriate to obtain offsets via the new Admin API.

Prototype Implementation

A prototype implementation targeting Kafka 0.10.2.0 is available here.

Example

•  TODO: Lag-aware assignment worked example (Grant Neale)

...