Page History

...

1. The cloud region has multiple availability zones (AZs).
2. The cluster brokers are uniformly distributed across AZs;.
3. Partition leaders are also distributed across AZs;.
4. Clients, producers particularly, are also distributed across AZs;.
5. Inter-AZ network traffic is not free (true for many cloud providers) and it’s desirable to reduce it.
   

...

If the producers and the partition leaders are more or less uniformly distributed and the target partition is periodically randomly rotated, it is possible to to limit the potential target partitions to a subset of those whose leaders are located in the same AZ as the producer itself. This will drastically reduce the inter-AZ network traffic originating from this producer, opening cost saving. Reduced network latency could potentially be considered as well.

For this to be stable, there are important conditions. The distribution of producers and partition leaders across availability zones should be uniform enough to avoid potential unhealthy imbalances. For example, if there are 2 producers in 2 (of 3 total) AZs and 3 partitions in all AZs, the two producers will produce to their nearby partitions, leaving the third one empty. Producers should generate similar load for the same reason. The degree of this uniformity needed depends heavily on the cluster and its ability to compensate and absorb imbalances. Naturally, this is not always the case and then the proposed algorithm should be avoided (as will be said later, the new logic will be behind configuration). Monitoring for imbalances should also be used.

...