Page History

...

When broker creates topic, broker will increment the global topic_epoch in the znode /topic_epoch by 1 and write this partition epoch in the znode /brokers/topics/[topic] according to its new json format.

2) Topic deletion.

When a topic is deleted, the global topic_epoch in the znode /topic_epoch will be incremented by 1.

3) Metadata propagation from controller to brokers and from brokers to clients.

Controller should include the topic_epoch and leader_epoch for each partition in the UpdateMetadataRequest. Broker should also include topic_epoch and leader_epoch for each partition in the MetadataResponse to clients.

34) Client's metadata refresh

...

Note that producer is interested in all partitions. Consumers can potentially be interested in only partitions that it has explicitly subscribed to. The purpose of checking only a subset of partitions is to avoid unnecessary metadata refresh when the metadata is only outdated for partitions not needed by client. In other words, it is for optimization and it is not needed for correctness.

45) Offset commit

When consumer commits offset, it looks up leader_epoch and topic_epoch in the cached metadata and includes this value in the OffsetCommitRequest. The leader_epoch and topic_epoch will then be written in the message to the offset topic.

...

If the client/broker library is old, both the leader_epoch and the topic_epoch are assumed to be -1 and broker will preserve the existing behavior without doing this additional check.

56) Consumer rebalance or initialization using the offset and epoch from the Kafka offset topic

...

For existing version of the offset topic, leader_epoch and topic_epoch will not be available in the value of the offset topic message. Both the leader_epoch and the topic_epoch are assumed to be -1 and the client will preserve the existing behavior without doing the additional metadata refresh.

67) Consumer initialization if offset is stored externally.

...