...

The period when the offset is unavailable should be brief.  During this period, the broker should simply return a retriable exception when it is asked for the offset of the partition.  For current versions of ListOffsetsRequest, this exception can be LeaderNotAvailableException.  For new versions of ListOffsetsRequest, we can return a new, more precise exception.  The main advantage of creating a new exception is that the client knows it can avoid re-fetching metadata.  A second advantage is that the more precise error message may help with debugging on the client side.  During this brief time period, we will be able to fetch records from the new leader, but not find the latest offset for the partition.

Public Interfaces

There will be a new version of ListOffsetsResponse API.  This will be the same as the existing one, except that we can return a new exception, OffsetNotAvailableException, for a partition.  This new exception will be a subclass of RetriableException.

Compatibility

The new OffsetNotAvailableException mentioned earlier will be sent only to post-KIP-207 clients.  As mentioned earlier, pre-KIP-207 clients will receive LeaderNotAvailableException.  Therefore, older clients should be able to communicate with servers implementing KIP-207.  Similarly, because the client changes are limited to handling a single additional exception, post-KIP-207 clients can communicate with pre-KIP-207 brokers.

Rejected Alternatives

Rather than returning a retriable exception, the broker could simply put the ListOffsetsRequest into a purgatory structure until the offset was available.  This avoids the need for the client to poll the server.  We would create a new version of the ListOffsetsRequest RPC which adds a maximum timeout field.

...