Page History

...

We will introduce a new exception type, which will be raised from KafkaConsumer.poll(Duration) as described above. This exception extends from `OffsetOutOfRangeException` for compatibility. If a user's offset reset policy is set to "none," they will still be able to catch `OffsetOutOfRangeException`. For new usage, users can catch the more specific exception type and use the new `seekToNearest()` API defined below.

class TruncatedPartitionExceptionLogTruncationException extends OffsetOutOfRangeException {
}}

We will also add a new retriable error code for the UNKNOWN_LEADER_EPOCH error code:

class UnknownLeaderEpochException extends RetriableException {}

This will be located in the public `errors` package, but the consumer will internally retry when it receives this error.

The leader epoch will be exposed in the ConsumerRecord and OffsetAndMetadata objects.

...

ListOffsetResponse => [TopicName [Partition Offset LeaderEpoch Timestamp ErrorCode]]
  TopicName => STRING
  Partition => INT32
  Offset => INT64
  LeaderEpoch => INT32  // New
  Timestamp => INT64
  ErrorCode => INT16

Offset Schema Changes

This KIP introduces a new schema version for committed offsets in the internal __consumer_offsets topic. This is not a public API, but we mention it for compatibility implications. The version will be bumped to 3. The new schema is given below:

Offset Commit Value Schema (Version: 3) =>
  Offset => INT64
  LeaderEpoch => INT32
  Metadata => STRING
  CommitTimestamp => INT64

As with previous changes to this schema, we will not begin using the new schema after an upgrade until the inter-broker version has been updated. This ensures that replicas on older versions do not fail if they need to parse the new schema version.

ACL Changes

Currently the OffsetForLeaderEpoch request is restricted to inter-broker communication. It requires authorization to the Cluster resource. As part of this KIP, we will change this to require Describe access on the Topic resource. For backwards compatibility, we will continue to allow the old authorization.

...