You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 5 Next »

Status

Current state: Under Discussion 

Discussion thread: TBD

JIRA:  https://issues.apache.org/jira/browse/KAFKA-5793 and https://issues.apache.org/jira/browse/KAFKA-5794

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

There are currently two situations where the behavior of the producer with idempotence enabled is less than satisfactory:

  1. Currently the OutOfOrderSequence exception may be raised spuriously, for instance, if the producer state was removed on the server due to segments which are older than the retention limit being deleted. We would like the OutOfOrderSequence exception to unequivocally indicate data loss, and hence need to detect and handle these false positives.
  2. There is no graceful way to handle enabling idempotence on the producer, and yet having some topics being on an older (pre 0.11.0) message format. This means that making idempotence the default is impossible, as the upgrade steps would simply not work. Hence we would like to introduce a 'safe' mode for idempotence where it will only be enabled if the underlying topic has the requisite message format. 

Background

The two problems described above are detailed in the following pages, along with proposed solutions.

  1. Kafka Exactly Once - Solving the problem of spurious OutOfOrderSequence errors
  2. Kafka Exactly Once - Dealing with older message formats when idempotence is enabled

Public Interfaces

ProduceRequest

We add an idempotenceLevel field to the produce request to indicate to the broker whether we want 'safe' or 'strict' idempotence. See Kafka Exactly Once - Dealing with older message formats when idempotence is enabled for a description of precisely how this will be used.

// ProduceRequest v4
 
ProduceRequest => TransactionalId
                  RequiredAcks
                  Timeout
				  idempotenceLevel
                  [TopicName [Partition MessageSetSize MessageSet]]
 TransactionalId => nullableString
 RequiredAcks => int16
 Timeout => int32
 Partition => int32
 MessageSetSize => int32
 MessageSet => bytes
 idempotenceLevel => int

ProduceResponse

We add a logStartOffset field to the produce response to help the producer identify when producer state has been lost due to retention time elapsing. See Kafka Exactly Once - Solving the problem of spurious OutOfOrderSequence errors for a precise description of how this will be used.

// ProduceResponse v4
ProduceResponse => [TopicName [Partition ErrorCode Offset Timestamp logStartOffset]]
                   ThrottleTime
 TopicName => string
 Partition => int32
 ErrorCode => int16
 Offset => int64
 Timestamp => int64
 ThrottleTime => int32
 logStartOffset => int64

Producer config changes

We introduce new values for the enable.idempotence configuration: safe, required, off.

Compatibility, Deprecation, and Migration Plan

For the Produce Request/Response updates, we follow the existing conventions for maintaining bacward compatibility. New producers will continue to talk with old brokers using the old versions of the protocol.

The legacy values for `enable.idempotence` will be interpreted as follows by the new producer: true will mean required, false will mean off.

Rejected Alternatives

This KIP contains changes to fix existing problems or clarify existing behavior. As such, there are not too many options for making these improvements within the existing solutions. 

 

 

  • No labels