You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Status

Current state: Under Discussion

Discussion thread: here[TODO]

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Each partition in broker maintains a list of the 5 most recent appended sequence numbers and corresponding offsets for each producerId (a.k.a PID). If a producer fails to receive a successful response and retries the produce request, broker can still return the offset of the successful append from the maintained cache via RecordMetadata. The maintained cache per producerId per partition in broker instance consumes lots of memory which causes OOM in production. This KIP aims at reducing memory usage for producer state and the complexity to manage it on the broker side.

Public Interfaces

None

Proposed Changes

The broker will only maintain the latest sequence/offset for each producerId instead of 5 recent appended sequence/offset list. The broker guarantees that any sequence number equal or less than the latest sequence will be durable.

When the sequence number reaches Int.MaxValue, client can wraparound starting from 0 again.

On broker side, 

  • Brokers will return DUPLICATE_SEQUENCE_NUMBER for any sequence that is within 1000 of the latest sequence number (accounting for overflow). In this case, we won't return offset of the appended batch.

  • Brokers will return OUT_OF_ORDER_SEQUENCE for any sequence that is outside 1000 of the latest sequence number.

Note: 1000 is an arbitrary number, we can pick one that makes sense.

On Client side,

  • When clients receive DUPLICATE_SEQUENCE_NUMBER, it will discard it and move on.

  • When clients receive OUT_OF_ORDER_SEQUENCE, it will handle as before - retry the send request.

Compatibility, Deprecation, and Migration Plan

With this proposal, restriction on max.in.flight.requests.per.connection can be removed. broker won't provide offset along with a DUPLICATE_SEQUENCE_NUMBER error response unless it sets max.in.flight.requests.per.connection to 1.

This feature may need document that the user must set this config parameter to 1 if the offset is required in `RecordMetadata`.

For old version of broker without this feature, it can restrict the number of inflight requests to 5 internally.

Mechanism may be needed to tell whether a broker supports the new duplicate detection logic, like bumping the Produce API version so that a client can tell whether the broker it talks to is an old version (a.k.a. limit to 5 inflight sequence number) upon communication.

Rejected Alternatives

There is no rejected alternatives. However, we thought about bumping producer epoch to solve the sequence ID wraparound issue so that we don't need to take an arbitrary number like 1000. There are couple of concerns with this approach:

  1. transactions with sequence number just across the Int.MaxValue will be aborted for new epoch bump up. This should be fine since we only abort at most one transaction per epoch life time.
  2. Sequence number is a concept within topic partition. If a producer serves tons of topic partitions, the chance of sequence number wraparound for every topic partition will increase. In this case we may see relatively higher frequency of epoch bump and transaction abort. We need some estimation before jumping into this approach.

Based on above concerns, we decided to start with the default wraparound solution.

  • No labels