You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

  • Motivation: 

Until Kafka 0.8.2, if a producer wanted to make sure a message was persisted at a specific number of replicas (N) before an "ack" was returned from the broker, they would specify request.required.acks=N. This was a bit misleading since specifying request.required.acks=2 would not actually protect against data loss in all cases.

In Kafka 0.8.2 we added min.isr feature which is a less misleading implementation of a similar behavior - users can specify request.required.acks=-1 (ack after writing to all ISR) and also specify minimum size of ISR to guarantee a minimum number of guaranteed copies.

In this KIP we propose to remove support of  request.required.acks > 1 and return an error message to producers which request that. 

  • Proposed Change: 

ProducerRequest protocol version will be bumped to 1.

We will add a new error to the client protocol: InvalidRequiredAcksException (error_code = 21)

If ProduceRequest version = 1, we will check request.required.acks of the request and return InvalidRequiredAcksException if request.required.acks > 1

If ProduceRequest version = 0, we will maintain current behavior.

  • New or Changed Public Interfaces: 

We will stop accepting request.required.acks > 1 from clients implementing the new version of the protocol and may return a new error code.

  • Migration Plan and Compatibility: 

Users upgrading to clients that implement new version of protocol will need to change their configuration and avoid request.required.acks >1. 

  • Deprecation:

We will support existing behavior and version 0 of the protocol for 0.8.X versions. I suggest deprecating for 0.9.

  • Rejected Alternatives: 

Instead of bumping protocol version and adding an error, silently replace request.required.acks>1 with request.required.acks=-1. Rejected as possibly very confusing to users who will see higher latency (and possibly lower reliability) with no explanation.

  • Questions that may need more debate:
    • When do we want to deprecate the existing behavior?
    • Is the error message of the right granularity? We can be more generic and use InvalidRequestParameters instead. It will allow using it for additional validations, but makes troubleshooting on client more complex.
  • No labels