...

Let's step back and think again. In the case of replication factor = 3, min.insync.replicas=2, acks=all, we can allow 1 replica doesn't ack, and still have durability guarantee

In the case of replication factor = 3 and all 3 replicas are in-sync replica, if we can achieve high durability with 2 replicas ack, then, could we increase write throughput by just need 2 replicas ack, not 3 of them?

...

This is what we have now for acks=all , we need to wait for the slowest follower acks (i.e. 500ms) before we can respond to the producer.

Image RemovedImage Added

I'd like to propose a new config: acks=min.insync.replicas (i.e. the number of replica acks is decided by the min.insync.replicas config in topic/broker) , we just need to wait for the fastest follower acks (i.e. 100ms) then respond to the producer. The slower follower will catch up later.

...

Topic A, we can tolerate data loseloss, but require high throughput, we set min.insync.replicas=1

Topic B, we don't want data loseloss, we set min.insync.replicas=2

...

• acks=0
  • If set to zero then the producer will not wait for any acknowledgment from the server at all.
• acks=1
  • This will mean the leader will write the record to its local log but will respond without awaiting full acknowledgement from all followers.
• acks=all(-1)
  • This means the leader will wait for the full set of in-sync replicas to acknowledge the record.
• acks=min.insync.replicas(-2)
  • This means the leader will wait for the min.insync.replicas number of replicas to acknowledge the record. The min.insync.replicas config can be defined in broker or topic (if overridden in topic level)

Note: The default value for acks  is still all (-1) as before.

Proposed Changes

When the producer sets acks=min.insync.replicas and send a record to the broker, the broker will do the same thing as acks=all did, firstly check if current number of in-sync replicas is greater or equal to min.insync.replicas value, if not, throw NotEnoughReplicasException. If yes, we wait for the number of min.insync.replicas acknowledge, and then respond to the producers.

...

This KIP introduces a new acks value and not default value change, it won't have backward compatibility issue for any existing applications.

...