...

Uploading log segments onto S3 is a tradeoff, faster shorter upload will reduce the latency but will pay for the cost of more uploads and smaller log segments.  To achieve a reasonable transfer cost and file size, we would need to combine the log segments from multiple topic partitions and increase the batch size of the upload (and thus increase the latency).

AZ Availability

In the proposed design, we propose using 1 leader in AZ1, 1 follower in AZ2 and 1 S3E1Z bucket in AZ1.  The current design co-locate the leader and S3E1Z bucket in the same AZ to have a shorter upload latency.  This design will cover 2 AZs but with 5 replicas (S3E1Z is also internally multi-way replicated in the same AZ), although we didn't cover AZ3 we feel this setup has a very strong data redundancy (with 5 replicas).  If the user worries about total AZ failure in both AZ1 and AZ2, they can set up another follower in AZ3 to have complete 3 AZ coverage, note even with 3 brokers they can still get the the across-AZ traffic free.  The alternative design is putting the S3E1Z bucket in AZ3 to cover 3 AZs but this would introduce longer latency during message replication.

Data Durability

In acks=all flow, we maintain the same data durability as today by requiring the data gets to the follower before we can acknowledge back to the producer;

In acks=1 flow, we have similar data durability as today by waiting to update the high watermark until the data arrives on the follower broker and the consumer won't be able to read the new message until the high watermark moves passing the new record.  There is an extra latency for the message flows from object storage to the follower broker (in the order of 10ms for bigger batches), this extra latency would mean the consumer needs to wait longer and it also mean the producer would have a longer window for data loss.  In today's code, we can lose about 10ms' data for the data on the way from the leader to the follower when the leader crashes, in this proposed KIP we can lose maybe 20ms' data for the data on the way from the leader to the object storage and then to the follower broker when the leader crashes.  Although the data loss window doubles but it only happens on leader broker crash which is a rare event, most of the acks=1 data flow can tolerate this occasional data loss.

Public Interfaces

We are proposing to reuse and extend the data structures and constructs introduced in KIP-405 to support active log segments uploading.  Most of the new classes in this KIP follow their counterparts from KIP-405 with the convention of adding WAL (WriteAheadLog) in the classes name or config parameters to indicate they are to support active write-ahead log segments.  

...