Page History

...

These cloud storage (S3E1Z or EBS) are internally multi-way replicated, once the kafka log data is uploaded into these cloud storage, there is not much need to replicate further into other Kafka follower brokers.  Therefore in theory, you can reduce the replication factor of a Kafka topic to 1 to save the cost of 2 Kafka brokers.  In practice and for the initial version of this KIP, you might want to set up one or more Kafka follower brokers as a hot-standby or serve as an extra read replica for the downstream consumers in the same AZ (as the follower broker) to save across AZ traffic cost for downstream consumers.  But the replication from cloud storage to those extra Kafka follower brokers are free.  For a future enhancement of this KIP, we can support a mode which ends the replication when the data is uploaded to object storage and elect a new leader from any brokers when the old leader crashes. The new leader would need to sync up with the object storage for the last few seconds of data when it starts up as the new leader (and read the rest of the data async in the background).

Note although this proposal is promoting to use fast cloud storage (S3E1Z or EBS) to store active log segment, the interfaces proposed in this proposal can also be easily implemented on a traditional cloud storage (e.g. S3) with a high latency.

Key Classes and Constructs

...

Uploading log segments onto S3 is a tradeoff, faster shorter upload will reduce the latency but will pay for the cost of more uploads and smaller log segments.  To achieve a reasonable transfer cost and file size, we would need to combine the log segments from multiple topic partitions and increase the batch size of the upload (and thus increase the latency).

Using a traditional cloud storage (e.g. S3) for active log segments

The proposal of using cloud storage for active log segment can also be implemented on traditional object storage (e.g. S3), you can implement both RemoteStorageManager and RemoteWalStorageManager on the same class.  Although S3 has higher latency than S3E1Z, S3 provides across-AZ durability which might be required for some use cases.  For an implementation using S3 as active log segment storage, you would need to configure the system to upload the segment in lower frequency with higher batch size.

AZ Availability

In the proposed design, we propose using 1 leader in AZ1, 1 follower in AZ2 and 1 S3E1Z bucket in AZ1.  The current design co-locate the leader and S3E1Z bucket in the same AZ to have a shorter upload latency.  This design will cover 2 AZs but with 5 replicas (S3E1Z is also internally multi-way replicated in the same AZ), although we didn't cover AZ3 we feel this setup has a very strong data redundancy (with 5 replicas).  If the user worries about total AZ failure in both AZ1 and AZ2, they can set up another follower in AZ3 to have complete 3 AZ coverage, note even with 3 brokers they can still get the the across-AZ traffic free.  The alternative design is putting the S3E1Z bucket in AZ3 to cover 3 AZs but this would introduce longer latency during message replication.

...