...

Authors: Henry Cai, Thomas Thornton

Status

Current state: Under Discussion Withdrawn

Discussion thread: here 

JIRA: KAFKA-19225 

...

The proposal of using cloud storage for active log segment can also be implemented on traditional object storage (e.g. S3), you can implement both RemoteStorageManager and RemoteWalStorageManager on the same class.  Although S3 has higher latency than S3E1Z, S3 provides across-AZ durability which might be required for some use cases.  For an implementation using S3 as active log segment storage, you would need to configure the system to upload the segment in lower frequency with higher batch size.

AZ Availability

In the proposed design, we propose using 1 leader a typical setup would involve a leader broker in AZ1, 1 a follower broker in AZ2 and 1 S3E1Z , and an S3 Express One Zone (S3E1Z) bucket in AZ1.  The current design co-locate Co-locating the leader and the S3E1Z bucket in the same AZ to have a shorter minimizes upload latency.  This design will cover 2 AZs but with 5 replicas (S3E1Z is also This configuration provides robust data redundancy across two AZs with what is effectively five replicas of the data (given that S3E1Z is internally multi-way replicated in the same AZ), although we didn't cover AZ3 we feel this setup has a very strong data redundancy (with 5 replicas).  If the user worries about total AZ failure in within its AZ). For users concerned about a simultaneous failure of both AZ1 and AZ2, they can set up another follower in AZ3 to have complete 3 AZ coverage, note even with 3 brokers they can still get the the across-AZ traffic free.  The alternative design is putting the S3E1Z bucket in AZ3 to cover 3 AZs but this would introduce longer latency during message replicationan additional follower can be placed in a third AZ to achieve complete three-AZ coverage, still benefiting from the elimination of cross-AZ replication costs.

Leader Failover and Recovery in an AZ Outage

A critical scenario to consider is the failure of an entire Availability Zone, which would take down both the leader broker and its co-located S3E1Z bucket. In this rare event, the cluster remains available by electing a new leader in a different AZ. Once a new leader (e.g., in AZ2) is elected, replica reconciliation takes place. This is similar to standard Kafka. If followers have more events than the high watermark, they will need to truncate them. Below is an example to illustrate this:

Suppose there is a remote WAL enabled topic with replication factor 3. The current leader of a partition is B1. B1 has received data up to offset 10, written data to S3E1Z up to offset 10, and published metadata up to offset 10. One follower, B2, has replicated up to offset 5, and follower B3, up to offset 7. Suppose B1 dies (e.g., AZ goes down), and B2 is elected as the new leader. B2 overwrites the LEO/high-watermark based on its own LEO, which is 5. B2 will tell its followers to reset their LEO to 5. So B3, which had an LEO of 7, will discard 2 messages and reset 5 to match B2, the new leader. In addition, B2 will also publish metadata DELETE events for any metadata entries it receives that are after its LEO (e.g., for messages 6-10), since they are no longer valid.

In the case that B3 was elected, with the higher end offset of offset 7. It will perform replica reconciliation and discard its data that is greater than the high watermark of 5. This prevents any sacrifices in availability, and is equivalent to standard Kafka.

Data Durability

In acks=all flow, we maintain the same data durability as today by requiring the data gets to the follower before we can acknowledge back to the producer;

...

100 records, 1.6 KB avg record size, max batch size 32kb, acks=1, consumer client.rack=use1-az6 (read from follower)

Metrics

Kafka Producer/Consumer Perf Test Scripts

...

Consumer: --messages 100000 --timeout 240000 --consumer.config consumer.properties
The consumer.properties file contained client.rack=use1-az6 (the rack of the follower broker, so we consumed from the follower).

Metrics

Analysis

Overall we see comparable performance between a standard topic and KIP-1176 for both producer & consumer metrics, in particular throughput and latency. The producer path is near identical performance, i.e., an ack to the producer is returned in the same time. Consumer performance is slightly slower as data must travel an additional hop to/from S3E1Z. There are additional tunings we are aware of that could further boost performance (e.g., tuning the interval that the leader/follower remote WAL tasks run).

...