Page History

...

The proposal of using cloud storage for active log segment can also be implemented on traditional object storage (e.g. S3), you can implement both RemoteStorageManager and RemoteWalStorageManager on the same class.  Although S3 has higher latency than S3E1Z, S3 provides across-AZ durability which might be required for some use cases.  For an implementation using S3 as active log segment storage, you would need to configure the system to upload the segment in lower frequency with higher batch size.

AZ Availability

In the proposed design, we propose using 1 leader a typical setup would involve a leader broker in AZ1, 1 a follower broker in AZ2 and 1 S3E1Z , and an S3 Express One Zone (S3E1Z) bucket in AZ1.  The current design co-locate Co-locating the leader and the S3E1Z bucket in the same AZ to have a shorter minimizes upload latency.  This design will cover 2 AZs but with 5 replicas (S3E1Z is also This configuration provides robust data redundancy across two AZs with what is effectively five replicas of the data (given that S3E1Z is internally multi-way replicated in the same AZ), although we didn't cover AZ3 we feel this setup has a very strong data redundancy (with 5 replicas).  If the user worries about total AZ failure in within its AZ). For users concerned about a simultaneous failure of both AZ1 and AZ2, they can set up another follower in AZ3 to have complete 3 AZ coverage, note even with 3 brokers they can still get the the across-AZ traffic free.  The alternative design is putting the S3E1Z bucket in AZ3 to cover 3 AZs but this would introduce longer latency during message replicationan additional follower can be placed in a third AZ to achieve complete three-AZ coverage, still benefiting from the elimination of cross-AZ replication costs.

Leader Failover and Recovery in an AZ Outage

A critical scenario to consider is the failure of an entire Availability Zone, which would take down both the leader broker and its co-located S3E1Z bucket. In this rare event, the cluster remains available by electing a new leader in a different AZ.Once a new leader (e.g., in AZ2) is elected, it must ensure that the data previously written by the old leader is available for replication. To achieve this, the new leader republishes its local log segments to a new S3E1Z bucket located in its own AZ (AZ2). There are two strategies for this republishing process:

1. Comprehensive Approach: A safe method where the new leader re-uploads all log segments newer than the topic's local.retention.ms configuration. This ensures all potentially needed active data becomes available in the new location.
2. Optimized Approach: The new leader identifies and re-uploads only the log segments that have not yet been tiered to long-term storage. Followers needing older data can still fetch it directly from the long-term tiered storage, reducing the amount of data that needs to be immediately republished.

After the data is uploaded, the new leader publishes corresponding metadata to the __remote_wal_log_metadata topic. This results in duplicated metadata entries: one set from the old leader pointing to the inaccessible bucket in AZ1, and a new set from the new leader pointing to the new bucket in AZ2.Followers are designed to handle this situation gracefully. When a follower consumes metadata, it will first try to fetch data from the location specified. If it encounters the old metadata from the failed leader, the fetch will fail. The follower will then ignore this stale metadata and proceed to the next entry, which will be the valid metadata from the new leader. This ensures that replication continues correctly despite the temporary appearance of out-of-order metadata.

Data Durability

In acks=all flow, we maintain the same data durability as today by requiring the data gets to the follower before we can acknowledge back to the producer;

...

100 records, 1.6 KB avg record size, max batch size 32kb, acks=1, consumer client.rack=use1-az6 (read from follower)

Metrics

Kafka Producer/Consumer Perf Test Scripts

...

Consumer: --messages 100000 --timeout 240000 --consumer.config consumer.properties
The consumer.properties file contained client.rack=use1-az6 (the rack of the follower broker, so we consumed from the follower).

Metrics

Analysis

Overall we see comparable performance between a standard topic and KIP-1176 for both producer & consumer metrics, in particular throughput and latency. The producer path is near identical performance, i.e., an ack to the producer is returned in the same time. Consumer performance is slightly slower as data must travel an additional hop to/from S3E1Z. There are additional tunings we are aware of that could further boost performance (e.g., tuning the interval that the leader/follower remote WAL tasks run).

...