THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
| Table of Contents |
|---|
Kafka Replication High-level Design
The purpose of adding replication in Kafka is for stronger durability and higher availability. We want to guarantee that any successfully published message will not be lost and can be consumed, even when there are server failures. Such failures can be caused by machine error, program error, or more commonly, software upgrades. We have the following high-level goals:
...
When a new broker is added, we will automatically move some partitions from existing brokers to the new one. Out goal is to minimize the amount of data movement while maintaining a balanced load on each broker. We use a standalone coordinator process to do the rebalance and the algorithm is given below.
Take brokers offline
We’d also like to support shrinking a cluster by taking existing set of brokers offline using an administrative command like the following.
| Code Block |
|---|
alter cluster remove brokers broker-1
|
This will start the reassignment process for partitions currently hosted on broker-1. Once that is complete, the broker-1 will be taken offline. This command will also delete the state change path for broker-1
Data replication
We’d like to allow a client to choose either asynchronous or synchronous replication. In the former case, a message to be published is acknowledged as soon as it reaches 1 replica. In the latter case, we will make our best effort to make sure that a message is only acknowledged after it reaches multiple replicas. When a client tries to publish a message to a partition of a topic, we need to propagate the message to all replicas. We have to decide:
...
- How can atomicity be guaranteed in the 2nd type of leader failure ?
- How can we avoid the problem of multiple leaders for the same partition at the same time ?
- If the brokers are in multiple racks, how to guarantee that at least one replica goes to a different rack?
Kafka Replication Detailed Design
The following are different proposals of the detailed implementation. In 0.8, the implementation is based on the v3 proposal.