...

This KIP aims to improve monitoring by proposing a new metric group AtMinIsr, which consists of partitions that only have the minimum number of insync replicas remaining in the ISR set (as configured by "min.insync.replicas"). This new metric can be used as a warning since any partitions that are AtMinIsr are at danger of causing at least producer unavailability (for acks=ALL producers) if one more replica drops out of the ISR set.

We can first objectively define UnderReplicated and AtMinIsr that applies in all scenarios:

UnderReplicated: A partition in which the isr_set_size is not equal to the replica_set_size (isr_set_size can be bigger or smaller than replica_replication_factor)
AtMinIsr: A partition in which the isr_set_size is equal to the min_isr_size, which also means a partition in which 1 more drop in isr_set_size will lead to at least producer (acks=ALL) failure

Examples

Let's first take a look at how UnderReplicated and AtMinIsr change depending on different configurations. All examples below look at 1 partition.

Example 1

...

(min_isr_size = replica_replication_factor - 1):

replica_replication_factor = 3

min_isr_size

...

1 partition

minIsrCount = 2

ISR set = [0,1,2]

1. Broker 0 fails

• ISR set = [1,2]
• partition is UnderReplicatedPartition UnderReplicated and AtMinIsr

2. Broker 1 fails

• ISR set = [2]
• partition is UnderReplicatedPartition and UnderMinIsr

In this example, AtMinIsr triggers when UnderReplicatedPartition triggers and tells us that 1 more failure will cause producers with ack=ALL to be unavailablehas the same outcome as UnderReplicated.

Example 2 (min.insync.replicas = 1 AND replica_replication_factor > 2): 

1 partition

replica_replication_factor = 3

min_isr_size minIsrCount = 1

ISR set = [0,1,2]

1. Broker 0 fails

• ISR set = [1,2]
• partition is UnderReplicatedPartitionUnderReplicated

2. Broker 1 fails

• ISR set = [2]
• partition is UnderReplicatedPartition UnderReplicated and AtMinIsr

3. Broker 2 fails

...

In this example, AtMinIsr triggers when there is only 1 insync replica remaining, and tells us that 1 more failure will cause producers with ack=ALL to be unavailable (the whole partition will be unavailable in this scenario).

Example 3 (replication-factor - min.insync.replicas > 1):

replica_replication_factor = 4

min_isr_size = 2

ISR set = [0,1,2,3]

1. Broker 0 fails

• ISR set = [1,2,3]
• partition is UnderReplicated

2. Broker 1 fails

• ISR set = [2,3]
• partition is UnderReplicated and AtMinIsr

3. Broker 2 fails

• ISR set = [3]
• partition is UnderMinIsr

In this example, AtMinIsr triggers when there is the min_isr_size remaining in the ISR set, while UnderReplicated triggers after the first failure.

Advantages Over UnderReplicatedPartition Metric

In some scenarios such as Example 1 where min_isr_size = replica_replication_factor - 1, the AtMinIsr metric is the exact same as the UnderReplicated metric.

However, here are a few scenarios in which AtMinIsr provides an improvement over UnderReplicated:

(1) Repartitioning

When an admin triggers a repartitioning, the ISR set is first expanded from [old_set] to [old_set + new_set], and then reduced to just the [new_set]. In this case, UnderReplicated will be non-zero even when the ISR set is [old_set + new_set]. AtMinIsr will not be non-zero during [old_set + new_set] step unless something goes wrong during repartitioning and replicas are failing to fetch (reducing the isr_set_size to min_isr_size), but we want to know if this happens.

(2) min.insync.replicas = 1

The default value for this configuration is 1, and users can change this to provide higher durability guarantees. In the default scenario where min.insync.replicas = 1 and replication-factor = 3, the AtMinIsr metric will be non-zero when isr_set_size = 1, which tells us that 1 more drop in this set will lead to a completely unavailable partition. This is very powerful for users that have min.insync.replicas = 1 and replication-factor > 2.

(3) replication-factor - min.insync.replicas > 1

Kafka is built to be fault-tolerant, so we ideally want to be able to tolerate more than 1 failure which means we want the difference between replication-factor and min.insync.replicas to be > 1. If it is equal to 1, then we can only tolerate 1 failure otherwise acks=ALL producers will fail.

We generally want isr_set_size to equal replica_replication_factor to have the best guarantees, but this is not always possible for all Kafka users depending on their environment and resources. In some situations, we can allow the isr_set_size to be reduced, especially if we can tolerate more than 1 failure (replication-factor - min.insync.replicas > 1). The only requirement is that the isr_set_size must be at least min_isr_size otherwise acks=ALL producers will fail.

One example is if we have a cluster with massive load and we do not want to trigger a repartition to make isr_set_size = replica_replication_factor unless absolutely necessary as repartitioning introduces additional load which can impact clients. Maybe we also expect the failed broker to be restored soon so we don't want to do anything unless absolutely necessary. In these scenarios, the AtMinIsr metric will tell us when we absolutely need to consider repartitioning or some other action to restore the health of the cluster (false negative is still possible but it tells us that we could not tolerate any more failure at the time it was non-zero if we do not want acks=ALL producers to fail).

In our Kafka environment, we do not even have alerts configured for UnderReplicated as it is too noisy for us and we can tolerate some failures. We run a periodic job to perform the same functionality as AtMinIsr, but it would be better to have it as a metric so we can configure an alert on it.

Usage

A potential usage of this new AtMinIsr category is:

1. Set up an alert to trigger when AtMinIsr > 0 for a period of time
2. If the alert is triggered, then assess the health of the cluster:
   1. If there is an ongoing maintenance, then no action is needed
   2. Otherwise a broker may be unhealthy. The AtMinIsr partition metric or --at-min-isr-partitions TopicCommand option can be used to determine the list of topics to repartition if the unhealthy broker(s) cannot be fixed quickly

NOTE: Alerts on OfflinePartition and UnderMinIsr should still be configured as AtMinIsr may not be triggered if the ISR set state goes directly to OfflinePartition/UnderMinIsr.

AtMinIsr Values + Possible Explanations

...