Page History

...

As of now, there's no possibility to ensure that standby tasks are distributed in different Kafka Streams racks.

Moreover, It It would be desirable to allow custom logic for the standby task distribution in cases when there're s more complex identification of rack IDs racks (e.g rack ID is concatenation of different clusters + availability zone (in AWS's terms)) and some additional parsing and processing is needed before deciding the desired standby task distribution over different racks.

This KIP aims to provide Kafka Streams users the ability to assign rack ID to each Kafka Streams instance and use that information to assign the standby tasks in different racks (by implementing custom logic if necessary) 

...

SubscriptionInfoData protocol will get a version bump as well as the new field that will encode rack ID of the Kafka Streams instance. Encoded rack ID will later be used by RackStandbyTaskAssignor implementation to distribute standby tasks over different racks (which is called by TaskAssignor implementation).

{
  "name": "SubscriptionInfoData",
  "validVersions": "1-10", // NEW: version bump to 10
  "fields": [
    ...
    {
      "name": "rackId", // NEW
      "versions": "10+",
      "type": "bytes"
    }
  ],
  ...
}

Changes in StreamsConfig

In order to give Kafka Stream's user the ability to define their desired standby task distribution, we need to add two new configuration options to StreamsConfig.

...