Page History

...

Note, that the currently used rebalance metadata version are 1 (0.10.0.x) and 2 (0.10.1.x, ..., 1.1.x). We increase the metadata version numer number to 3 in 1.2 release.

// Metadata v2

Metadata => VersionNumber ClientId PreviosAssignedTasks StandbyTasks UserEndpoint
  VersionNumber => int32
  ClientId => int64 int64     // UUID most signification bits + least significant bits
  PreviosAssignedTasks => numTasks [TaskId]
  StandbyTasks => numTasks [TaskId]
  UserEndpoint => int32 String     // string length + string


numTasks => int32
TaskId => GroupId Partition
  GroupId => int32
  Partition => 32
 
// Metadata v3





// ProduceRequest v3
 
ProduceRequest => TransactionalId
                  RequiredAcks
                  Timeout
                  [TopicName [Partition MessageSetSize MessageSet]]
 TransactionalId => nullableString
 RequiredAcks => int16
 Timeout => int32
 Partition => int32
 MessageSetSize => int32
 MessageSet => bytes 

Upgrading to 1.2:

1. prepare a jar hot swap from old version to 1.2; Kafka Streams need to be configured with upgrade.from="<old.version>"for startup
2. do a rolling bounce to get the new jar and config in place for each instance
   1. config "upgrade.from" tells the application to send Subscription using old endocing (version 1 or 2), to be compatible with potential version 1 or 2 leader in the group
   2. instances will receive a corresponding version 1 or 2 Assigment in this stage
3. user prepares a second round of rebalance; this time, the configuration parameter upgrade.from must be removed for new startup
4. do a second rolling bounce for each instance to get new config (ie, upgrade.from removed)
   1. bounced instance can send a version 3 Subscription as we know that leader understands version 3
   2. the leader sends version 1 or 2 Assigment back as long as at least one version 1 or 2 Subsription is received
   3. if the leader receives only version 3 Subscirption, it send version 3 Assigment and the upgrad is completed

...