Versions Compared

Old Version 7

changes.mady.by.user Matthias Sax

Saved on

compared with

New Version 8

changes.mady.by.user Matthias Sax

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
titleRebalance Metadata
// Metadata v2

MetadataSubscriptionMetadata => VersionNumber ClientId PreviousAssignedTasks StandbyTasks UserEndpoint
  VersionNumber => int32
  ClientId => int64 int64     // UUID most signification bits + least significant bits
  PreviousAssignedTasks => [TaskId]
  StandbyTasks => [TaskId]
  UserEndpoint => string     // UUID most signification bits + least significant bits
  PreviousAssignedTasks user specified endpoint for Interactive Queries

AssignmentMetadata => VersionNumber AssignedTasks AssignedStandbyTasks GlobalAssignment
  VersionNumber => int32
  AssignedTasks => [TaskId]
  StandbyTasksAssignedStandbyTasks => [TaskId]
  UserEndpointGlobalAssignment => string[Host Port [Topic Partition]]]     // metadata for Interactive Queries

TaskId => GroupId Partition
  GroupId => int32
  Partition => int32
 

Host => string
Port => int32
Topic => string
Partitio => int32


// Metadata v3

MetadataSubscriptionMetadata => VersionNumber SupportedVersionNumber ClientId PreviousAssignedTasks StandbyTasks UserEndpoint
  VersionNumber => int32
  SupportedVersionNumber => int32            // NEW
  ClientId => int64 int64
  PreviousAssignedTasks => numTasks [TaskId]
  StandbyTasks => numTasks [TaskId]
  UserEndpoint => string

 
AssignmentMetadata => VersionNumber SupportedVersionNumber AssignedTasks AssignedStandbyTasks GlobalAssignment
  VersionNumber => int32
  SupportedVersionNumber => int32     // NEW
  AssignedTasks => [TaskId]
  AssignedStandbyTasks => [TaskId]
  GlobalAssignment => [Host Port [Topic Partition]]]

Upgrading to 1.2:

  1. prepare a jar hot swap from old version to 1.2; Kafka Streams need to be configured with upgrade.from="<old.version>"for startup
  2. do a rolling bounce to get the new jar and config in place for each instance
    1. config "upgrade.from" tells the application to send Subscription using old endocing (version 1 or 2), to be compatible with potential version 1 or 2 leader in the group
    2. instances will receive a corresponding version 1 or 2 Assigment in this stage
  3. user prepares a second round of rebalance; this time, the configuration parameter upgrade.from must be removed for new startup
  4. do a second rolling bounce for each instance to get new config (ie, upgrade.from removed)
    1. bounced instance can send a version 3 Subscription as we know that leader understands version 3
    2. the leader sends version 1 or 2 Assigment back as long as at least one version 1 or 2 Subsription is received
    3. if the leader receives only version 3 Subscirption, it send version 3 Assigment and the upgrad is completed

...