Versions Compared

Old Version 1

changes.mady.by.user David Arthur

Saved on

compared with

New Version 2

changes.mady.by.user David Arthur

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
ZNode /migration

{
  "lastOffset": 100,
  "lastTimestamp": "2022-01-01T00:00:00.000Z",
  "kraftControllerId": 3000,
  "kraftControllerEpoch": 1
}

...

Controller ZNodes

The two controller ZNodes "/controller" and "/controller_epoch" will be managed by the KRaft quorum during the migration. Rather than using ephemeral ZNodes, the KRaft controller will use a persistent ZNode for "/controller" to prevent ZK brokers from attempting to become the active controller. The "/controller_epoch" ZNode will be managed by the active KRaft controller and incremented anytime a new KRaft controller is elected.

Operational Changes

Forwarding Enabled on Brokers

As detailed in KIP-500 and KIP-590, all brokers (ZK and KRaft) must forward administrative requests such as CreateTopics to the active KRaft controller once the migration has started. When running the new metadata.version defined in this KIP, all brokers will enable forwarding.

Migration Trigger

The migration from ZK to KRaft will be triggered by the cluster's state. To start a migration, the cluster must meet two requirements:

  1. The metadata.version is set to the version added by this KIP. This indicates the software is at a minimum version which includes the necessary logic to perform the migration
  2. All ZK brokers have kafka.metadata.migration.enable set to “true”. This indicates an operator has declared some intention to start the migration

Once these conditions are satisfied, an operator can start a KRaft quorum with kafka.metadata.migration.enable set to “true” to begin the migration.

By utilizing configs and broker/controller restarts, we follow a paradigm that Kafka operators are familiar with.

Migration Overview

Here is a state machine description of the migration. 


State

Description

ZooKeeperMode

The cluster is in ZooKeeper mode

MigrationEligible

The cluster has been upgraded to a minimum software version and has set the necessary static configs

MigrationReady

The KRaft quorum has been started

MigrationActive

ZK state has been migrated, controller is in dual-write mode, brokers are being restarted in KRaft mode

MigrationFinished

All of the brokers have been restarted in KRaft mode, controller still in dual-write mode

KRaftMode

The cluster is in KRaft mode


And a state machine diagram:

Image Added

...

Proposed Changes

Describe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.



Compatibility, Deprecation, and Migration Plan

...