DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
- Status
- Motivation
- Why Share Kafka Connect Internal Topics Across Clusters?
- Why this will not effect Kafka Connect Startup
- BenchMarking with topic shared across 20 clusters each having 5 Sink jobs:
- Public Interfaces
- Proposed Changes
- Compatibility, Deprecation, and Migration Plan
- Test Plan
- Rejected Alternatives
...
- A single worker-level configuration property will be added for distributed mode:
Name
Type
Default
ImportanceDescription
connect.management.topic.sharing.supportStringdisabledHIGHWhether to enable sharing of Kafka Connect management topic sharing across clusters by using cluster group ID to filter the different states along the connect journey.
To enable the connect management topic sharing on a new cluster, set this property to "enabled"
- Another constructor parameter “clusterGroupId” is added to the following classes:
- KafkaConfigBackingStore
...
We have created a new Transformer to handle all type of the migrations and the consumer need not worry. The migration will be similar to how we migrate in the current world with only the addition of a very light weight transformer handling the cluster transformation logic. The configs are as below:
| Property Name | Purpose | Default |
| source.cluster.names | list of cluster names which are to be migrated, empty in case migrating old client or migrating a particular job. | Empty List |
| destination.cluster.name | destination cluster name to which the migration is to be performed. | No default value as this will be required in all the cases. |
| topic.type | type of the topic which is being migrated | No default value as this is always needed |
| migrate.job | config to control whether only a particluar job is to be migrated | FALSE |
| job.name | name of the job in case a particular job is being migrated | empty string |
| old.client | config to control the migration of an old client to a new client | True, as we assume that migration will be primarily needed for an old client to the new client |
Proposed Changes
As part of this KIP, we are proposing the add the “ClusterGroupId” as prefix to the existing keys. So for ex:
If the current key for “connector status” for connector “A” for Cluster "B” is :
...
- The changes are backward compatible, meaning anyone can freely and smoothly upgrade to the new client to which these changes will be added, without any issues. As the changes are dependent on whether it can extract the “CLUSTER_ID” from the key, in case if it doesn’t it goes into the normal flow.
- A cluster using the older client can still share the management topics with the cluster upgraded to the newer client.
| Kafka Client A | Kafka Client B | Can Share Topic? |
| Old | Old | No |
| Old | New | Yes |
| New | Old | Yes |
| New | New | Yes |
Deprication
- No interfaces or api’s are being deprecated as part of this change.
...