This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: [One of "Under Discussion", "Accepted", "Rejected"]

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Apache Kafka is widely adopted for building distributed event streaming platforms. As Kafka clusters grow in size and usage, a common challenge arises: different teams or workloads often share the same physical Kafka cluster, which can cause operational and organizational friction.

...

This KIP proposes introducing Virtual Clusters (VCs) to Kafka, a mechanism to logically separate workloads while sharing the same physical Kafka cluster. VCs will allow teams to operate independently, improving resource isolation, security, and management capabilities.

Goals

Introduce Virtual Clusters as a logical abstraction within a physical Kafka cluster.
Enable resource and metadata isolation between Virtual Clusters.
Simplify multi-tenant management and improve operational efficiency.
Allow administrators to define and manage VCs without significant overhead.
Ensure backward compatibility and minimal disruption for existing deployments.
Provide a transparent view of the cluster for the clients.

Non-Goals

Introducing hard physical isolation (e.g., dedicated hardware).
Replacing or deprecating existing multi-tenant mechanisms like quotas and ACLs.

High Level Design

After considering a few approaches, listed in the Rejected Alternatives section, below is what we think is the least obtrusive approach to support virtual clusters in Kafka.

At the very highest level, we would like to create a new type of resource called “virtual clusters” and connect other resources like topics and groups and users to them. This would allow the brokers to handle requests transparently for the clients. In addition to this, generalizing the Cluster ACL resource, we could properly form the authorization layer necessary in a backward compatible way.

Linking with Topics

We would form a directional one level tree-like structure where we’d link topics to virtual clusters which would be stored in the __cluster_metadata topic. This solution is more advantageous over other approaches as links can be created and modified much easier than for instance a directory structure that has been proposed originally in KIP-37, allowing more elasticity to manage namespaces and there would be no changes required in the storage layer. Moreover this can align with the proposal for topic renames inKIP-516: Topic Identifiers.

...

Furthermore, links can be created with an alternative name to topics, allowing topic renaming in virtual clusters, but also linking topics between virtual clusters in order to achieve more complex use cases, like sharing topics between virtual clusters to provide readable views, or implement aggregation use-cases.

Creating Topics in Virtual Clusters

Currently topic names are heavily embedded into the protocol, therefore it’s challenging to rely only on topic IDs. We wanted to make our implementation as independent from physical topic names as possible while also respecting the current state of the protocol.

...

Create a topic with a user that isn’t part of a virtual cluster: everything stays the same as it is currently.
Create a topic with a user that is already assigned to a virtual cluster: topics that are created within a virtual cluster would have generated names, where one part continues to be the user-given name while another part is unique. This is needed to avoid conflicting names between virtual clusters. Links would also have names with the user-given topic name being the default, to be able to identify them in a user-friendly way. Also links having names would accommodate use-cases when users share topics between virtual clusters as they can opt to use a different name if they conflict in different virtual clusters.

Deleting Topics

Deleting topics in virtual clusters would simply mean deleting the links. If all links are deleted, then the topic doesn’t belong to any virtual cluster, so administrators would simply need to delete the topic to get rid of it forever. Deleting the physical topic wouldn’t be allowed as long as there are links pointing to it. This is necessary to prevent dangling links.

Users and Clients

One of our assumptions is that clients and users usually belong to the same virtual cluster and would rarely use topics in others. But if they’d do, then virtual cluster admins could link topics across virtual clusters. Therefore it would make sense to bind users and client-ids to virtual clusters and create one-to-many connections, where one virtual cluster could have many users/client-ids associated with, but user/client-ids can only be associated with one virtual cluster. Storing this information in the brokers also opens up the possibility for users/client-ids to act transparently in the virtual cluster. For instance they wouldn’t need to prefix topics they address with the name of the VC, instead they would continue using the same topic name. This is advantageous for backward compatibility reasons as clients can simply be added to virtual clusters without them having to make any protocol or configuration changes.

Consumer Groups

In a similar fashion to users, consumer groups can be linked to virtual clusters as well, so groups in different virtual clusters may have the same local name. Storing the offset of virtual groups would be the same as in the classical case. They would continue using __consumer-offsets. Furthermore, __consumer-offsets should stay as a global internal topic to preserve backward compatibility. When consumers are moved into virtual clusters, their group ID will be prefixed with the name of the virtual cluster and that will be the reference for the __consumer-offsets topic. Since clients don’t use this topic directly but through various APIs (admin APIs, offset commit, list offsets, etc.), they aren’t exposed directly to them and therefore virtual clusters don’t need to expose it towards clients.

Transactions and Idempotency

Transactions should behave similarly to consumer groups. Since transactions are more cross-cutting since there are markers in user topics and in the __consumer-offsets topic as well. Therefore logically it makes sense to handle transactions similarly to consumer groups. This means that transactions will be linked to virtual clusters (to the same as the user creating them) and the transaction ID will be translated to a unique ID on the broker side.

ACLs

The Authorization Model

We would like to extend the current authorization model of Kafka to accommodate the needs of virtual clusters. Kafka already provides a CLUSTER ACL resource but currently it is hardcoded with the “kafka-cluster” value and it’s not possible to specify a different one. By generalizing and elevating it to a proper resource, we could actually be able to enable administrators to create clusters and properly manage ACLs within resources. Since many inter-broker protocols already depend on this resource, we would leave the pre-defined “kafka-cluster” as it is for internal protocols. Furthermore any user or client who has privileges on this resource, would have global access on the whole Kafka cluster for compatibility reasons, so on upgrades, everything stays the same.

...

A cluster-wide admin is defined by being able to give permissions to other admins in the whole cluster. This can be achieved by granting CREATE_ACLS, DELETE_ACLS and DESCRIBE_ACLS on the “kafka-cluster” resource. Lower level administrators in virtual clusters wouldn’t need to be allowed to perform operations on the whole cluster, just in their assigned virtual clusters, therefore they would be granted access to CREATE_ACLS, DELETE_ACLS and DESCRIBE_ACLS on their respective virtual clusters.

Operations

We have to reinterpret some of the actions that can be performed on a CLUSTER resource since we would be able to specify the cluster as well. Generally, with protocols that are meant to be for inter-broker communication, we wouldn’t allow changing them from the “kafka-cluster” cluster, since that would compromise the behavior of the whole cluster.

Alter

On the cluster resource, these are the following protocols that can be used:

Exposed protocols that are allowed for virtual clusters:

WRITE_TXN_MARKERS (27)
CREATE_ACLS (30)
DELETE_ACLS (31)
ALTER_REPLICA_LOG_DIRS (34)
ALTER_USER_SCRAM_CREDENTIALS (51)

Exposed new protocols that are allowed for virtual clusters:

ALTER_VIRTUAL_CLUSTER: Users who are allowed for ALTER_VIRTUAL_CLUSTER on CLUSTER resources, will be able to modify resource connections in a virtual cluster. This means that they can add and remove topic links, associate users and clients with the CLUSTER resource and also disassociate them as described in the protocol section.

Exposed protocols that are not allowed for virtual clusters:

ALTER_PARTITION_REASSIGNMENTS (45): a user will be able to initiate partition reassignments on the physical cluster. We wouldn’t allow this to be applied on a virtual cluster level because this operation is tightly coupled with the physical layout of a cluster and it doesn’t make sense to be controlled from a virtual cluster level.

Internal protocols that are not allowed for virtual clusters:

UPDATE_FEATURES (57)
UNREGISTER_BROKER (64)

AlterConfigs

Exposed protocols that are allowed for virtual clusters:

ALTER_CONFIGS (33)
INCREMENTAL_ALTER_CONFIGS (44)
ALTER_CLIENT_QUOTAS (49)

ClusterAction

Exposed protocols that are allowed for virtual clusters:
- ALTER_CONFIGS (33)
- INCREMENTAL_ALTER_CONFIGS (44)
- ALTER_CLIENT_QUOTAS (49)
- ALTER_PARTITION (56)
- WRITE_TXN_MARKERS (27)
- READ_SHARE_GROUP_STATE (84)
- WRITE_SHARE_GROUP_STATE (85)
- DELETE_SHARE_GROUP_STATE (86)
- READ_SHARE_GROUP_STATE_SUMMARY (87)
Internal protocols that are not allowed for virtual clusters:
- FETCH (1)
- LEADER_AND_ISR (4)
- STOP_REPLICA (5)
- UPDATE_METADATA (6)
- CONTROLLED_SHUTDOWN (7)
- OFFSET_FOR_LEADER_EPOCH (23)
- ELECT_PREFERRED_LEADERS (43)
- VOTE (52)
- BEGIN_QUORUM_EPOCH (53)
- END_QUORUM_EPOCH (54)
- ENVELOPE (58)
- FETCH_SNAPSHOT (59)
- BROKER_REGISTRATION (62)
- BROKER_HEARTBEAT (63)
- ALLOCATE_PRODUCER_IDS (67)
- CONTROLLER_REGISTRATION (70)
- INITIALIZE_SHARE_GROUP_STATE (83)
Exposed protocols that aren't allowed for virtual clusters:
- ASSIGN_REPLICAS_TO_DIRS (73)

Create

METADATA (3)
CREATE_TOPICS (19)

...

CREATE_VIRTUAL_CLUSTER: Users who are allowed for CREATE_VIRTUAL_CLUSTER on CLUSTER resources, will be able to create virtual clusters as described in the protocol section.

Describe

LIST_GROUPS (16)
DESCRIBE_ACLS (29)
DESCRIBE_LOG_DIRS (35)
LIST_PARTITION_REASSIGNMENTS (46)
DESCRIBE_USER_SCRAM_CREDENTIALS (50)
DESCRIBE_QUORUM (55)
DESCRIBE_VIRTUAL_CLUSTER: Users who are allowed for DESCRIBE_VIRTUAL_CLUSTER on CLUSTER resources, will be able to describe a virtual cluster with its user, client and topic associations as described in the protocol section.

List

Exposed new protocols that are allowed for virtual clusters:

LIST_VIRTUAL_CLUSTERS: Users who are allowed for LIST_VIRTUAL_CLUSTERS on CLUSTER resources, will be able to retrieve a list of virtual clusters in the Kafka cluster.

Delete

Exposed new protocols that are allowed for virtual clusters:

DELETE_VIRTUAL_CLUSTER: Users who are allowed for DELETE_VIRTUAL_CLUSTER on CLUSTER resources, will be able to delete a virtual cluster. A virtual cluster can only be deleted when it’s empty and no users, clients and topics are associated with it.

Read

No protocols can be associated with the “Read” action on virtual clusters.

Write

No protocols can be associated with the “Write” action on virtual clusters.

Administrators

By properly setting the virtual cluster’s *_VIRTUAL_CLUSTER ACLs, one should be able to configure users to have an administrator role for a given virtual cluster. A super admin that is responsible for managing the physical cluster will be a user who has the appropriate *_CLUSTER ACLs on the “kafka-cluster” cluster resource. This admin will be able to grant further permissions to lower level administrators.

Quotas

We would like to introduce a new family of quotas, called “virtual-clusters”. These are similar in behavior to other types of quotas and applied from top to bottom in decreasing priority. Quotas outside “virtual-clusters” would be global and applied in every cluster. We wouldn’t allow the scenario for setting quotas for the “kafka-cluster” virtual cluster as that is a reserved name and would introduce ambiguity as it would have the same effect as using the non-virtual-clusters quota.

...

/config/clients/<default>

Kafka Connect

Kafka Connect is a fairly independent framework from core Kafka and lacks some capabilities. On an operational level from Kafka’s point of view, we have connect-offsets, connect-configs and connect-status internal topics. We see no reason against using them in virtual clusters. This makes it possible to host multiple Connect clusters with the same Kafka cluster.

Mirror Maker

Mirror Maker is very similar to Connect as it builds on that. Similarly to connect, one should be able to configure their MM2 both for an existing Connect cluster using VCs, but also in dedicated mode, where MM2 runs its own Connect cluster. We wouldn’t like to add exceptions to MM2-internal topics, so users should be able to run multiple dedicated MM2 clusters on Kafka too.

Kafka Streams

Streams applications could be placed inside virtual clusters. This would mean that topics created by streams would be created in a virtual cluster already. From this perspective, they would fit virtual clusters well.

Migrating to new Virtual Clusters

When users upgrade to a virtual cluster compatible Kafka version, everything stays the same as it was before. There will be no virtual cluster resources created and the only cluster ACL is the “kafka-cluster” (and this is also true for a fresh installation).

Users should first create an administrator account that is a super admin and manage virtual clusters. Then with the appropriate command line tools it can create virtual clusters and set the ACLs as it fits to them. After this, they link their existing topics to these newly created virtual clusters and assign users to virtual clusters. Virtual clusters can co-exist with pre-upgrade topics and other resources, so the migration wouldn’t cause operational downtime.

Migration from Prefixes

One common use-case is when users use prefixes to simulate tenants. A typical example would be a topic called “emea.eu.sales.SalesStream” where “emea.eu.sales” prefix could be a user space for a company’s sales department. Then a prefix ACL could be enforced with a CreateTopicPolicy to control what kind of topics users of the sales department can create.

...

The administrator creates a virtual cluster
Links the topic to that with its prefixed name (emea.eu.sales.SalesStream) and with its relative name (SalesStream).
At this point clients since they aren’t in the virtual cluster, would continue addressing the real topic.
The administrator assigns them to the virtual cluster, they would continue using the same prefixed name but through the virtual cluster.
When the clients are ready, they can be restarted with the relative topic name. From this point onwards, they would stop using the prefixed name.
The link with the prefixed name can be dropped.

Technical Details

Protocol Changes

CREATE_VIRTUAL_CLUSTER

Users who are allowed for CREATE_VIRTUAL_CLUSTER on CLUSTER resources, will be able to create virtual clusters.

...

Code Block

title	CreateVirtualClusterResponse

{
  "apiKey": 91,
  "type": "response",
  "name": "CreateVirtualClustersResponse",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "ThrottleTimeMs", "type": "int32", "versions": "0+", "ignorable": true,
      "about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
    { "name": "VirtualClusters", "type": "[]CreatableVirtualClustersResult", "versions": "0+",
      "about": "Results for each topic we tried to create.", "fields": [
      { "name": "Name", "type": "string", "versions": "0+", "mapKey": true,
        "about": "The name of the virtual cluster." },
      { "name": "ErrorCode", "type": "int16", "versions": "0+",
        "about": "The error code, or 0 if there was no error." },
      { "name": "ErrorMessage", "type": "string", "versions": "1+", "nullableVersions": "0+", "ignorable": true,
        "about": "The error message, or null if there was no error." }
    ]}
  ]
}

ALTER_VIRTUAL_CLUSTER

Users who are allowed for ALTER_VIRTUAL_CLUSTER on CLUSTER resources, will be able to modify resource connections in a virtual cluster. This means that they can add and remove topic links, associate users and clients with the CLUSTER resource and also disassociate them.

Code Block

title	AlterVirtualClusterRequest

{
  "apiKey": 92,
  "type": "request",
  "listeners": ["broker", "controller"],
  "name": "AlterVirtualClustersRequest",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "VirtualCluster", "type": "[]VirtualClusterData", "versions": "0+",
      "about": "The virtual clusters to Alter.", "fields": [
      { "name": "Name", "type": "string", "versions": "0+", "mapKey": true,
        "about": "The name of the virtual cluster." },
      { "name": "Resources", "type": "[]VirtualClusterResource", "versions": "0+",
        "about": "The updates for each resource.", "fields": [
        { "name": "ResourceType", "type": "int8", "versions": "0+", "mapKey": true,
          "about": "The resource type." },
        { "name": "ResourceName", "type": "string", "versions": "0+", "mapKey": true,
          "about": "The resource name." },
        { "name": "ResourceOperation", "type": "int8", "versions": "0+", "mapKey": true,
          "about": "The operation on the resource." }
      ]}
    ]},
    { "name": "timeoutMs", "type": "int32", "versions": "0+", "default": "60000",
      "about": "How long to wait in milliseconds before timing out the request." },
    { "name": "validateOnly", "type": "bool", "versions": "1+", "default": "false", "ignorable": false,
      "about": "If true, check that the virtual clusters can be created as specified, but don't create anything." }
  ]
}

DELETE_VIRTUAL_CLUSTER

Users who are allowed for DELETE_VIRTUAL_CLUSTER on CLUSTER resources, will be able to delete a virtual cluster. A virtual cluster can only be deleted when it’s empty and no users, clients and topics are associated with it.

LIST_VIRTUAL_CLUSTERS

Users who are allowed for LIST_VIRTUAL_CLUSTERS on CLUSTER resources, will be able to retrieve a list of virtual clusters in the Kafka cluster.

DESCRIBE_VIRTUAL_CLUSTER

Users who are allowed for DESCRIBE_VIRTUAL_CLUSTER on CLUSTER resources, will be able to describe a virtual cluster. The listing would include the topics, users, clients, groups and transactions assigned to that virtual cluster.

Public Interfaces

Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.

...

Binary log format
The network protocol and api behavior
Any class in the public packages under clientsConfiguration, especially client configuration
- org/apache/kafka/common/serialization
- org/apache/kafka/common
- org/apache/kafka/common/errors
- org/apache/kafka/clients/producer
- org/apache/kafka/clients/consumer (eventually, once stable)
Monitoring
Command line tools and arguments
Anything else that will likely break existing users in some way when they upgrade

Proposed Changes

Describe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.

Compatibility, Deprecation, and Migration Plan

What impact (if any) will there be on existing users?
If we are changing behavior how will we phase out the older behavior?
If we need special migration tools, describe them here.
When will we remove the existing behavior?

Test Plan

Describe in few sentences how the KIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.

Space shortcuts

Child pages

Page History

Versions Compared

Old Version 1

New Version 2

Key