Page History

...

Apache Kafka is widely adopted for building distributed event streaming platforms. As Kafka clusters grow in size and usage, a common challenge arises: different teams or workloads often share the same physical Kafka cluster, which can cause operational and organizational friction. To solve this problem, several solutions exist today but they have some drawabacks. Organizationally the first step is often when teams adopt Kafka and run their own installations. In this scenario they're completely in control of the resources (cpu, network, disk utilization), however from the organization's point of view this is the most expensive form of adoption as many capacity is left unused: often the smallest units of clusters are 3 node clusters to satisfy the most common replication factor requirements but for smaller use-cases a 3 broker environment is already overprovisioned. Of course with larger installations the overprovisioning may not be so apparent. A more economic solution is when the organization manages Kafka clusters together in a virtual environment. This is more efficient as the hardware is often utilized better, however the hardware is often managed by a different (often central) team while the teams are in control of the clusters themselves. Updating hardware can be problematic as maybe multiple teams have to be involved. It is also harder to grow the clusters as again, multiple parties need to be involved. The next organizational step is to create a central Kafka installation that is governed by a central team that provides Kafka as a service to other departments. This however takes control from individual teams and puts administrative burden on the central team. Often in this case with the correct setting of ACLs and CreateTopicPolicies a good enough solution can be provided where resource management becomes a bit easier. Virtual clusters aim to help by defining logical clusters where administrators can be promoted to make a Kafka installation self-service.

To summarizeCurrently, Kafka lacks native support for logical separation of tenants (e.g., applications, teams) within a shared cluster. This leads to several challenges:

...

1. Operational Complexity: Administrators manually implement workarounds like naming conventions for topics, quotas, and ACLs.
2. Resource Isolation: Shared resources create a risk of noisy neighbor effects, impacting performance and stability. Therefore it is less effective to host multiple clusters for multiple teamsWaste: Dedicated brokers waste hardware resources and more costly.
3. Management Overhead: Multi-tenant management often involves custom tools or external systems.

...

1. .

Goals

• Introduce Virtual Clusters as a logical abstraction within a physical Kafka cluster.
• Enable resource and metadata isolation between Virtual Clusters.
• Simplify multi-tenant management and improve operational efficiency.
• Allow administrators to define and manage VCs without significant overhead.
• Ensure backward compatibility and minimal disruption for existing deployments.
• Provide a transparent view of the cluster for the clients.

...