...

Current state: "Under Discussion"

Discussion thread: will be updated here

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

• Drastically Reduces Topic Proliferation – Eliminates unnecessary topic creation.
• Faster Kafka Connect Cluster Deployment – No waiting for new topic provisioning.
  • Large Enterprises with Multiple Teams Using Kafka Connect
    • Scenario: In large organisations, multiple teams manage different Kafka Connect clusters for various data pipelines.
    • Benefit: Instead of waiting for new internal topics to be provisioned each time a new cluster is deployed, teams can immediately start using pre-existing shared topics, reducing lead time and improving efficiency.
  • Cloud-Native & Kubernetes-Based Deployments
    • Scenario: Many organisations deploy Kafka Connect in containerised environments (e.g., Kubernetes), where clusters are frequently scaled up/down or recreated dynamically.
    • Benefit: Since internal topics are already available, new clusters can spin up instantly, without waiting for topic provisioning or Kafka ACL approvals.
  • How this will help different organisations:
• Lower Operational Load – Reduces disk-intensive cleanup operations.
  • Broker resource utilization is expected to decrease by approximately 20%, primarily due to reduced partition count and metadata overhead. This optimization can enable further cluster downscaling, contributing directly to lower infrastructure costs (e.g., fewer brokers, reduced EBS storage footprint, and lower I/O throughput).
  • Administrative overhead and monitoring complexity are projected to reduce by 30%, due to:
    • Fewer topics to configure, monitor, and apply retention/compaction policies to.
    • Reduced rebalancing operations during cluster scale-in or scale-out events.
    • Minimized data movement and replication workload in failure scenarios (e.g., EBS volume replacement), as fewer partitions are involved.
Considering the current load
• • Simplified Management – Less overhead in monitoring and maintaining internal topics.

Anchor
Why Share Kafka Connect Internal Topics Across Clusters? Why Share Kafka Connect Internal Topics Across Clusters?

Why Share Kafka Connect Internal Topics Across Clusters?

...