...

• Binary log format

  • No changes expected

• The network protocol and api behavior

  • No changes expected

• Any class in the public packages under clientsConfiguration, especially client configuration

  • org/apache/kafka/common/serialization

    • No changes expected

  • org/apache/kafka/common

    • No changes expected

  • org/apache/kafka/common/errors

    • InvalidNamespaceException will be added.

  • org/apache/kafka/clients/producer

    • No changes expected

  • org/apache/kafka/clients/consumer (eventually, once stable)

    • No changes expected

• Monitoring

• Command line tools and arguments

  • Add create, list and delete for namespaces to kafka-topics and AdminUtils.

  • An optional "namespace" argument will be added to kafka-topics, and kafka-configs. If namespace argument is not provided, a default namespace of "" will be used. This will help in keeping the current behavior of Kafka and cli tools intact.

• Anything else that will likely break existing users in some way when they upgrade
  • To avoid changing request/ response formats and public APIs, this KIP proposes to append namespaces to topic names. However, some separator is required to be able to parse namespace and topic name from a topic name. We need an out of band character (or string) to make sure that we are not breaking any existing topic. Kafka does not disallow any characters as of now for topic names. That makes it hard to come up with some character (or string) that is guaranteed to be not present in any of existing topics. We suggest to make this demarcation string configurable at cluster level, with a default of "::". However, if the users end up setting this to a string that appears in some existing topics, then this will lead to unexpected behavior.None

Proposed Changes

After considering a few approaches, listed in Rejected Alternatives section, below is what we think is the least obtrusive approach to support namespaces in Kafka. We suggest to represent namespaces at storage layer, i.e., storage layout of Zookeeper entities and logs on disk. Internal and public APIs can pass around namespaces, as part of, prepended to, topic names. However, we need someway to separate namespace and topic. This can be done by using a delimiter character that is not allowed in Kafka topics. Kafka allows a topic name to contain characters only from [a-zA-Z0-9\\._\\-]. We suggest to have ":" as the delimiting char, but can be any of the following.

...

1. <namespace>:<topic>
2. <namespace>%<topic>
3. <namespace>@<topic>
4. <namespace>|<topic>
5. <namespace>#<topic>
6. <namespace>*<topic>
7. <namespace>~<topic>
8. <namespace>$<topic>
9. <namespace>^<topic>
10. <namespace>&<topic>
11. <namespace>><topic>

Namespace Name Restrictions

...

Multi-tiered Namespaces

Namespace can have any char in [a-zA-Z0-9\\.]. The "." in namespaces will be used to separate tiers. For instance, a namespace "org.apache.kafka" will translate to "org/apache/kafka". 

Create

List

Delete

Changes in Storage Layouts

Following are the changes in layout structures on ZK and disks.

...