...

• Binary log format

  • No changes expected

• The network protocol and api behavior

  • No changes expected

• Any class in the public packages under clientsConfiguration, especially client configuration

  • org/apache/kafka/common/serialization

    • No changes expected

  • org/apache/kafka/common

    • No changes expected

  • org/apache/kafka/common/errors

    • InvalidNamespaceException will be added.

  • org/apache/kafka/clients/producer

    • No changes expected

  • org/apache/kafka/clients/consumer (eventually, once stable)

    • No changes expected

• Monitoring

• Command line tools and arguments

  • Add create, list and delete for namespaces to kafka-topics and AdminUtils.

  • An optional "namespace" argument will be added to kafka-topics, kafka-configs. If namespace argument is not provided, a default namespace of "" will be used. This will help in keeping the current behavior of Kafka and cli tools intact.

• Anything else that will likely break existing users in some way when they upgrade
  • To avoid changing request/ response formats and public APIs, this KIP proposes to append namespaces to topic names. However, some separator is required to be able to parse namespace and topic name from a topic name. We need an out of band character (or string) to make sure that we are not breaking any existing topic. Kafka does not disallow any characters as of now for topic names. That makes it hard to come up with some character (or string) that is guaranteed to be not present in any of existing topics. We suggest to make this separating demarcation string configurable at cluster level, with a default of "::". However, if the users end up setting this to a string that appears in some existing topics, then this will lead to unexpected behavior.

Proposed Changes

After considering a few approaches, listed in Rejected Alternatives section, below is what we think is the least obtrusive approach to support namespaces in Kafka. We suggest to represent namespaces at storage layer, i.e., storage layout of Zookeeper entities and logs on disk. Internal and public APIs can pass around namespaces, as part of, prepended to, topic names.

...

• What impact (if any) will there be on existing users?
  • If the users set up the demarcation string to a string that appears in some existing topics, then this will lead to unexpected behavior.
    
• If we are changing behavior how will we phase out the older behavior?
  • No need to phase out old behavior.
• If we need special migration tools, describe them here.
  • NA
• When will we remove the existing behavior?
  • NA

Rejected Alternatives

Following are a few alternatives that we considered.

1. Just prepend namespace to topic names, inheritance will be tricky and not so intuitive. Will not enable encrypting namespaces with different keys or namespace level management.
2. Manage namespaces separately, this will still have the issue of topic name collisions even if they belong to separate namespaces.
3. Modify request/ response formats to take namespace specifically. Solves the issue of demarcation string required in proposed approach. However, will be backwards incompatible change.
4. Add namespace to session object. Will avoid each request and response to have have namespace with topic name, however this probably is violating separation of concerns.