THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Binary log format
No changes expected
The network protocol and api behavior
No changes expected
Any class in the public packages under clientsConfiguration, especially client configuration
org/apache/kafka/common/serialization
No changes expected
org/apache/kafka/common
No changes expected
org/apache/kafka/common/errors
InvalidNamespaceException will be added.
org/apache/kafka/clients/producer
No changes expected
org/apache/kafka/clients/consumer (eventually, once stable)
No changes expected
Monitoring
Command line tools and arguments
Add create, list and delete for namespaces to kafka-topics and AdminUtils.
An optional "namespace" argument will be added to kafka-topics, kafka-configs. If namespace argument is An optional {{namespace}} option will be added to existing tools. If namespace is not provided, a default namespace of "" will be used. This will help in keeping the current behavior of Kafka and cli tools intact.
- Anything else that will likely break existing users in some way when they upgrade
Proposed Changes
- To avoid changing request/ response formats and public APIs, this KIP proposes to append namespaces to topic names. However, some separator is required to be able to parse namespace and topic name from a topic name. We need an out of band character (or string) to make sure that we are not breaking any existing topic. Kafka does not disallow any characters as of now for topic names. That makes it hard to come up with some character (or string) that is guaranteed to be not present in any of existing topics. We suggest to make this separating string configurable at cluster level, with a default of "::".
Proposed Changes
After considering a few approaches, listed in Rejected Alternatives section, below is what we think is the least obtrusive approach to support namespaces in Kafka. We suggest to represent namespaces at storage layer, i.e., storage We are proposing following changes in the storage layout of Zookeeper entities and logs on disk. 1. Store topics within hierarchy of namespace.
Internal and public APIs can pass around namespaces, as part of, prepended to, topic names.
Following are the changes in layout structures on ZK and disks.
| Type | Existing layout | Proposed layout |
|---|---|---|
| ZK | /brokers/topics/ |
...
| <topic> | /brokers/topics |
...
| /<namespace>/<topic> | ||
| ZK | /configs/<entity_name>/<entity> | /configs/<namespace>/<entity_name>/<entity> |
| ZK | /admin/delete_topics/<topic> | /admin/delete_topics/<namespace>/<topic> |
| ZK | /kafka_acls/<entity_name>/<entity> | /kafka_acls/<namespace>/<entity_name>/<entity> |
| Disk | /<log_dir>/ |
...
| <topic>_<partition> | /<log_dir>/ |
...
| namespace/<topic>_<partition> |
By default, namespace will be empty string. All existing entities will be part of default namespace and the current storage layouts will be in accordance with the proposed storage layouts.
Pros:
- Will not change existing behavior. No impact on upgrades.
Cons:
- Namespace needs to be parsed out of topic names and that requires some separating string or character. We suggest to make this string configurable at cluster level with "::" as default.
- Longer topic names in requests and responses
...
- .
Compatibility, Deprecation, and Migration Plan
...
- Just prepend namespace to topic names
- Manage namespaces separately
- Change the world
- Add namespace to session object