...

This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: Under Discussion

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Kafka and its upstream applications treat internal topics differently from non-internal topics. For example:

...

To solve the pain point, I'm proposing support for clients to register and query their own internal topics. 

Public Interfaces

1. TopicConfig will have a new topic config `internal`, which indicates if the topic is internal or not.
2. KafkaZkClient will have a new method getInternalTopics() which returns a set of internal topic name strings.

Proposed Changes

Server-side:

The static internal topic testing is defined in Topic.java. At server-side, it's dependents are:

...

After this process, all the servers will be aware of the latest set of internal topics and can cache internal topics in MedatadaCache. Thus, that KafkaApi can construct the metadata response with the information of all clients created internal topics by referring MetadataCache. 

Client-side:

To get the internal topic information, instead of using the static internal topic testing, clients can utilize KafkaAdminClients and make a MetadataRequest (ApiKey.METADATA). 

Compatibility, Deprecation, and Migration Plan

There are no compatibility concerns in this KIP.

Rejected Alternatives

1. Specify a naming convention that all internal topic should start with the prefix `_`.
   1. It's hard to make all clients adjust their topic names.
2. Change several public APIs to make the clients pass a flag indicating if the topic is internal or not when it creates a topic. Add a new ZK path such as `topics/internal`.
   1. May require a new flag in TopicCommand