This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: Under Discussion

Discussion thread:: https://www.mail-archive.com/dev@kafka.apache.org/msg70723.html TBD

JIRA: TDB

Motivation

The AdminClient introduced by KIP-117 has a method which lists all the topics in the cluster.  However, we currently don't have an efficient way to implement this.  The AdminClient can make a MetadataRequest which will return information about all the topics in the cluster.  However, the request will also return:

...

MetadataRequest was designed around the needs of consumers and producers, which need to periodically refresh detailed information about each topic they are producing to or consuming from.  In contrast, the AdminClient often just needs to know about the presence or absence of a topic.  It would be better to have a method which just listed the names of the topics which exist without adding all the other information.

Public Interfaces

We will add a new request for listing the topics in a cluster.

ListTopicsRequest (Version 0) => list_internal
    list_internal => BOOLEAN

ListTopicsResponse (Version 0) => error_code [topic_listing]
    topic_listing => name internal
    name => STRING
    internal => BOOLEAN

...

ListTopicsResponse will not return information about topics that the current user does not have permission to describe.

Proposed Changes

ListTopicsRequest will be implemented by the brokers.  On the client side, AdminClient#listTopics should try to use the new request if available.

Compatibility, Deprecation, and Migration Plan

When communicating with older brokers that do not implement ListTopicsRequest, the AdminClient can fall back to making a MetadataRequest.

Rejected Alternatives

Rather than creating a new request type, we could add flags to MetadataRequest directing it to leave out certain information in its response.  This would result in a smaller MetadataResponse, but not quite as small as the ListTopicsResponse. MetadataResponse has fields that take up space even if they're empty, such as arrays of partitions, leaders, ISRs, and so forth.  It would also add complexity to a request which is already somewhat complex.  It seems better to simply create a new request type.