...

Discussion thread: here

JIRA: here 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Folks Users of Kafka have created dozens of different systems to work with Kafka. If we can provide Providing a wire protocol that allows the brokers to execute administrative code then clients can more simply execute commands. We have a lot of ties with zookeeper making it near impossible for some code to-do nothing better/more than shell script wrapping kafka-topics.sh etc. With the wire protocol we should be able to have client in any language work with administrating Kafka. If someone wants a REST interface (for example) then can write that in whatever language they like. We should have a client from the project that is not only an example but a fully functionality replacement for the kafka-topics, reassign-partitions, consumer offset tools.

Public Interfaces

• Changes to Wire Protocol:

 -   Extending protocol with these messages: CreateTopic(Request | Response), AlterTopic, DeleteTopic

 code and public api/client has many benefits including:

• Allows clients in any language to administrate Kafka
  • Wire protocol is supported by any language
• Provides public client for performing admin operations
  • Ensures integration test code in other projects and clients maintains compatibility
  • Prevents users from needing to use the Command classes and work around standard output and system exits
• Removing the need for admin scripts (kafka-topics.sh, kafka-acls.sh, etc) to talk directly to Zookeeper. 
  • Allows ZNodes to be completely locked down via ACLs
  • Further hides the Zookeeper details of Kafka

Public Interfaces

• Changes to Wire Protocol:
  • Adds the following new Request/Response messages:
    • CreateTopic
    • AlterTopic
    • DeleteTopic
    • ListAcls
    • AlterAcls
    • DescribeConfig
    • AlterConfig
  • Modifies Metadata Response to allowing polling for in-progress or complete admin operations. Added fields include:
    • Boolean showing if a topic is marked for deletion
    • Rack information (if not added by KIP-36 Rack aware replica assignment)

• New Java New client: AdminClient - a Wire Protocol client for administrative operationsNew CLI tool: Kafka Shell - a single unified command line interface for administrative operations

...