...

• Changes to Wire Protocol:
  • Adds the following new Request/Response messages:
    • CreateTopics
    • AlterTopics
    • DeleteTopics
    • ListAcls
    • AlterAcls
    • DescribeConfig
    • AlterConfig
  • Modifies Metadata Request/Response to allowing polling for in-progress or complete admin operations. Added fields include:
    • Add the ability to request no topics with a null topics list
    • Boolean indicating if a topic is marked for deletion
    • Boolean indicating if a topic is an internal topic
    • Rack information (if not added by KIP-36 Rack aware replica assignment)
    • Boolean indicating if a broker is the controller

• New Java client: AdminClient - a Wire Protocol client for administrative operations

Proposed Changes

Proposed changes include 4 2 parts:

1. Wire protocol additions and changes
2. Server-side message handlers and authorization

3. New Java AdminClient implementation

4. Refactor admin scripts and code to use new client where appropriate

Follow Up Changes

Changes that should be considered shortly after or are enabled by this KIP included:

• General

  • New Java AdminClient implementation (KIP-117)
  • Refactor admin scripts and code to use new client where appropriate
  General 
  • Support forwarding requests to the required broker (coordinator, group leader, partition leader) (KAFKA-1912)
    • See Request Forwarding below

• Metadata Schema

  • Consider supporting regex topic filters in the request 
  • Filter internal topics using the returned metadata (

    Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key KAFKA-3954

    )

• Topic Admin Schema

  • Improve the broker side delete topic implementation
    • Delete is likely to get used

  Topic Admin Schema

  • Improve the broker side delete topic implementation
    • Delete is likely to get used more once creation/deletion is made easier with the client. The broker side implimentation implementation has had many jiras. 
    • Currently can't delete unhealthy topics. 
  • Support cluster consistent blocking to wait until all relevant brokers have the required metadata
    • This may require significant re-work of the controller to do correctly
    • See Cluster Consistent Blocking below
  • Implement auto-topic creation client side (KAFKA-2410)
  • Add topic creation to the MirrorMaker client?
  • Support renaming topics (KAFKA-2333)
    • This might required unique ids for topics instead of using the name (this would improve delete too)
  • Improve reliability and speed of topic deletion
    • Support force deleting unhealthy topics
    • Support marking for deletion and async data cleanup
      • This would required unique ids for topics instead of using the name (this is needed for rename too)
      • The topic can then be marked as deleted instead of requiring all data to be removed immediatly immediately and in the mean time a new topic with the same name can be created. 

• ACL Admin Schema

  • Review privileges for listing and altering ACLs to be more fine grained.
  • Provide an Authorizer interface using the new Java classes used by the ACL requests/responses (KAFKA-3509)
    • Deprecate the old one to encourage transition
  • Define standard Exceptions that can be thrown by the Authorizer in the interface (KAFKA-3266)
    • Otherwise all exceptions are unknown server exception to the client
  • Consider building a sync call into the Authorizer to ensure changes are propagated

...

Because we already add a timeout field to the requests wire protocols, changing the behavior to block until the cluster is consistent in the future would not require a protocol change. Though the version could be bumped to indicate a behavior change. 

3. Admin Client

This component is intended to be a Kafka out-of-box client implementation for Admin commands.

Admin client will use Kafka NetworkClient facility from /clients for cluster communication. Besides Admin commands, client will handle cluster metadata cache and will provide user with a convenient way of handling long running commands (e.g. reassign partitions).

Since Topic commands will support batching (and so will AdminClient) user besides Admin API will be provided with request builders which will help to create requests correctly.

Proposed API: TBD 

...