Note that this KIP proposal is derived from a proposal by Grant Henke that was part of KIP-4 - Command line and centralized administrative operations.
Status
Current state: Draft
Discussion thread: TBD
JIRA:
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
KIP-4 - Command line and centralized administrative operations outlines the motivation for exposing admin operations via the protocol:
- Allows clients in any language to administrate Kafka
- Wire protocol is supported by any language
- Provides public client for performing admin operations
- Ensures integration test code in other projects and clients maintains compatibility
- Prevents users from needing to use the Command classes and work around standard output and system exits
- Removing the need for admin scripts (kafka-configs.sh, kafka-topics.sh, etc) to talk directly to Zookeeper.
- Allows ZNodes to be completely locked down via ACLs
- Further hides the Zookeeper details of Kafka
A couple of specific use cases are worth pointing out:
- The Metadata request exposes topic metadata, but it does not expose topic configs. ListConfigs will make that information available to any client of the Kafka protocol (including the upcoming Java AdminClient).
- AlterConfigs would make it possible to update topic configs, but also client and replication quotas via the protocol.
Public Interfaces
List Configs
ListConfigs Request (Version: 0) => [entities] entities => entity_type entity_name entity_type => INT8 entity_name => STRING
Request semantics:
- Can be sent to any broker
- If there are multiple instructions for the same entity in one request the extra request will be ignored
- This is because the list of entities is modeled server side as a set
- Multiple entities results in the same end goal, so handling this error for the user should be okay
- This is similar to how delete topics handles requests
- Entity types are "Topic", "Client", and "Broker".
- If entity_type is "Broker" and entity_name matches the broker that has received the request, read-only configs for this broker are also returned.
- Below are the authorization requirements for each type:
- Broker: Must be authorized to the "Describe" Operation on the "Cluster" resource
- Topic: Must be authorized to the "Describe" Operation on the "Topic" resource
- Client: Must be authorized to the "Describe" Operation on the "Client" resource
- This is a new resource needed
- TODO: best way to handle this...
- Arbitrary configurations are allowed
- This provides flexibility for custom clients, and allows all "plugin" or extra configs to be shown
- The user can validate the configs after the describe command in their client to check for errors, but the wire protocol should relay all information.
ListConfigs Response (Version: 0) => [entities] entities => entity_type entity_name error_code [configs] entity_type => INT8 entity_name => STRING error_code => INT16 configs => config_name => STRING config_value => STRING read_only => BOOLEAN is_default => BOOLEAN is_sensitive => BOOLEAN
For each config, If is_sensitive
is true
, config_value
will always be null
to ensure that we don't leak sensitive information like passwords.
Alter Configs
AlterConfigs Request (Version: 0) => [entities] validate_only validate_only => BOOLEAN entities => entity_type entity_name [configs] entity_type => INT8 entity_name => STRING configs => config_name => STRING config_value => STRING
Request Semantics
- Can be sent to any broker
- If there are multiple instructions for the same entity in one request, an InvalidRequestException will be logged on the broker and a single error code for that topic will be returned to the client
- This is because the list of entities is modeled server side as a map with entity as the key
- Entity types are "Topic", "Client", and "Broker".
- Below are the authorization requirements for each type:
- The principal must be authorized to the "Create" Operation on the "Cluster" resource to create topics.
- Unauthorized requests will receive a ClusterAuthorizationException
- If an
Alter
operation is attempted on a read-only config, anUnsupportedOperation
error will be returned for the relevant entity. - The request is not transactional.
- If an error occurs for an entity, others could still be updated.
- Errors are reported independently per entity.
AlterConfigs Response (Version: 0) => [responses] responses => entity_type entity_name error_code error_message entity_type => INT8 entity_name => STRING error_code => INT16 error_message => STRING
Proposed Changes
TBD
Compatibility, Deprecation, and Migration Plan
TBD
Rejected Alternatives
TBD
Future work
- Forward requests to the relevant brokers in order to return `read-only` broker configs.