DescribeConfigsRequest version will be bumped up to 1.

DescribeConfigsRequest version will be bumped up to 1.

DescribeConfigs Request (Version: 1) => [resource [config_name]] include_synonyms
  resource => resource_type resource_name
    resource_type => INT8
    resource_name => STRING
  config_name => STRING
  include_synonyms => BOOLEAN

DescribeConfigs Response (Version: 1) => throttle_time_ms [entities]
  throttle_time_ms => INT32
  entities => error_code error_message resource [configs]
    error_code => INT16
    error_message => STRING
    resource => resource_type resource_name
      resource_type => INT8
      resource_name => STRING
    configs => [config_entry [synonym]]    <= Added [synonym]
    config_entry =>
      config_name => STRING
      config_value => NULLABLE_STRING
      read_only => BOOLEAN
      config_source => INT8                <= Replaced boolean is_default with more generic config_source (see below for values)
      is_sensitive => BOOLEAN
    synonym =>                             <= NEW
      config_name => STRING
      config_value => NULLABLE_STRING
      config_source => INT8                <= may be one of (TOPIC|DYNAMIC_BROKER|DYNAMIC_DEFAULT_BROKER|STATIC_BROKER|DEFAULT)    <= may be one of (TOPIC|DYNAMIC_BROKER|DYNAMIC_DEFAULT_BROKER|STATIC_BROKER|DEFAULT)

When MetadataRequest version is increased after 1.1.0 release, a new error code ENDPOINT_NOT_FOUND_ON_LEADER will be added to notify clients when a listener is available on the broker used to obtain metadata, but not on the leader of a partition. This could be a transient error when listeners are added and will be retried in the same way as LEADER_NOT_AVAILABLE. Broker will continue to return LEADER_NOT_AVAILABLE to clients using older version of MetadataRequest. In 1.1.0, brokers will return LEADER_NOT_AVAILABLE instead of UNKNOWN_SERVER_ERROR in older versions.

Public Interface Changes

A new interface Reconfigurable will be added to notify reconfigurable objects of configuration changes. For example, metrics reporters that support reconfiguration can implement the interface Reconfigurable to enable reconfiguration without broker restart. The interface will also be implemented by all internal classes which support reconfiguration (e.g. ChannelBuilder)

...

public enum ConfigSource {
    DYNAMIC_TOPIC_CONFIG,                   // dynamic topic config that is configured for a specific topic
    DYNAMIC_BROKER_CONFIG,          // dynamic broker config that is configured for a specific broker
    DYNAMIC_DEFAULT_BROKER_CONFIG,  // dynamic broker config that is configured as default for all brokers in the cluster
    STATIC_BROKER_CONFIG,           // static broker config provided as broker properties at start up (e.g. server.properties file)
    DEFAULT_CONFIG                  // built-in default configuration for configs that have a default value
}

Tools

 a default value
}

Tools

kafka-configs.sh will be updated to configure defaults and overrides for dynamic configuration options for the entity type brokers. This will be done using the new AdminClient that talks to brokers rather than to ZooKeeper so that password encryption and config validation need to be implemented only in the broker. The full migration of ConfigCommand to use the new AdminClient will be done under KIP-248.

Two new options will be added to kafka-configs.sh will be updated to configure defaults and overrides for dynamic configuration options for the entity type brokers.under this KIP to enable broker config describe and update using the new AdminClient.

• --bootstrap-server <host>:<port>
• --command-config <config-file>

Example:

Add/update broker-specific config for broker with broker id 0:

bin/kafka-configs  configs.sh  --zookeeper bootstrap-server localhost:2181 9092 --command-config /path/adminclient.props --alter --add-config 'ssl.keystore.location=keystore1.jks,ssl.keystore.password=password1' --entity-name 0 --entity-type brokers

Add/update cluster-wide default config:

bin/kafka-configs  configs.sh  --bootstrap-zookeeper server localhost:2181 9092 --command-config /path/adminclient.props --alter --add-config 'unclean.leader.election.enable=false' --entity-default --entity-type brokers

Delete broker-specific config (the actual config will revert to its default value, see precedence of configs):

bin/kafka-configs.sh --zookeeper localhost --bootstrap-server localhost:9092 --command-config /path/adminclient.props --alter --delete-config unclean.leader.election.enable --entity-type brokers --entity-name 0

...

Listener Configs

• listeners
• advertised.listeners
• listener.security.protocol.map
• inter.broker.listener.name
• security.inter.broker.protocol
• advertised.listeners
• listener.security.protocol.mapsasl.mechanism.inter.broker.protocol

Common security config

• principal.builder.class

...

• SSL configs will be updated by reconfiguring ChannelBuilder and creating a new SslFactory. If SSL is used for inter-broker communication, inconsistent changes (e.g changing CA) should be made by adding a new listener with the new properties. This is true for SASL as well.
• SASL configuration updates will be supported using the dynamic JAAS configuration option sasl.jaas.config
• Updates to advertised.listeners will re-register the new listener in ZK. This update will be not allowed for the listener used in inter-broker communication. In addition to this, AdminClient will not allow updates to the listener that was used to make the alter request.
• When changes are made to listeners, additional logic will be required in the controller to broadcast the updated metadata to all brokersto all brokers.
• All the security configs can be dynamically configured for new listeners. In the initial implementation, only some configs will be dynamically updatable for existing listeners (e.g. SSL keystores). Support for updating other security configs dynamically for existing listeners will be added later.

 Limitations:

• Configuration updates will not be allowed for the listener used in inter-broker communication. To update This KIP will not allow dynamic updates to inter-broker configs without restarting the broker, a new listener can be created with new configs and the inter-broker listener can be changed to the new listener (similar to the currently documented security upgrade process)security protocol or listener name. Support for changing inter-broker security configuration without a restart will be done in a follow-on KIP along with additional validation to ensure that all brokers have enabled the new config.

Future Work

Message Format and Inter Broker Protocol

...

• log.message.format.version
• inter.broker.protocol.version
• inter.broker.listener.name
• security.inter.broker.protocol
• sasl.mechanism.inter.broker.protocol

Dynamic update changes:

• AdminClient needs to verify that all brokers in the cluster support the new version before making an update. To enable this, the version of the broker will be added to the JSON registered by each broker during startup at /brokers/ids/id. To avoid all brokers reading version information from ZooKeeper, the controller should propagate this information in UpdateMetadataRequest. This requires protocol change. Note also that this approach only allows verification of the version of brokers that are alive at the time AlterConfigs is executed.
• Additional validation is required before changing inter-broker listener or security configuration. This should ensure that the client-side and server-side configs match and also that all brokers in the cluster support the new configuration.

Compatibility, Deprecation, and Migration Plan

...