Page History

...

Config entities

Client entity types and names are more dynamic than broker and topic entity types and names. For example, client quotas can be tied to a user principle that is associated with a session as well as a client-id which is a generic workload identifier. This is not as simple as a broker with a broker id, so dynamic client configs should also have similar expressibility and extensibility to that which was introduced in KIP-546 for client quotas. The ClientConfigs APIs will follow the design pattern of the ClientQuotas APIs with a few differences.

...

The user config is updated when the client-id component entity name in the AlterClientConfigsRequest is nullis missing from an AlterClientConfigsRequest. The <user, client-id> config is updated otherwise.

Client configs will have the following order of precedence:

/config/users/<user>/clients/<client-id> 

/config/users/<user>

When the broker handles a DescribeClientConfigsRequest that a client is making for it's own dynamic configs (e.g. ResolveEntity field set to true), the user config and the <user, client-id> config will be returned as one entity whose configs are resolved with the above following order of precedence .from most precedent to least precedent:

/config/users/<user>/clients/<client-id> 

/config/users/<user>

For example, any config key value pairs found in /config/users/<user>/clients/<client-id> will override any config key value pairs found in /config/users/<user>. The final resolved map of configs will then be sent back to the client and will overwrite statically provided client configs.

Client quotas are stored in these znodes as well. However, all configs that are not quota configs are filtered out when constructing a DescribeClientQuotasResponse. Similar to this, all configs that are not dynamic client configs will be Client quotas are stored in these znodes as well. However, all configs that are not quota configs are filtered out when constructing a DescribeClientQuotasResponse. Similar to this, all configs that are not dynamic client configs will be filtered out when constructing a DescribeClientConfigsResponse. The value for each key will also be validated against the allowed values for that key. For example, if the user tries to set acks=2, an InvalidRequest error code will be sent back. The client will also have to validate dynamic configs against user-provided configs, so the broker is only doing partial validation here. This is worth doing since partially validated configs may only be invalid for a subset of clients, whereas acks=2 would be invalid for all clients.

...

For example, the user can add the new configs supported with this KIP along with the quota configs that are supported for the admin client in KIP-546 to their default dynamic config. In this example the user mixes some dynamic client configs that this KIP introduces with the quota config producer_byte_ratewith the quota configs that are supported for the admin client in KIP-546 to their default dynamic config. In this example the user mixes some dynamic client configs that this KIP introduces with the quota config producer_byte_rate:

bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --alter \
  --entity-type users \
  --entity-name alice \
  --add-config acks=-1,session.timeout.ms=11000,producer_byte_rate=50000
  Completed updating config for user alice.

The user can also add configs specific to a client-id that will override the user's default dynamic configs:

bin/kafka-configs.sh.sh --bootstrap-server localhost:9092 \
  --alter \
  --bootstrapentity-servertype localhost:9092users \
  --alterentity-name alice \
  --entity-type usersclients \
  --entity-name aliceclientid-override \
  --add-config acks=-10,sessionheartbeat.timeoutinterval.ms=110002000,producer_byte_rate=5000060000
  Completed updating config for user alice.

The user can also add configs specific to a client-id that will override the user's default dynamic configs:

.

The user can describe these configs the same way that client quotas are described with the users and clients entity types. To make this possible, kafka-configs.sh will be sending a DescribeClientConfigsRequest as well as the DescribeClientQuotasRequest. Dynamic client configs must at least be scoped by a user:

bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --alterdescribe \
  --entity-type users \
  --entity-name alice \
Quota configs --entity-type clients \
  --entity-name clientid-override \
  --add-config acks=0,heartbeat.interval.ms=2000,for user-principal 'alice' are producer_byte_rate=6000050000.0
Dynamic configs Completed updating config for user-principal 'alice.

The user can describe these configs the same way that client quotas are described with the users and clients entity types. To make this possible, kafka-configs.sh will be sending a DescribeClientConfigsRequest as well as the DescribeClientQuotasRequest. Dynamic client configs must at least be scoped by a user:

' are session.timeout.ms=11000, acks=-1

They may optionally be scoped by a client-id:

bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --describe \
  --entity-type users \
  --entity-name alice \ 
  --entity-type clients \
  --entity-name clientid-override
Quota configs for user-principal 'alice', client-id 'clientid-override' are producer_byte_rate=5000060000.0
Dynamic configs for user-principal 'alice', client-id 'clientid-override' are sessionheartbeat.timeoutinterval.ms=110002000, acks=-10

If They may optionally be scoped by a client-id is not specified when describing, all of the <user, client-id> entity configs will be returned:

bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --describe \
  --entity-type users \
  --entity-name alice \ 
  --entity-type clients \
  --entity-name clientid-override
Quota configs for user-principal 'alice', client-id 'clientid-override' are producer_byte_rate=60000.0
Dynamic configs for user-principal 'alice', client-id 'clientid-override' are heartbeat.interval.ms=2000, acks=0

...

Dynamic configs for user-principal 'alice', client-id '""' are acks=-1

The default dynamic config will be used in the case that the client-id dynamic config does not contain a key that the default does contain, but only if the client is requesting configs with the ResolveEntity flag set to true.

...

• Introducing new entity types for kafka-configs.sh that producers and consumers can associate themselves with. This would make the tool more cumbersome to use and it is most intuitive that client configurations be dynamically altered with the clients and users entity types.

• Use the {Describe, IncrementalAlter}Configs APIs. Client config entities are more dynamic than entities with a singular resource name and type which makes it hard to fit them into generic APIs that expect a distinct entity name and type.
• Use the <user/client-id> hierarchy implemented for client quotas in KIP-55 and extended for the admin client in KIP-546. Quotas are inherently hierarchical but client configs are not, so it seems reasonable to use a hierarchy of shallow depth for dynamic client configs.

• Making client config compatibility information available to the user

  The user should be able to see what dynamic configs are supported for each application. However, clients that are using the same <user, client-id> entity may not necessarily support the same dynamic configs. A flawed solution to this is to store a map of ClientInformation to a , storing a list of supported configs in each entityconfigs alongs side quotas and configs is a flawed solution.

  A better solution is to store config registrations in an internal topic. The Java producer and consumer clients can register the configs that they support with a DescribeClientConfigsRequest. The broker can write a key-value pair to an internal topic upon receiving the request where the key is the <user, client-id> entity and the value is ClientVersion along with the list of supported configs.is ClientVersion along with the list of supported configs.

  Ad hoc aggregations of registration data for a particular entity could be performed to give descriptive information about client config compatibility to the user. All versions of clients that registered with an a <user, client-id> entity along with the supported configs for each version of client could be aggregated and when a DescribeClientConfigsRequest from an admin client is received. This information would then be returned to the user admin client in the DescribeClientConfigsResponse. For example, supported dynamic configs for user-principal 'alice', client-id 'clientid-override' are "{ 'ClientInformation(softwareName=apache-kafka-java, softwareVersion=x.y.za-SNAPSHOT)': 'acks', 'ClientInformation(softwareName=apache-kafka-java, softwareVersion=x.y.b-SNAPSHOT)': 'acks, enable.idempotence' }".

• Interesting hierarchies for config hierarchies overrides could be constructed if the Java producer and consumer resolved the dynamic configs instead of the broker. For example,  from most precedent to least precedent:
  • /config/users/<user>/clients/<client-id>
  • .properties file configs
  • /config/users/<user>
  • Static default configs defined in ProducerConfig and ConsumerConfig.

...