Page History

...

The hierarchy for resolving client quotas is rather complex. This is because quotas need to be set on every application in the system to achieve consistent results. If a quota is set for client A but not client B, client B can end up hoarding resources because it is not limited. This would potentially cause client A to do work at a lower rate than what the quota specifies. Having a robust hierarchy for quotas allows the user to set quotas on all applications rather easily and then fine tune as needed. However, the hierarchy for client configs does not need to have the same amount of depth. There will just be a dynamic default and a dynamic config so that dynamic client configuration is consistent with static client configuration (e.g. .properties file & client defaults). Since users should not be able to change each others dynamic configs, the entities in the hierarchy will be scoped by at least user principle and optionally by client-id.

...

Making client config compatibility information available to the user

The user should be able to see what dynamic configs are supported for each application. However, clients may not necessarily support the same dynamic configs even if they are using the same <user, client-id> entity. A solution to this is to store a map of client software names and versions to lists of supported configs in each <user, client-id> entity config. The producer and consumer clients can register the configs they they support and the broker can tie the registered keys to the serialized ClientInformation from the RequestContext so that this compatibility information can be displayed to the user. As long as the user knows what version of client each application is using, they will know which applications will be affected by each dynamic config.

...

• Introducing new entity types for kafka-configs.sh that producers and consumers can associate themselves with. This would make the tool more cumbersome to use and it is most intuitive that client configurations be dynamically altered with the clients entity typeand users entity types.

• Use the {Describe, IncrementalAlter}Configs APIs. Client config entities are more dynamic than entities with a singular resource name and type which makes it hard to fit them into generic APIs that expect a distinct entity name and type.
• Use the <user/client-id> hierarchy implemented for client quotas in KIP-55 and extended for the admin client in KIP-546. Quota APIs are different and quotas are inherently hierarchical, so it seems reasonable to use a different approach here for levels of hierarchy and how the hierarchy is resolved. Sending user-provided client configuration values to the broker on startup. The user-provided client configurations are not needed by the broker to send back dynamic configurations. DescribeConfigsRequest does not have a field for config values so a new message format would need to be createdhierarchy of shallow depth for dynamic client configs.
• Adding a config enable.dynamic.config to producers and consumers to enable the feature. Adding the dynamic config supported.configs is a better alternative since the user doesn't have to remember what types of clients support the capability but instead can just see which entities have clients that are registered along with the configs that are supported for each software name and version.

• Making certain client configurations topic level configurations on the broker.

  • The semantic for the ProduceRequest API would be undefined since the producer would not receive a response with an offset for the ProduceRequests with acks=0.

  • If this were implemented for acks there would also be quite a bit of overhead associated with extra round trips since the RecordAccumulator sends batches that may contain records from multiple topics. If these topics have different acks configurations the records would need to be sent in different batches based on the acks value.

  • For example, if a producer is consistently producing to 2 different topics and one is configured as acks=0 while the other is acks=-1. This would require twice the amount of round trips to produce the same number of messages.