Table of Contents |
---|
Status
Current state: Under DiscussionAccepted
Discussion thread: here
Vote: here
JIRA: Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key KAFKA-7466
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
The AlterConfigs RPC gives users a way to alter the configuration of a topic, broker, or other resource. However, the new configuration replaces any existing configuration. This makes AlterConfigs unusable in cases where the client does not know the full existing configuration before making the modification. This makes AlterConfigs less efficient, since it means that the client needs to call DescribeConfigs to retrieve the existing configuration before making any modification. It also introduces the possibility of "lost updates" when multiple clients enter a read-modify-write cycle around the same time.
...
In order to fix these issues, we should introduce a new RPC named modifyConfigs incrementalAlterConfigs. The new RPC should operate incrementally, modifying only the configuration values that are specified. We should deprecate AlterConfigs.
Proposed Changes
...
Public Interfaces
Code Block | |||
---|---|---|---|
| |||
@InterfaceStability.Evolving public class AlterConfigOp { public ModifyConfigsResult modifyConfigs( Map<ConfigResource,public Config>enum changes, Set<ConfigResource> removals, OpType { final ModifyConfigsOptions options); |
If a configuration key is specified in both changes and removals, it will be removed.
Similar to AlterConfigsResult, ModifyConfigsResult will have a values function which allows callers to determine the result of a specific configuration resource modifications, and an all function which will throw an exception if any operation failed.
Code Block | ||
---|---|---|
| ||
@InterfaceStability.Evolving public class ModifyConfigsResult { SET((byte) 0), private final Map<ConfigResource, KafkaFuture<Void>> futuresDELETE((byte) 1), APPEND((byte) 2), SUBTRACT((byte) 3); ModifyConfigsResult(Map<ConfigResource, KafkaFuture<Void>> futures) { private static final Map<Byte, OpType> OP_TYPES = thisCollections.futuresunmodifiableMap( = futures; } /** * Return a map from resources to futures which can be used to check the status of the operation on each resource. Arrays.stream(values()).collect(Collectors.toMap(OpType::id, Function.identity())) ); private final byte id; */ OpType(final public Map<ConfigResource, KafkaFuture<Void>> values(byte id) { return futures; } /**this.id = id; * Return a future} which succeeds only if all the alter configspublic operations succeed. */byte id() { public KafkaFuture<Void> all() { return id; return KafkaFuture.allOf(futures.values().toArray(new KafkaFuture[0])); } } |
Similar to AlterConfigsOptions, ModifyConfigsOptions will include a timeout value and a dry-run flag.
Code Block | ||
---|---|---|
| ||
@InterfaceStability.Evolving public class ModifyConfigsOptions extends AbstractOptions<ModifyConfigsOptions> { private boolean validateOnly = false; public static OpType forId(final byte id) { /** * Set the request timeout in milliseconds for this operation or {@code null} if the default request timeout for thereturn OP_TYPES.get(id); } } private *final AdminClientConfigEntry shouldconfigEntry; be used. private final OpType */opType; public ModifyConfigsOptions timeoutMs(Integer timeoutMs AlterConfigOp(ConfigEntry configEntry, OpType operationType) { this.timeoutMsconfigEntry = timeoutMsconfigEntry; this.opType = return thisoperationType; } /** public ConfigEntry configEntry() { * Return true if the request shouldreturn beconfigEntry; validated without altering the configs. */ }; public booleanOpType shouldValidateOnlyopType() { return validateOnlyopType; }; } /** * Set to true if the request should be validated without altering the configs. */ public ModifyConfigsOptionsAlterConfigsResult validateOnly(boolean validateOnly) {incrementalAlterConfigs( Map<ConfigResource, Collection<AlterConfigOp>> configs, this.validateOnlyfinal = validateOnly; return this; } } |
Protocol APIs
AlterConfigsOptions options); |
Proposed Changes
The new IncrementalAlterConfigs API in AdminClient will take a collection of operations describing the configuration modifications. There are four types of operations.
- Set: set a configuration to a value. The value must not be null.
- Delete: delete a configuration key
- Append: if a configuration key is a list of values, add to the list.
- Subtract: if a configuration key is a list of values, subtract from the list.
We will use existing AlterConfigsOptions, AlterConfigsResult API classes to pass the API config options and to return the result of configuration resource modifications.
Similar to existing alterConfigs API, we will to keep the "transactionality" of updating several configs for the same ConfigResource at once. We guarantee that we never do a partial update of a collection of configs
for a ConfigResource from a single request. On validation/update error, we will return the error for the ConfigResource.
Protocol APIs
There will be a new Incremental AlterConfigsRequestThere will be a new ModifyConfigsRequest. This request is very similar to AlterConfigsRequest. However, unlike in AlterConfigsRequest, the config_value is nullable in ModifyConfigsRequest. In the case where config_value is null, the existing configuration key will be erased.
Code Block | ||||
---|---|---|---|---|
| ||||
ModifyConfigsRequestIncrementalAlterConfigsOp => INT8 0: SET 1: REMOVE 2: APPEND 3: SUBTRACT IncrementalAlterConfigsRequest (Version: 0) => [resources] validate_only validate_only => BOOLEAN resources => resource_type resource_name [configs] resource_type => INT8 resource_name => STRING configs => config_name config_op config_value config_name => STRING config_op => INT8 config_value => NULLABLE_STRING |
The ModifyConfigsResponse Incremental AlterConfigsResponse is the same as the AlterConfigsResponse.
Code Block | ||
---|---|---|
| ||
ModifyConfigsResponseIncrementalAlterConfigsResponse (Version: 0) => [responses] responses => resource_type resource_name error_code error_message resource_type => INT8 resource_name => STRING error_code => INT16 error_message => NULLABLE_STRING |
There are two new error conditions.
- If a configuration key was specified more than once in the request for given ConfigResource, we fail ConfigResource update with the INVALID_REQUEST.
- If APPEND or SUBTRACT was specified for a configuration key that does not take a list of elements, we fail ConfigResource update with the INVALID_REQUEST.
Compatibility, Deprecation, and Migration Plan
This change is backwards compatible because the existing alterConfigs RPC is retained. Clients will migrate to the new modifyConfigs incrementalAlterConfigs RPC as needed.
Rejected Alternatives
We could have changed alterConfigs so that it had an incremental mode. This would have avoided creating a new RPC. However, in order to avoid breaking compatibility, the incremental mode could not have been made the default for AdminClient. We would also not have been able to deprecate the non-incremental mode. This would create a confusing and dangerous stumbling block for new users. Because the problems with non-incremental mode are not immediately obvious, it is likely that many users would have made the wrong decision about what API to use.