Current state: Under Discussion
Discussion thread: WIP
JIRA:
Kafka allows users to configure its behavior through multiple methods:
However, there is no standardized or consistent approach for determining which source takes precedence when multiple are provided simultaneously. This inconsistency can cause user confusion and make the system more bug-prone. This KIP aims to establish a clear and uniform precedence order for property loading.
1) The precedence order for loading properties into Kafka configurations is as follows:
2) A new option added for all affected tool scripts:
All the tool scripts that can be configured through either the command line or configuration files, or both, will be adjusted accordingly.
Add the helper methods to CommandLineUtils.java so developers can more easily honor the proposed property loading precedence. All the tool scripts that can be configured through either the command line or configuration files, or both, will be adjusted to call the helper method to follow the proposed property loading precedence.
The following code demonstrates the key helper methods:
/**
* Merge multiple configuration sources according to priority, from highest to lowest:
* 1) Command line arguments
* 2) Properties passed as key=value pairs via command line
* 3) Configuration files
* 4) Default values set by tool scripts
*/
public static Map<String, Object> mergePropertiesWithPrecedence(
Map<String, Object> commandLineMap,
Map<String, Object> commandLineKeyValMap,
Map<String, Object> configMap,
Map<String, Object> toolDefaultMap
) {
Map<String, Object> map = new HashMap<>();
// Default values set by tool scripts and default values of command line arguments
if (toolDefaultMap != null) {
map.putAll(toolDefaultMap);
}
// Configuration file
if (configMap != null) {
map.putAll(configMap);
}
// Properties passed as key=value pairs via command line
if (commandLineKeyValMap != null) {
map.putAll(commandLineKeyValMap);
}
// Command line arguments
if (commandLineMap != null) {
map.putAll(commandLineMap);
}
return map;
} |
Map<String, Object> producerProps() throws IOException {
// Prepare the map from command line arguments
Map<String, Object> commandlineMap = new HashMap<>();
CommandLineUtils.maybeMergeOption(options, commandlineMap, BOOTSTRAP_SERVERS_CONFIG, bootstrapServerOpt);
CommandLineUtils.maybeMergeOption(options, commandlineMap, COMPRESSION_TYPE_CONFIG, compressionCodecOpt);
CommandLineUtils.maybeMergeOption(options, commandlineMap, LINGER_MS_CONFIG, sendTimeoutOpt);
CommandLineUtils.maybeMergeOption(options, commandlineMap, ACKS_CONFIG, requestRequiredAcksOpt);
CommandLineUtils.maybeMergeOption(options, commandlineMap, REQUEST_TIMEOUT_MS_CONFIG, requestTimeoutMsOpt);
CommandLineUtils.maybeMergeOption(options, commandlineMap, RETRIES_CONFIG, messageSendMaxRetriesOpt);
CommandLineUtils.maybeMergeOption(options, commandlineMap, RETRY_BACKOFF_MS_CONFIG, retryBackoffMsOpt);
CommandLineUtils.maybeMergeOption(options, commandlineMap, SEND_BUFFER_CONFIG, socketBufferSizeOpt);
CommandLineUtils.maybeMergeOption(options, commandlineMap, BUFFER_MEMORY_CONFIG, maxMemoryBytesOpt);
CommandLineUtils.maybeMergeOption(options, commandlineMap, BATCH_SIZE_CONFIG, batchSizeOpt);
CommandLineUtils.maybeMergeOption(options, commandlineMap, BATCH_SIZE_CONFIG, maxPartitionMemoryBytesOpt);
CommandLineUtils.maybeMergeOption(options, commandlineMap, METADATA_MAX_AGE_CONFIG, metadataExpiryMsOpt);
CommandLineUtils.maybeMergeOption(options, commandlineMap, MAX_BLOCK_MS_CONFIG, maxBlockMsOpt);
// The map of default values set by tool scripts
Map<String, Object> defaultMap = Map.of(
BOOTSTRAP_SERVERS_CONFIG, CompressionType.NONE.name,
CLIENT_ID_CONFIG, "console-producer",
KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.ByteArraySerializer",
VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.ByteArraySerializer"
);
// producerConfigOpt - Configuration files
// producerPropertyOpt - Properties passed as key=value pairs via command line
// commandlineMap - Command line arguments
// defaultMap - Default values set by tool scripts
Map<String, Object> map = CommandLineUtils.mergeByPriority(options, producerConfigOpt, producerPropertyOpt, commandlineMap, defaultMap);
return map;
} |
Impact: Existing users who have configured their settings based on the current tool implementations, rather than the proposed precedence, may be affected.
Deprecation: Adjusting property loading precedence might break existing users. For this reason, we have an option added to each affected tool script to enable or disable this change. The plan is to always use the proposed precedence in the next major release, Kafka 5.0. Concurrently, the option will be deprecated in Kafka 5.0 and subsequently removed in Kafka 6.0.
Unit tests will be added to ensure that the properties are loaded according to the precedence order as proposed.
1. Deprecate the "modern" option during the proposed rollout and remove it in the next major release, Kafka 5.0
Reason for Rejection: Users who have enabled the modern option in scripts will face the fatal error once Kafka 5.0 is released.
2. Preferring Configuration Files Over Command Line Arguments
Reason for Rejection: Intuitively, users are more likely to expect properties set via command line arguments to take precedence over those defined in configuration files.
3. Allow Each Tool to Have Its Own Property Loading Precedence
Reason for Rejection: While this would provide more flexibility for Kafka developers, it would introduce undesirable complexity to the configuration logic, be confusing to users, and increase the potential for bugs.