DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
/**
* Merge multiple configuration sources according to priority, from highest to lowest:
* 1) Command line arguments
* 2) Properties passed as key=value pairs via command line
* 3) Configuration files
* 4) Default values set by tool scripts and default values of command line arguments
*/
public static Map<String, Object> mergePropertiesWithPrecedence(
Map<String, Object> commandLineMap,
Map<String, Object> commandLineKeyValMap,
Map<String, Object> configMap,
Map<String, Object> toolDefaultMap
) {
Map<String, Object> map = new HashMap<>();
// Default values set by tool scripts and default values of command line arguments
if (toolDefaultMap != null) {
map.putAll(toolDefaultMap);
}
// Configuration file
if (configMap != null) {
map.putAll(configMap);
}
// Properties passed as key=value pairs via command line
if (commandLineKeyValMap != null) {
map.putAll(commandLineKeyValMap);
}
// Command line arguments
if (commandLineMap != null) {
map.putAll(commandLineMap);
}
return map;
} |
...
All of the tools listed below should use the helper methods to ensure they honor the proposed precedence.
| Tools | *Need New "modern" Option for Deprecationand List properties that do not follow the proposed precedence | Validating Required Arguments from Multiple Sources | Previous Jira/Email Discussion | Note | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| kafka-acls.sh | --bootstrap-server --bootstrap-controller | |||||||||||
kafka-broker-api-versions.sh | --bootstrap-server | |||||||||||
| kafka-client-metrics.sh | --bootstrap-server | |||||||||||
| kafka-cluster.sh | --bootstrap-server --bootstrap-controller | |||||||||||
| kafka-configs.sh | --bootstrap-server --bootstrap-controller | |||||||||||
| kafka-console-consumer.sh | For formatter: KEY_DESERIALIZER_CLASS_CONFIG VALUE_DESERIALIZER_CLASS_CONFIG | --bootstrap-server --from-beginning (AUTO_OFFSET_RESET_CONFIG) --group (GROUP_ID_CONFIG) is validated, but not following the proposed flow | ||||||||||
| kafka-console-producer.sh | KEY_SERIALIZER_CLASS_CONFIG VALUE_SERIALIZER_CLASS_CONFIG COMPRESSION_TYPE_CONFIG | --bootstrap-server |
| |||||||||
| kafka-console-share-consumer.sh | For formatter: KEY_DESERIALIZER_CLASS_CONFIG VALUE_DESERIALIZER_CLASS_CONFIG | --bootstrap-server --group (GROUP_ID_CONFIG) is validated, but not following the proposed flow | ||||||||||
| kafka-consumer-groups.sh | --bootstrap-server | |||||||||||
| kafka-consumer-perf-test.sh | GROUP_ID_CONFIG RECEIVE_BUFFER_CONFIG MAX_PARTITION_FETCH_BYTES_CONFIG AUTO_OFFSET_RESET_CONFIG KEY_DESERIALIZER_CLASS_CONFIG VALUE_DESERIALIZER_CLASS_CONFIG CHECK_CRCS_CONFIG | --bootstrap-server |
| |||||||||
| kafka-delegation-tokens.sh | --bootstrap-server | |||||||||||
| kafka-delete-records.sh | --bootstrap-server | |||||||||||
kafka-e2e-latency.sh | For consumer: GROUP_ID_CONFIG ENABLE_AUTO_COMMIT_CONFIG AUTO_OFFSET_RESET_CONFIG KEY_DESERIALIZER_CLASS_CONFIG VALUE_DESERIALIZER_CLASS_CONFIG FETCH_MAX_WAIT_MS_CONFIG For producer: LINGER_MS_CONFIG MAX_BLOCK_MS_CONFIG ACKS_CONFIG KEY_SERIALIZER_CLASS_CONFIG VALUE_SERIALIZER_CLASS_CONFIG | --bootstrap-server --producer-acks (ACKS_CONFIG) | ||||||||||
| kafka-features.sh | --bootstrap-server --bootstrap-controller | |||||||||||
kafka-get-offsets.sh | CLIENT_ID_CONFIG | --bootstrap-server | ||||||||||
kafka-groups.sh | --bootstrap-server | |||||||||||
kafka-leader-election.sh | --bootstrap-server | |||||||||||
kafka-log-dirs.sh | --bootstrap-server | |||||||||||
kafka-metadata-quorum.sh | --bootstrap-server --bootstrap-controller | |||||||||||
kafka-producer-perf-test.sh | KEY_SERIALIZER_CLASS_CONFIG VALUE_SERIALIZER_CLASS_CONFIG | |||||||||||
kafka-reassign-partitions.sh | --bootstrap-server --bootstrap-controller | |||||||||||
kafka-share-consumer-perf-test.sh | GROUP_ID_CONFIG RECEIVE_BUFFER_CONFIG MAX_PARTITION_FETCH_BYTES_CONFIG AUTO_OFFSET_RESET_CONFIG KEY_DESERIALIZER_CLASS_CONFIG VALUE_DESERIALIZER_CLASS_CONFIG CHECK_CRCS_CONFIG | --bootstrap-server | ||||||||||
kafka-share-groups.sh | --bootstrap-server | |||||||||||
kafka-streams-application-reset.sh | --bootstrap-server | This tool script is the only one with a default bootstrap-server; consider aligning it with the others. | ||||||||||
kafka-streams-groups.sh | --bootstrap-server | |||||||||||
kafka-topics.sh | --bootstrap-server | |||||||||||
kafka-transactions.sh | --bootstrap-server | |||||||||||
kafka-verifiable-consumer.sh | GROUP_PROTOCOL_CONFIG GROUP_REMOTE_ASSIGNOR_CONFIG PARTITION_ASSIGNMENT_STRATEGY_CONFIG --group-id (GROUP_ID_CONFIG) ENABLE_AUTO_COMMIT_CONFIG AUTO_OFFSET_RESET_CONFIG | --bootstrap-server --group-id (GROUP_ID_CONFIG) | ||||||||||
kafka-verifiable-producer.sh | KEY_SERIALIZER_CLASS_CONFIG VALUE_SERIALIZER_CLASS_CONFIG ACKS_CONFIG RETRIES_CONFIG | --bootstrap-server | ||||||||||
kafka-verifiable-share-consumer.sh | --bootstrap-server --group-id (GROUP_ID_CONFIG) |
...
Deprecation: Adjusting property loading precedence might break existing users. For this reason, we have an option added to each affected tool script to enable or disable this change. The plan is to always use the proposed precedence in the next major release, Kafka 5.0. Concurrently, the option will be deprecated in Kafka 5.0 and subsequently removed in Kafka 6.0.
Test Plan
- Unit tests for helper methods.
- Unit tests will be added to ensure that the properties are loaded according to the
...
- precedence order as proposed.
- Unit tests for validating required arguments from multiple sources.
Rejected Alternatives
1. Deprecate the "modern" option during the proposed rollout and remove it in the next major release, Kafka 5.0
...