DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
Proposed Changes
...
You can refer to https://github.com/apache/kafka/pull/20811 for the detailed changes.
Configuration Loading:
| Code Block | ||||
|---|---|---|---|---|
| ||||
public static final String REMOTE_LOG_METADATA_TOPIC_MIN_ISR_PROP = "remote.log.metadata.topic.min.isr";
public static final String REMOTE_LOG_METADATA_TOPIC_MIN_ISR_DOC = "The minimum number of replicas that must acknowledge a write to remote log metadata topic";
public static final short DEFAULT_REMOTE_LOG_METADATA_TOPIC_MIN_ISR = 2; |
20913 for the detailed changes.
Topic Creation:
Code Block language java title TopicBasedRemoteLogMetadataManager topicConfigs.put(TopicConfig.CLEANUP_POLICY_CONFIG, TopicConfig.CLEANUP_POLICY_DELETE); topicConfigs.put(TopicConfig.REMOTE_LOG_STORAGE_ENABLE_CONFIG, "false"); topicConfigs.put(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, Short.toString(rlmmConfig.metadataTopicMinIsr())); //add the configure return new NewTopic(rlmmConfig.remoteLogMetadataTopicName(), rlmmConfig.metadataTopicPartitionsCount(), rlmmConfig.metadataTopicReplicationFactor()).configs(topicConfigs);
When TopicBasedRemoteLogMetadataManager creates the __remote_log_metadata topic, it will include the min.isr configuration:
...
* Create __remote_log_metadata topic with default configuration
* Verify topic can be created with custom replication.factor and min.isr
Rejected Alternatives
This another approach is maintaining the current default of min.isr=1, It was rejected for the following reasons:
1. Security by Default Principle
Kafka should provide secure defaults for critical internal topics. The current default of min.isr=1 creates an unacceptable data loss risk that most users will not proactively address.
Alternative 1: Make this the default behavior
Reason for rejection: Some users require real-time remote analytics and need data uploaded as soon as possible. Breaking their use case would be unacceptable.
Alternative 2: Global broker-level configuration only
Reason for rejection: Different topics have different requirements. Topic-levelgranularity is essential2. Inconsistency with other critical metadata topics
The __transaction_state topic explicitly sets min.isr=2 via transaction.state.log.min.isr, There is no justifiable reason for treating remote log metadata differently.