Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Proposed Changes

...

You can refer to https://github.com/apache/kafka/pull/20811 for the detailed changes.

Configuration Loading:

Code Block
languagejava
titleTopicBasedRemoteLogMetadataManagerConfig
public static final String REMOTE_LOG_METADATA_TOPIC_MIN_ISR_PROP =  "remote.log.metadata.topic.min.isr";
public static final String REMOTE_LOG_METADATA_TOPIC_MIN_ISR_DOC =  "The minimum number of replicas that must acknowledge a write to remote log metadata topic";
public static final short DEFAULT_REMOTE_LOG_METADATA_TOPIC_MIN_ISR = 2;

20913 for the detailed changes.

  • Topic Creation:

    Code Block
    languagejava
    titleTopicBasedRemoteLogMetadataManager
    topicConfigs.put(TopicConfig.CLEANUP_POLICY_CONFIG, TopicConfig.CLEANUP_POLICY_DELETE);
    topicConfigs.put(TopicConfig.REMOTE_LOG_STORAGE_ENABLE_CONFIG, "false");
    topicConfigs.put(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, Short.toString(rlmmConfig.metadataTopicMinIsr())); //add the configure
    return new NewTopic(rlmmConfig.remoteLogMetadataTopicName(), rlmmConfig.metadataTopicPartitionsCount(), rlmmConfig.metadataTopicReplicationFactor()).configs(topicConfigs);

    When TopicBasedRemoteLogMetadataManager creates the __remote_log_metadata topic, it will include the min.isr configuration:          

...

 * Create __remote_log_metadata topic with default configuration
 * Verify topic can be created with custom replication.factor and min.isr

Rejected Alternatives

This another approach is maintaining the current default of min.isr=1, It was rejected for the following reasons:

1. Security by Default Principle
   Kafka should provide secure defaults for critical internal topics. The current default of min.isr=1 creates an unacceptable data loss risk that most users will not proactively address. 

Alternative 1: Make this the default behavior

Reason for rejection: Some users require real-time remote analytics and need data uploaded as soon as possible. Breaking their use case would be unacceptable.

Alternative 2: Global broker-level configuration only

Reason for rejection: Different topics have different requirements. Topic-levelgranularity is essential2. Inconsistency with other critical metadata topics
   The __transaction_state topic explicitly sets min.isr=2 via transaction.state.log.min.isr, There is no justifiable reason for treating remote log metadata differently.