Page History

...

1. Users modify the topic configuration:
1. Users set the configuration "remote.storage.enable=false" for the desired topic, indicating the disablement of tiered storage.
2. Optionally, users specify a disablement policy to either retain or delete the remote data associated with the topic: remote.log.disable.policy=retain|delete.
2. Controller persists configuration change:
1. The controller creates a ConfigRecord and persists it in the metadata topic.
2. The controller also updates the Topic metadata to increment the tiered_epoch and update the tiered_stateto DISABLING state.
3. Broker detects the configuration change:
1. Broker replicate the metadata topic and fetch the latest records of ConfigRecord changes.
2. The broker calls the ConfigHandler to process the ConfigRecord change with tiered storage disablement.
3. The broker ConfigHandler calls RemoteLogManager#stopPartitions().
4. Execution of RemoteLogManager#stopPartitions():
1. Removes the scheduled tasks for the topic-partition from the thread pool responsible for archiving logs to remote storage.
2. If the disablement policy is set to delete, the Log start offset (LSO) is updated to match the Local Log Start Offset and the remote log is deleted by calling the RemoteStorageManager#deleteLogSegmentData().
3. If the disablement policy is set to retain, there is no extra call required to delete remote data as the RemoteLogManager leader partition task will periodically expire segments that are earlier than the log start offset.
5. Controller interaction:
1. The controller enqueues the disablement intent for tiered storage to a local queue.
2. It sends an API call, called DisableRemoteTopic, to each broker with a callback to disable the remote topic.
3. DisableRemoteTopic is a new controller-to-broker API that will be introduced in this KIP.
4. Note that the current approach for communication between the Controller and brokers involves a familiar pattern. Tasks are enqueued in a local queue within the Controller Context, and a request is sent to the brokers along with a callback mechanism. This method is currently employed by the controller for operations, such as deleting topics. For instance, when the controller initiates the deletion of topics, it maintains a local queue called topicsToBeDeleted. This queue serves the purpose of monitoring the progress of topic deletions.
6. Completion of disablement:
1. Once the controller receives responses from all brokers for the DisableRemoteTopic request, it updates the tiered_state in the TopicMetadata to "DISABLED".
2. This update marks the completion of the disablement process, indicating that tiered storage has been successfully disabled for the Kraft-based clusters.

Failure Modes

• Controller Failover in DISABLING State: Following a new controller election, the controller context will be reconstructed for all topic-partitions that are in the DISABLING state. Additionally, the controller will initiate a new request for DisableRemoteTopic to all brokers.
• Broker Dies in DISABLING State or Fails to Complete the DisableRemoteTopic Request: The controller maintains an internal queue to track completed DisableRemoteTopic calls. In the event of a leader failover during this state, the controller will retry the operation until it receives a successful response. This behavior mirrors how we currently handle topic deletion.

Internal Interface Changes

1. DisableRemoteTopic is a new controller-to-broker API with the following schema. Note that this will bump the
inter.broker.protocol.version.

...