Page History

...

1. Replace all local storage with remote storage - Instead of using local storage on Kafka brokers, only remote storage is used for storing log segments and offset index files. While this has the benefits related to reducing the local storage, it has the problem of not leveraging the OS page cache and local disk for efficient latest reads as done in Kafka today. 
2. Implement Kafka API on another store - This is an approach that is taken by some vendors where Kafka API is implemented on a different distributed, scalable storage (example HDFS). Such an option does not leverage Kafka other than API compliance and requires the much riskier option of replacing the entire Kafka cluster with another system.
3. Client directly reads remote log segments from the remote storage - The log segments on the remote storage can be directly read by the client instead of serving it from Kafka broker. This reduces Kafka broker changes and has benefits of removing an extra hop. However, this bypasses Kafka security completely, increases Kafka client library complexity and footprint, causes compatibility issues to the existing Kafka client libraries, and hence is not considered. 
4. Store all remote segment metadata in remote storage. This approach works with the storage systems that provide strong consistent metadata, such as HDFS, but does not work with S3 and GCS. Frequently calling LIST API on S3 or GCS also incurs huge costs. So, we choose to store metadata in a Kafka topic in the default implementation but allow users to use other methods with their own RLMM implementations.
5. Cache all remote log indexes in local storage. Store remote log segment information in local storage. 

   

Meeting Notes

09 Feb 2021

• Notes
  • Discussed the downgrade path, KIP will be updated with that.
  • Discussed the limitation of not allowing disable tiered storage on a topic.
  • All are agreed that KIP is ready for voting.

26 Jan 2021

• Notes
  • Discussed the latest review comments from the mail thread.
    
  • Manikumar will review and provide comments.
    

12 Jan 2021

• Notes
  • Satish discussed the edge cases around upgrade path with KIP-516 updates. Jun clarified on how topic-id is received after IBP is udpated on all brokers.

  • Jun suggested to update the KIP with more details on Remote Partition Remover.

  • RLMM flat file format was discussed and Jun asked to clarify the header section.

  • Kowshik and Jun will provide Log layer refactoring writeup.

15 Dec 2020

• Notes
  • Discussed producer snapshot fix missing in 2.7
  • Satish discussed memory growth due to RLMM cache and it looks to be practically negligible. The proposal is to use inmemory cache and checkpoint that to disk.
  • Satish will update the KIP with Upgrade path.
  • Kowshik and Jun will look into LOg refactoring.

08 Dec 2020

•  Discussion Recording
• Notes

  1. Tiered storage upgrade path dicussion:

  • Details need to be documented in the KIP.
  • Current upgrade path plan is based on IBP bump.
  • Enabling of the remote log components may not mean all topics are eligible for tiering at the same time.
  • Should tiered storage be enabled on all brokers before enabling it on any brokers?
  • Is there any replication path dependency for enabling tiered storage?

  
  

  2. RLMM persistence format:

  • We agreed to document the persistence format for the materialized state of default RLMM implementation (topic-based).
  • (carry over from earlier discussion) For the file-based design, we don't know yet the % of increase in memory, assuming the majority of segments are in remote storage. It will be useful to document an estimation for this.

  
  

  3. Topic deletion lifecycle discussion:

  • Under topic deletion lifecycle, step (4) it would be useful to mention how the RemotePartitionRemover (RPRM) gets the list of segments to be deleted, and whether it has any dependency with the RLMM topic.

  
  

  4. Log layer discussion:

  • We discussed the complexities surrounding making code changes to Log layer (Log.scala).
  • Today, the Log class holds attributes and behavior related with local log. In the future, we would have to change the Log layer such that it would also contain the logic for the tiered portion of the log. This addition can pose a maintenance challenge.
  • Some of the existing attributes in the Log layer such as LeaderEpochCache and ProducerStateManager can be related with global view of the log too (i.e. global log is local log + tiered log). It can be therefore useful to think about preparatory refactoring, to see whether we can separate responsibilities related with the local log from the tiered log, and, perhaps provide a global view of the log that combines together both as and when required. The global view of the log could manage the lifecycle of LeaderEpochCache and ProducerStateManager.
  
  Follow-ups:
  
  • KIP-405 updates (upgrade path, RLMM file format and topic deletion)
  • Log layer changes

  
  (Notes taken by Kowshik)

...