Status

Current state: ["Under DiscussionAccepted"]

Discussion thread: here

JIRA: KAFKA-17928 and KAFKA-17980

Motivation

RemoteLogManager uses multiple thread-pools to copy the segments, delete the expired segments, and read the data from remote storage. There can be situation to tune the number of threads in those pool to manage the incoming load. The proposal is to make them dynamic. 

...

• Make the below configs in the RemoteLogManager to be updated dynamically. Once they are updated, the respective thread-pool will be resized:
  • remote.log.manager.copier.thread.pool.size
  • remote.log.manager.expiration.thread.pool.size size and
  • remote.log.reader.threads andremote.log.manager.thread.pool.size (deprecated in KIP-950, we can skip making this config as dynamic)
• Add a new method isReady in RemoteLogMetadataManager.

...