Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Currently, log manager chooses a directory configured in `log.dirs` by calculating the number partitions in each directory and then choosing the one with the fewest partitions. But in some real production scenarios where data volumes of partitions are not even, some disks become nearly full whereas the others have a lot of spaces which lead to a poor data distribution.

We should could offer a new strategy to users to have log manager honor the real disk free spaces and choose the directory with the most disk space

Public Interfaces

This log directory selection strategy, which is implemented on the broker side, would add a new broker configuration parameter 'log.direcotry.select.strategy' that specifies which strategy is used to determine the log directory in which new log is created. Two possible values are :

  • partition:Same as the current design. LogManager selects log directory based on the partition number on each directories
  • size: Kafka select 

 


Maybe a new broker configuration parameter is needed, `log.directory.strategy` for instance.

...

This assignment strategy, which is implemented for the new consumer, would add a StickyAssignor class that can be used as org.apache.kafka.clients.consumer.StickyAssignor for the value of the consumer property partition.assignment.strategy. It would not affect the default value of this consumer property.

Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.

...