...

Currently, log manager chooses a directory configured in `log.dirs` by calculating the number partitions in each directory and then choosing the one with the fewest partitions. But in some real production scenarios environments where data volumes of partitions are not even, some disks become we already observed that some directories became nearly full whereas the others have had a lot of spaces which lead to a poor data distribution. The directory with fewest partitions might not be the one with minimum disk spaces.  

We could offer a new strategy to users to have log manager honor the real disk free spaces and choose select the directory one with the most disk space as the next log directory. 

Public Interfaces

This log directory selection strategy, which is implemented on the broker side, would add a new broker configuration parameter 'log.direcotry.select.strategy' that specifies which strategy is used to determine the log directory in which new log is created. Two possible values are :

• partition：Same as the current design . LogManager selects log directory which is based on the partition number numbers on each directories.
• size: Kafka select 

Maybe a new broker configuration parameter is needed, `log.directory.strategy` for instance.

This assignment strategy, which is implemented for the new consumer, would add a StickyAssignor class that can be used as org.apache.kafka.clients.consumer.StickyAssignor for the value of the consumer property partition.assignment.strategy. It would not affect the default value of this consumer property.

Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.

A public interface is any change to the following:

• Binary log format

• The network protocol and api behavior

• Any class in the public packages under clientsConfiguration, especially client configuration

  • org/apache/kafka/common/serialization

  • org/apache/kafka/common

  • org/apache/kafka/common/errors

  • org/apache/kafka/clients/producer

  • org/apache/kafka/clients/consumer (eventually, once stable)

• Monitoring

• Command line tools and arguments

• Anything else that will likely break existing users in some way when they upgrade

Proposed Changes

• the newly introduced strategy which is based on the real disk spaces for each directories.

Proposed Changes

The size-based strategy could employ Java File's getFreeSpace API to retrieve the free space for a directory. If SecurityException is captured,  LogManager could step back and use the partition-count-based strategyDescribe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.

Compatibility, Deprecation, and Migration Plan

...