This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.
Status
Current state: Under Discussion
Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: KAFKA-1646
Motivation
Currently, when create on LogSegment, always create on empty file and keep APPEND data to it, for Linux file system (ext2/ext3/ext4 etc), it works fine. But for windows and some old unix/linux file system, after a while, there will be more and more fragments on hard disk, and affect consume performance a lot. So if we pre allocate file with one bigger value ( for example, 512MB) when create file, it will help us reduce fragments on hard disk and improve consume performance.
Public Interfaces
This proposal add one config.
log.preallocate - Should pre allocate file when create new segment? Default value is false for backward compatible.
Proposed Changes
- Configuration - add one configuration item "log.preallocate", parse it in KafkaConfig.scala, and transfer to KafkaServer.scala, LogConfig.scala.
- In Log.scala
- add one function initFileSize(). if the log preallocation is enabled, the value is config.segmentSize - 2 * config.maxMessageSize, else the value is 0.
- pass the initFileSize and config.preallocate to LogSegment when need create one LogSegment.
- When need roll to a new file, trim log file of active segment
- In LogSegment.scala - pass the initFileSize and preallocation to FileMessageSet.
- In FileMessageSet.scala
- add one function trim(). Call it when close or roll a new LogSegment.
- Move openChannel from CoreUtil.scala and add two more parameter "initFileSize" and "preallocate".
- Add one constructor function with two more parameters "initFileSize" and "preallocate".
Compatibility, Deprecation, and Migration Plan
This change is fully backward compatible.
Rejected Alternatives