Status

Current state: Under Discussion

Discussion thread: Thread

JIRA: KAFKA-8522 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Note: A pull request has been created for this KIP. (See here: PR#7175)

Changes

Currently, in Kafka Core, the checkpointing system in log cleaner works like this. One checkpoint file is created per disk, and then each topic partition will have an assigned disk into which log cleaner's cleaning offsets are written. In most use cases, the number of partitions far exceed the number of disks. Thus, due to this organization, one file stores cleaning offsets from multiple topic partitions.  We wish to improve upon this system by switching to a per-partition checkpoint file policy. In this case, each topic partition will write into one designated checkpoint file (meaning no other partitions would have their cleaning offsets written into the same file). This has a number of advantages since we no longer have to store which topic partition a cleaning offset belongs to in a checkpoint file. 

Another change we wish to introduce is also store the cleaning time in addition to the offsets in the checkpoint file. This helps resolve a long-standing bug regarding the retention of tombstones in Kafka log (the cleaning time is used as a more accurate indicator of when the segment was last modified, see KAFKA-8522 for more details).  

Upgrade Path

It is possible that when the user upgrades Kafka to a newer version, information could still be stored in the old checkpoint files. In which case, an upgrade path must be provided. 

When a new log cleaner instance is created, we propose to read any remaining data that is stored in the old checkpoint files and rewrite the offsets we have read into the new file system we implemented. Any operations that are then carried out will no longer use the old checkpoint file system but the new one.   

Rejected Alternatives

None as of the moment.