Page History

...

Currently, in Kafka Core, the checkpointing system in log cleaner works like this. One checkpoint file is created per disk, and then each topic partition will have an assigned file disk into which log cleaner's cleaning offsets are written. In most use cases, the number of partitions far exceed the number of disks. Thus, due to this organization, one file stores cleaning offsets from multiple topic partitions.  We wish to improve upon this system by switching to a per-partition checkpoint file policy. In this case, each topic partition will write into one designated checkpoint file (meaning no other partitions would have their cleaning offsets written into the same file). This has a number of advantages since we no longer have to store which topic partition a cleaning offset belongs to in a checkpoint file. 

...