Page History

...

Introduce new cleanup policy, compact_and_delete. When set, both compact and delete cleanup strategies will run.

 Implementation outline

Currently for The LogCleaner.CleanerThread is currently responsible for triggering the cleaning of topics with cleanup.policy=delete, the delete is triggered from a scheduled thread in LogManager. Every interval it calls cleanupLogs which deletes segments such that it respects both the retention.size and retention.ms configs. In this approach we’d remove this scheduled thread and relevant functions in LogManager and perform the cleanup via LogCleaner.CleanerThread. We will extend this to also support cleanup.policy=compact_and_delete. In the cleanOrSleep method we’d first run compaction and then run deletion. We’d need to add some extra code to Log to check if we have segments ready to be deleted and add any that are ready to the inProgress map (so we don’t get multiple threads trying to delete the same segments), run the delete operation, and then remove them from the inProgress map (this is the same as it currently works for compacted logs).

There is no change for topics with cleanup.policy=delete, i.e, the cleanup will still be scheduled via LogManager. The benefits of this approach are that it requires no further locking, all compacted topic cleaning is triggered from LogCleaner.CleanerThread and topics that are cleanup.policy=delete are not impacted. 

...