This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state:  "Under Discussion"

...

JIRA: KAFKA-7362

Motivation

When partition reassignment removes topic partitions from a offline broker, those removed partitions become orphan partitions to the broker. When the offline broker comes back online, it is not able to clean up both data and folders that belong to orphan partitions.  Log manager will scan all log dirs during startup, but the time based retention policy on a topic partition will not be kicked out until the replicaHighWatermark  has been set. replicaHighWatermark  replicaHighWatermark of a partition is set when the broker becomes either leader or follower of the partition.  Orphan   Orphan partitions will never have chance to get replicaHighWatermark set.  In addition, we do not have logic to delete folders that belong to orphan partition today.  This KIP provides a mechanism for brokers to remove orphan partitions automatically. 

Public Interfaces

...

• Add a broker config "auto.orphan.partition.removal.delay.ms" :  the delay after which orphan partitions will start to be removed. The timer starts when the broker receives the first leaderAndISR request.  The default value is set to -1, which indicates the orphan partition removal is disabled.

  
  

Proposed Changes

a) Provide a mechanism to remove orphan partition automatically. 

...

1 ) Manual deletion of orphan partitions via provided API. Kafka provides a new API for orphan partition deletion and require manual trigger of partition deletion. Kafka   Kafka provides an API that the user can specify what topic partitions that he wants to delete and what is the time retention rule. Kafka only remove a partition if all the following conditions are met:

...