Page History

...

Currently, the first LeaderAndIsrRequest sent to a broker by a controller contains all TopicPartitions that a broker is a replica for. We will formalize this behavior by also including an is_every_partition flag to denote when this is one such request. When is_every_partition = true, the broker can reconcile its local state on disk with the request, and safely stage deletions for any partitions that are present on disk and are not contained in the request. This may include cases where a TopicPartition is not present in the LeaderAndIsrRequest, or it may be due to a topic partition containing a topic ID that does not match the local topic partition stored on the broker. Such reconciliation may also be necessary if is_every_partition = false, and the topic ID set on a partition does not match the ID contained in the request.

Deletions will be staged by logging a warning and then moving the partition directory to deleting directory under resulting from LeaderAndIsrRequests will:

1. Log a warning
2. Move the partition's directory to log.dir/deleting/{topic_id}_{partition}.

...

2. A final deletion event will

...

1. be scheduled for X ms after the LeaderAndIsrRequest was first received. This will clear the deleting directory of the partition's files.

Migration

When a broker receives a LeaderAndIsrRequest containing a topic ID for an existing partition without an associated topic ID, it will associate the topic ID with the partition. This will effectively migrate a broker's local replicas to include topic IDs.

...

This strategy would allow brokers to effectively delete topics immediately, ensuring deletions do not block creation and use of a new topic with the same name. This is the proposed option to optional proposed by this KIP, that will clean up the deletion logic and would not block topic recreationsimplify the topic deletion and creation flow.

Upon receiving a DeleteTopicsRequest, if the IBP is >= MIN_TOPIC_ID_VERSION move the /brokers/topics/[topic] znode payload to /admin/delete_topics_by_id/[topicId], and immediately reply to the DeleteTopicsRequest with a successful response. At this point, the topic is considered deleted, and a topic with the same name can be created.

...

We need the changes to FetchRequest/ListOffsetRequest/OffsetsForLeaderEpochRequest described above earlier to make the above scenario safe. By including the topic ID in these requests, we can prevent a broker from accidentally replicating a stale version of the topic.

...