Versions Compared

Old Version 5

changes.mady.by.user Lucas Bradstreet

Saved on

compared with

New Version 6

changes.mady.by.user Lucas Bradstreet

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Today, Kafka uniquely identifies a topic by its name. This is generally sufficient, but there are flaws in this scheme if a topic is deleted and recreated with the same name. As a result, Kafka attempts to prevent these classes of stale topic issues by ensuring a topic is deleted from all replicas before completing a deletion. This solution is not perfect, as it is possible for partitions to be reassigned from brokers while they are down, and there are no guarantees that this state will ever be cleaned up and will not cause issues in the future.

As the controller must wait for all replicas to delete their local partitionsreplicas, deletes can also become blocked, preventing topics from being created with the same name until the deletion is complete on all replicas. This can mean that downtime for a single broker can effectively cause a complete outage for everyone producing/consuming to that topic name, as the topic cannot be recreated without manual intervention.

Topic IDs aim to address this issue by associating a truly unique ID with each topic, ensuring a newly created topic with a previously used name cannot be confused with a previous topic with that name.

...


LeaderAndIsrRequest v3 adds the topic ID to the topic_states field, as well and a boolean flag for to denote whether this is the initial "full" LeaderAndIsrRequest sent by a controller.

The first LeaderAndIsrRequest sent to a broker contains all TopicPartitions that a broker is a replica for. If is_every_partition = true, the broker can reconcile its local state, and safely stage deletions for any partitions that are present on disk and are not contained in the request. This may include cases where a topic_partition combination is not present in the LeaderAndIsrRequest, or it may be due to a conflicting topic partition, with a topic ID that does not match the topic partition stored on the broker. Such reconciliation may also be necessary if is_every_partition = false, and the topic ID set on a partition does not match the ID contained in the request.

...

When a broker receives a LeaderAndIsrRequest containing a topic ID for an existing partition without a an associated topic ID, it will associate the topic ID with the partition. This will effectively migrate a broker's local replicas to include a topic IDIDs.

LeaderAndIsrResponse v3

LeaderAndIsr Response (Version: 3) => error_code [partitions]
  error_code => INT16
  partitions => topic topic_id* partition error_code
    topic => STRING
    topic_id* => UUID
    partition => INT32
    error_code => INT16

...

StopReplica Request (Version: 2) => controller_id controller_epoch broker_epoch delete_partitions [partitions]
  controller_id => INT32
  controller_epoch => INT32
  broker_epoch => INT64
  delete_partitions => BOOLEAN
  partitions => topic topic_id* [partition_ids]
    topic => STRING
    topic_id* => UUID
    partition_ids => INT32

...

StopReplicaResponse v2

StopReplica Response (Version: 2) => error_code [partitions]
  error_code => INT16
  partitions => topic topic_id* partition error_code
    topic => STRING
    topic_id* => UUID
    partition => INT32
    error_code => INT16

...

DeleteTopics

The controller can could now be implemented to respond to a DeleteTopicsRequest in one of several the following ways:

Existing strategy

In this approach, we would maintain the current delete logic. The controller will send a StopReplicaRequest to all replicas for a topic, and will keep retrying this request until all replicas respond successfully. In this implementation, the deletion logic will not be simplified.

...