Versions Compared

Old Version 58

changes.mady.by.user Justine Olshan

Saved on

compared with

New Version 59

changes.mady.by.user Justine Olshan

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

When type = FULL, the broker is able to reconcile its local state on disk with the request. Any partition not contained in this request and present on local disk can be staged for deletion. There are two such types of stale request. In both cases the broker's topic will be staged for deletion.

1. The TopicPartition is not present in the LeaderAndIsrRequest.

...

Reconciliation may also be necessary if type = INCREMENTAL and the topic ID set on a local partition does not match the topic ID contained in the request. A TopicPartition with the same name and a different topic ID by implies that the local topic partition is stale, as the topic must have been deleted to create a new topic with a different topic ID. This is similar to the type 2 stale request above, and the topic will be staged for deletion.

Deletion

Deletion of stale partitions triggered by LeaderAndIsrRequest(s) will take place by:

...

StopReplica Request (Version: 4) => controller_id controller_epoch broker_epoch delete_partitions [topicstopic_states]
  controller_id => INT32
  controller_epoch => INT32
  broker_epoch => INT64
  delete_partitions => BOOLEAN
    topics topic_states => topic topic_id* [partitions]
    topic => STRING_states]
    topic_id* => UUID 
    partitions partition_states => [ partition_ids]id leader_epoch delete_partition
      partition_ids id => INT32

StopReplicaResponse v4

      leader_epoch => INT32
      delete_parition => BOOLEAN

StopReplicaResponse v4

StopReplica Response StopReplica Response (Version: 4) => error_code [topics]
  error_code => INT16
    topics => topic topic_id* [partitions]
    topic => STRING
    topic_id* => UUID 
    partitions => topic_id* partition error_code
      partition => INT32
      error_code => INT16

...

ListOffset Request (Version: 6) => replica_id isolation_level [topics]
  replica_id => INT32
  isolation_level => INT8
  topics => topic topic_id* [partitions]
    topic => STRING
    topic_id* => UUID
    partitions => partition current_leader_epoch timestamp
      partition => INT32
      current_leader_epoch => INT32
      timestamp => INT64

...

ListOffset Response (Version: 6) => throttle_time_ms [responses]
  throttle_time_ms => INT32
  responses => topic topic_id* [partition_responses]
    topic => STRING
    topic_id* => UUID
    partition_responses => partition error_code timestamp offset leader_epoch
      partition => INT32
      error_code => INT16
      timestamp => INT64
      offset => INT64
      leader_epoch => INT32

...

OffsetForLeaderEpoch Request (Version: 4) => replica_id [topics]
  replica_id => INT32
  topics => topic topic_id* [partitions]
    topic => STRING
    topic_id* => UUID 
    partitions => partition current_leader_epoch leader_epoch
      partition => INT32
      current_leader_epoch => INT32
      leader_epoch => INT32

...

OffsetForLeaderEpoch Response (Version: 4) => throttle_time_ms [topics]
  throttle_time_ms => INT32
  topics => topic topic_id* [partitions]
    topic => STRING
    topic_id* => UUID 
    partitions => error_code partition leader_epoch end_offset
      error_code => INT16
      partition => INT32
      leader_epoch => INT32
      end_offset => INT64

...

Metadata schema version: 0

Topic ID: 46bdb63f-9e8d-4a38-bf7b-ee4eb2a794e4


One important use for this file is  the directory structure does not allow us to reload the broker's view of topic ID on startup (perhaps after a failure). It is necessary to persist this file to disk so this information can be reloaded. 

During LeaderAndIsrRequests, this file may be used to disambiguate topics safely and delete topics if necessary. More details on this process are explained in the LeaderAndIsrRequest v5 section.


It will be easy to update the file to include more fields in the future.

...

OptionUnitDefaultDescription
delete.stale.topic.delay.ms ms14400 (4 hours)When a FULL or INCREMENTAL LeaderAndIsrRequest is received and the request does not contain a partition that exists on a broker or a broker's topic ID does not match the ID in the request, a deletion event will be staged for that partition which will complete after delete.stale.topic.delay.ms milliseconds.

...