Versions Compared

Old Version 38

changes.mady.by.user Calvin Liu

Saved on

compared with

New Version Current

changes.mady.by.user Calvin Liu

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

{
  "apiKey":62,
  "type": "request",
  "listeners": ["controller"],
  "name": "BrokerRegistrationRequest",
  "validVersions": "0-2",
  "flexibleVersions": "0+",
  "fields": [
...
// New fields begin.
    { "name": "PreviousBrokerEpoch", "type": "int64", "versions": "2+", "default": "-1",
      "about": "The epoch before a clean shutdown." }
// New fields end.
  ]
}

...

DescribeTopicPartitionsRequest (Coming with ELR)

Should be issued by admin clients. More admin client related details please refer to the Admin API/Client changes

ACL: Describe Topic

Limit: 1000 partitions max per response.

The caller can query the partitions starting from the first partition id.list the topics interested or keep the field empty if requests all of the topics.

Pagination.

This is a new behavior introduced. The caller can specify the maximum number of If the server finds more than 1000 partitions to be included , only the first 1000(alphabet order) will be returned with infoin the response.

If

...

there are more partitions than the limit, these partitions and their topics will not be sent back. In this case, the Cursor field will be populated. The caller can include this cursor in the next request. 

Note,

  • There is also a server-side config to control the maximum number of partitions to return. max.request.partition.size.limit
  • There is no consistency guarantee between requests.
  • It is an admin client facing API, so there is no topic id supported.
{
  "apiKey": 74,
  "type": "request",
  "listeners": ["broker"],
  "name": "DescribeTopicPartitionsRequest",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "Topics", "type": "[]TopicRequest", "versions": "0+",
      "about": "The topics to fetch details for.",
      "fields": [
        { "name": "Name", "type": "string", "versions": "0+",
          "about": "The topic name", "versions": "0+", "entityType": "topicName"}
      ]
    },
    { "name": "ResponsePartitionLimit", "type": "int32", "versions": "0+", "default": "2000",
      "about": "The maximum number of partitions included in the response." },
    { "name": "Cursor", "type": "Cursor", "versions": "0+", "nullableVersions": "0+", "default": "null",
      "about": "The first topic and partition index to fetch details for.", "fields": [
      { "name": "TopicName", "type": "string", "versions": "0+",
        "about": "The name for the first topic to process", "versions": "0+", "entityType": "topicName"},
      { "name": "PartitionIndex", "type": "int32", "versions": "0+", "about": "The partition index to start with"}
    ]}
  ]
}

DescribeTopicsResponse

{
  "apiKey": 74,
  "type": "response",
  "name": "DescribeTopicPartitionsResponse",
 

Note, the request can have a mix of partition-specific topics and range-query topics.

{
  "apiKey":71,
  "type": "request",
  "listeners": ["broker"],
  "name": "DescribeTopicsRequest",
  "validVersions": "0",
  
  "flexibleVersions": "0+",
  
  "fields": [
    
    { "name": "TopicsThrottleTimeMs", "type": "[]TopicRequest", "versions": "0+",
      "about": "The topics to fetch details for.int32", "versions": "0+", "entityTypeignorable": "topicRequest",
      "fields": [
         true,
      "about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
    { "name": "NameTopics", "type": "string[]DescribeTopicPartitionsResponseTopic", "versions": "0+",
      
       "about": "TheEach topic name", "versions": "0+", "entityType": "topicName"}
        in the response.", "fields": [
      { "name": "FirstPartitionIdErrorCode", "type": "int32", "default": "0"int16", "versions": "0+", 
                "about": "The first partition index to fetch details for."}
]} ] }

DescribeTopicsResponse

{
  "apiKey":71,
  "type": "response",
  " topic error, or 0 if there was no error." },
      { "name": "DescribeTopicsResponseName",
   "validVersionstype": "0string",
   "flexibleVersionsversions": "0+",
  "fields": [
    {", "namemapKey": "Topics"true, "typeentityType": "[]TopicResponsetopicName", "versionsnullableVersions": "0+",
      
        "about": "EachThe topic in the responsename." }, "fields": [
      
      { "name": "ErrorCodeTopicId", "type": "int16uuid", "versions": "0+",
         "ignorable": true, "about": "The topic error, or 0 if there was no errorid." },
      
      { "name": "NameIsInternal", "type": "stringbool", "versions": "0+", "mapKeydefault": true"false", "entityTypeignorable": "topicName", "nullableVersions": "0+",
        "true,
        "about": "TheTrue if the topic is nameinternal." },
      
      { "name": "TopicIdPartitions", "type": "uuid[]DescribeTopicPartitionsResponsePartition", "versions": "0+",
  "ignorable": true      "about": "Each partition in the topic.", "aboutfields": "The topic id." },
       [
        { "name": "IsInternalErrorCode", "type": "boolint16", "versions": "0+",
  "default        "about": "false", "ignorable": true,
        "about": "TrueThe partition error, or 0 if thethere topicwas isno internalerror." },

          { "name": "PartitionsPartitionIndex", "type": "[]MetadataResponsePartitionint32", "versions": "0+",
        
          "about": "EachThe partition in the topicindex." }, "fields": [
        
        { "name": "ErrorCodeLeaderId", "type": "int16int32", "versions": "0+",
          "about": "The partition error, or 0 if there was no error." },
        { "name "entityType": "PartitionIndexbrokerId", "type": "int32", "versions": "0+",
          "
          "about": "The partition indexID of the leader broker." },
        
        { "name": "LeaderIdLeaderEpoch", "type": "int32", "versions": "0+", "entityTypedefault": "brokerId",
          -1", "ignorable": true,
          "about": "The leader IDepoch of thethis leader brokerpartition." },
        
        { "name": "LeaderEpochReplicaNodes", "type": "[]int32", "versions": "0+", "defaultentityType": "-1brokerId",
          "ignorableabout": true,
          "about": "The leader epoch of"The set of all nodes that host this partition." },
        
        { "name": "ReplicaNodesIsrNodes", "type": "[]int32", "versions": "0+", "entityType": "brokerId",
          
          "about": "The set of all nodes that host are in sync with the leader for this partition." },
        
        { "name": "IsrNodesEligibleLeaderReplicas", "type": "[]int32", "versionsdefault": "0+null", "entityType": "brokerId",
          "about
          "versions": "0+", "nullableVersions": "The set of nodes that are in sync with the leader for this partition0+",
          "about": "The new eligible leader replicas otherwise." },
            { "name": "EligibleLeaderReplicasLastKnownELR", "type": "[]int32", "default": "null", "entityType": "brokerId",
          
          "versions": "0+", "nullableVersions": "0+",
          
          "about": "The newlast eligible leader replicas otherwiseknown ELR." },

            { "name": "LastKnownELROfflineReplicas", "type": "[]int32", "defaultversions": "null0+", "entityTypeignorable": "brokerId"true,
          "versions": "0+", "nullableVersions": "0+",
         "entityType": "brokerId",
          "about": "The last known ELRset of offline replicas of this partition." }]},
           { "name": "LastKnownLeaderTopicAuthorizedOperations", "type": "int32", "defaultversions": "null0+", "entityTypedefault": "brokerId-2147483648",

              "versionsabout": "0+", "nullableVersions": "0+",
          "about": "The last known leader32-bit bitfield to represent authorized operations for this topic." },]
    },
     { "name": "OfflineReplicasNextTopicPartition", "type": "[]int32Cursor", "versions": "0+", "ignorablenullableVersions": true"0+", "entityTypedefault": "brokerIdnull",
          
      "about": "The next topic setand ofpartition offlineindex replicasto offetch thisdetails partitionfor." },
 "fields": [
         { "name": "TopicAuthorizedOperationsTopicName", "type": "int32string", "versions": "0+",
        "defaultabout": "-2147483648",
The name for the first topic to process", "versions": "0+", "aboutentityType": "32-bit bitfield to represent authorized operations for this topic." }]topicName"},
           { "name": "NextPartitionPartitionIndex", "type": "int32", "versions": "0+", "default": "-1",
          "about": "The first partition thatindex exceedto the request limit. " }]},
]
start with"}
    ]}
  ]
}

CleanShutdownFile (Coming with ELR)

...

  1. eligible.leader.replicas.enabled. It controls whether the controller will record the ELR-related metadata and whether ISR can be empty. False is the default value. It will turn true in the future.
  2. max.request.partition.size.limit. The maximum number of partitions to return in a API response.

The new configs are introduced for Unclean Recovery.

...

  1. The client will split a large request into proper pieces and send them one after another if the requested topics count reaches the limit.
  2. The client will retry querying the topics if they received the new retriable error REQUEST_LIMIT_REACHEDresponse with Cursor field. 
  3. The output of the topic describe will be updated with the ELR related fields.
  4. TopicPartitionInfo will be updated to include the ELR related fields.

...

  • min.insync.replicas will no longer be effective to be larger than the replication factor. For existing configs, the min.insync.replicas will be min(min.insync.replicas, replication factor).

  • Cluster admin should update the min.insync.replicas to 1if they want to have the replication going when there is only the leader in the ISR.

  • Note that, this new requirement is not guarded by any feature flags/Metadata version.

ELR

It will be guarded by a new metadata version and the eligible.leader.replicas.enabled. So it is not enabled during the rolling upgrade.

...