Versions Compared

Old Version 116

changes.mady.by.user Grant Henke

Saved on

compared with

New Version 117

changes.mady.by.user Grant Henke

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

 

MetadataResponse => [brokers] controllerId [topic_metadata]   
  brokers => node_id host port rack 
    node_id => INT32
    host => STRING
    port => INT32
    rack => NULLABLE_STRING
  controllerId => INT32
  topic_metadata => topic_error_code topic is_internal [partition_metadata] 
    topic_error_code => INT16
    topic => STRING
    is_internal => BOOLEAN
    partition_metadata => partition_error_code partition_id leader [replicas] [isr] 
      partition_error_code => INT16
      partition_id => INT32
      leader => INT32
      replicas => INT32
      isr => INT32

Adds rack, controller_id, and is_internal to the version 0 response.

The behavior of the replicas and isr arrays will be changed in order to support the admin tools, and better represent the state of the cluster:

  • In version 0, if a broker is down the replicas and isr array will omit the brokers entry and add a REPLICA_NOT_AVAILABLE error code.
  • In version 1, no error code will be set and a the broker id will be included in the replicas and isr array. 
    • Note: A user can still detect if the replica is not available, by checking if the broker is in the returned broker list.

Topic Admin Schema

Create Topic Request (KAFKA-2945)

 

CreateTopic Request (Version: 0) => [create_topic_requests] timeout 
  create_topic_requests => topic partitions replication_factor [replica_assignment] [configs] 
    topic => STRING
    partitions => INT32
    replication_factor => INT32
    replica_assignment => partition_id [replicas] 
      partition_id => INT32
      replicas => INT32
    configs => config_key config_value 
      config_key => STRING
      config_value => STRING
  timeout => INT32

CreateTopicRequest is a batch request to initiate topic creation with either predefined or automatic replica assignment and optionally topic configuration.

Request semantics:

  1. Must be sent to the controller broker
  2. Multiple instructions for the same topic in one request will be silently ignored, only the last from the list will be executed.
    • This is because the list of topics is modeled server side as a map with TopicName as the key
  3. The principle must be authorized to the "Create" Operation on the "Cluster" resource to create topics. 
    • Unauthorized requests will receive a ClusterAuthorizationException

  4. Only one from ReplicaAssignment or (Partitions + ReplicationFactor), can be defined in one instruction. If both parameters are specified - ReplicaAssignment takes precedence.

    • In the case ReplicaAssignment is defined number of partitions and replicas will be calculated from the supplied ReplicaAssignment. 
    • In the case of defined (Partitions + ReplicationFactor) replica assignment will be automatically generated by the server.
    • One or the other must be defined. The existing broker side auto create defaults will not be used (default.replication.factor, num.partitions). The client implementation can have defaults for these options when generating the messages.
  5. Setting a timeout > 0 will allow the request to block until the topic metadata is "complete" on the controller node.
    • Complete means the topic metadata has been completely populated (leaders, replicas, ISRs)
    • If a timeout error occurs, the topic could still be created successfully at a later time. Its up to the client to query for the state at that point.
  6. The request is not transactional. 
    1. If an error occurs on one topic, the other could still be created.
    2. Errors are reported independently.

QA:

  • Why is CreateTopicRequest a batch request?
    • Scenarios where tools or admins want to create many topics should be able to with fewer requests
    • Example: MirrorMaker may want to create the topics downstream
  • What happens if some topics error immediately? Will it return immediately? 
    • The request will block until all topics have either been created, errors, or the timeout has been hit
    • There is no "short circuiting" where 1 error stops the other topics from being created
  • Why implement "partial blocking" instead of fully async of fully consistent?
  • Why require the request to go to the controller?
    • The controller is responsible for the cluster metadata and its propogation 
    • See Request Forwarding below

...