Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

New Protocol Errors

It is proposed to add these error codes to the protocol.

Error

Code

Description

Requests
TopicAlreadyExists1001Topic with this name already exists.CreateTopicRequest
InvalidArgumentPartitions1002Either partition field is invalid (e.g. negative), or not defined when needed.CreateTopicRequest, AlterTopicRequest
DecreasePartitionsNotAllowed1003Invalid partitions argument: decreasing partitions is prohibited.AlterTopicRequest
InvalidArgumentReplicationFactor1004Either replication-factor field is invalid (e.g. negative), or not defined when needed.CreateTopicRequest
InvalidArgumentReplicaAssignment1005Either replication-factor field is invalid (e.g. contains duplicates), or not defined when needed.

CreateTopicRequest, AlterTopicRequest,

ReassignPartitionsRequest

InvalidTopicConfig1006

Either topic-level config setting or value is incorrect.

CreateTopicRequest, AlterTopicRequest
PreferredReplicaLeaderElectionInProgress1007Preferred replica leader election procedure has been already started.PreferredReplicaLeaderElectionRequest
InvalidArgumentPreferredReplicaElectionData1008Preferred replica leader election data is in invalid (bad json, duplicates etc).PreferredReplicaLeaderElectionRequest
ReassignPartitionsInProgress1009Reassign partitions procedure has been already started.ReassignPartitionsRequest

Generally, the Admin Client (see section 3) or another request dispatcher should have enough context to provide descriptive error message.

Topic

Admin Schema

The Overall the idea is to introduce Wire protocol messages that extend existing Wire Protocol to cover all existing topic commands (- create-topic, alter-topic, delete-topic, describe-topic, list, describe). The motivation behind the proposed schema is the following:

1) Topic commands must inherit options from TopicCommand tool

2) If some of the options are not used in particular command (e.g. ReplicaAssignment in CreateTopicRequest) - the special marker value is used instead (e.g. in case of ReplicaAssignment - empty string)

3) Topic commands must support batching and provide command execution result per-topic

4) Topic commands are asynchronous - the request to create/alter/delete just initiates the corresponding commands and returns immediately-topics, reassign-partitions, preferred-replica-leader-election. At the same time, since Wire Protocol is a public API to Kafka cluster, it was agreed that new Admin schema needs to be "orthogonal", i.e. new messages shouldn't duplicate each other or existing messages, if those already cover particular use cases. It is proposed to map existing command to Wire Protocol in the following way:

 

CommandWire Protocol MessageNote
create-topicCreateTopicRequest 
alter-topicAlterTopicRequest 
delete-topicDeleteTopicRequest 
describe-topicTopicMetadataRequest_V1Using new version on TopicMetadataRequest which will include topic-level config
list-topicsTopicMetadataRequestUsing an empty list as input for request, which results in returning metadata for all existing topcis
reassign-partitionsAlterTopicReqeustUsing batch AlterTopicRequest with replica assignment specified
preferred-replica-leader-electionPreferredReplicaLeaderElectionRequest 

 

It is also important that all Admin requests will be asynchronous. This means requests will only initiate particular command, and will not wait until the command is actually completed. There are different reasons to that, but we wan't to make sure that we give users all needed tools to verify whether issues command has been completed. E.g. one can consider topic is created once it is possible to consumer from / produce to this topic. In this case user can leverage TopicMetadataRequest to check all brokers received metadata about the newly created topic.

Finally, Topic Admin schema requests are likely to be used not only in CLI tool, where the common use case in create/change/delete/get a single entity. Since Kafka is able to maintain a huge number of topics it is important that user can efficiently issue many requests at one time. That's why all Topic Admin messages are batch requests, i.e. it is possible to group commands of one type for many topics in one batch reducing network calls.

Create Topic

CreateTopic schema is a 

Create Topic Request

 

CreateTopicRequest => [TopicName Partitions Replicas ReplicaAssignment [ConfigEntry]]
TopicName => string
Partitions => int32
Replicas => int32
ReplicaAssignment => [PartitionId [ReplicaId]]
ConfigEntry => ConfigKey ConfigValue
ConfigKey => string
ConfigValue => string
CreateTopicRequest requires topic name and either (Partitions+Replicas) or ReplicasAssignment to create a topic. A special value -1 should be used to denote an empty value for Partitions and ReplicasAlso user will be able to specify topic-level configs for the created topic (to use default an empty array should be provided).

The (Partitions, Replicas)/ReplicaAssignment semantics is the following:

1) If ReplicaAssignment is specified - Partitions and Replicas are not taken into account, topic is created with provided replica assignment and number of topics and replication factor are defined from ReplicaAssignment
2) If ReplicaAssignment is empty - number of topic partitions and replication factor must be defined with Partitions and Replicas, the replica assignment for topic is automatically generated on server

...