You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 5 Next »

This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: Under Discussion

Discussion thread:

JIRA: Unable to render Jira issues macro, execution error.

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

This is a proposal for an admin tool - say, kafka-brokers.sh to provide broker related useful information. Some of the key factors for Kafka's success are its performant architecture and operational simplicity. This is further complemented with a set of commandline tools and utilities for managing topics as well as testing/stress-testing. However currently Kafka lacks commands/tools to get cluster and broker overview. Although it should be mentioned that Kafka does expose cluster information via API and broker metrics via JMX.

Public Interfaces

The kafka-broker.sh command is modeled after the kafka-topic.sh and has the following options:

--zookeeper <zookeeper,port> : zookeeper URL

--rack <list of comma-seperated racks>: filter details on brokers in the list of racks only

-- topic <list of comma-seperated topcis>: : filter details on brokers that serve partitions for the provided topic

 

Proposed Changes

The command will essentially provide the following pieces of information:

Broker Information

  • Broker Id
  • Hostname
  • Rack
  • Endpoints
  • Topic count (i.e. how many topics are hosted by the broker)
  • All partition count (i.e. how many partition replicas are hosted by the broker)
  • Leader partition count (i.e. how many leader partitions are hosted by the broker)
  • Insync partition count (i.e, how many follower partitions are insync)
  • Under-replicated partition count (i.e. how many partitions are not insync)
  • An asterisk would indicate that the broker is the leader for that partition.

With commandline options, topic names and partition numbers can be enumerated.

Command Options

The command kafka-brokers.sh requires zookeeper information and additional accepts other options as show below:

kafka-brokers.sh Usage/Options
$ ./kafka-brokers.sh 
Missing required argument "[zookeeper]"
Option                         Description                           
------                         -----------                           
--broker <Integer: broker-id>  Filter for a broker. Option can be    
                                 used multiple times for multiple    
                                 broker-ids                          
--details                      if specified, shows detailed listing  
                                 of topics and partitions            
--host <hostname>              Filter for a hostname. Option can be  
                                 used multiple times for multiple    
                                 hostnames                           
--partition-details            if specified, shows partitions in each
                                 topic                               
--rack <rack>                  Filter for a rack. Option can be used 
                                 multiple times for multiple racks   
--topic <topic>                Filter for a topic. Option can be used
                                 multiple times for multiple topics  
--topic-details                if specified, shows topics and        
                                 partition counts in each topic      
--zookeeper <zkurl>            REQUIRED: The connection string for   
                                 zookeeper(s) in the form host:port, 
                                 host:port/zkchroot or host1,host2,  
                                 host3:port/zkchroot.

 

The options "–broker", "–host", "–rack" and "–topic" provide filtering for the specified broker (broker-id), host, rack or topic. Each of the options can be specified multiple times. The option "--topic-details" makes the command to enumerate all the topic names in addition to giving the topic count. The option "--partition-details" makes the command to enumerate all the partitions for each topic. The partitions have an an optional prefix of "+" or "-" to indicate that the partition is a leader partition or an under-replicated partition respectively. An insync replica partition will not have any prefix.

Here are examples usages of the command.

Summary Output Without Any Details
$ ./kafka-brokers.sh --zookeeper zk1,zk2,zk3:2181
BrokerId: 1	Hostname: host1	Rack: rack1	Endpoints: (host1,9092,PLAINTEXT), (host1,9093,SSL)	Topics: 2	Partitions: 12	Leaders: 4	InSync: 8	Trailing: 0	
BrokerId: 2	Hostname: host2	Rack: rack2	Endpoints: (host2,9092,PLAINTEXT), (host2,9093,SSL)	Topics: 2	Partitions: 12	Leaders: 4	InSync: 8	Trailing: 0	
BrokerId: 3	Hostname: host3	Rack: rack3	Endpoints: (host3,9092,PLAINTEXT), (host3,9093,SSL)	Topics: 2	Partitions: 12	Leaders: 4	InSync: 8	Trailing: 0
Output with Topic Details
BrokerId: 1	Hostname: host1	Rack: rack1	Endpoints: (host1,9092,PLAINTEXT), (host1,9093,SSL)	Topics: 2	Partitions: 12	Leaders: 4	InSync: 8	Trailing: 0	Topic Details: topic2 with 6 partitions, topic1 with 6 partitions	
BrokerId: 2	Hostname: host2	Rack: rack2	Endpoints: (host2,9092,PLAINTEXT), (host2,9093,SSL)	Topics: 2	Partitions: 12	Leaders: 4	InSync: 8	Trailing: 0	Topic Details: topic2 with 6 partitions, topic1 with 6 partitions	
BrokerId: 3	Hostname: host3	Rack: rack3	Endpoints: (host3,9092,PLAINTEXT), (host3,9093,SSL)	Topics: 2	Partitions: 12	Leaders: 4	InSync: 8	Trailing: 0	Topic Details: topic2 with 6 partitions, topic1 with 6 partitions
Output with Partition Details
BrokerId: 1	Hostname: host1	Rack: rack1	Endpoints: (host1,9092,PLAINTEXT), (host1,9093,SSL)	Topics: 2	Partitions: 12	Leaders: 4	InSync: 8	Trailing: 0	Partition Details: (topic2: 5,4,+3,2,1,+0), (topic1: 5,4,+3,2,1,+0)	
BrokerId: 2	Hostname: host2	Rack: rack2	Endpoints: (host2,9092,PLAINTEXT), (host2,9093,SSL)	Topics: 2	Partitions: 12	Leaders: 4	InSync: 8	Trailing: 0	Partition Details: (topic2: 5,+4,3,2,+1,0), (topic1: 5,+4,3,2,+1,0)	
BrokerId: 3	Hostname: host3	Rack: rack3	Endpoints: (host3,9092,PLAINTEXT), (host3,9093,SSL)	Topics: 2	Partitions: 12	Leaders: 4	InSync: 8	Trailing: 0	Partition Details: (topic2: +5,4,3,+2,1,0), (topic1: +5,4,3,+2,1,0)
Output with Topic and Partition Details
BrokerId: 1	Hostname: host1	Rack: rack1	Endpoints: (host1,9092,PLAINTEXT), (host1,9093,SSL)	Topics: 2	Partitions: 12	Leaders: 4	InSync: 8	Trailing: 0	Topic Details: topic2 with 6 partitions, topic1 with 6 partitions	Partition Details: (topic2: 5,4,+3,2,1,+0), (topic1: 5,4,+3,2,1,+0)	
BrokerId: 2	Hostname: host2	Rack: rack2	Endpoints: (host2,9092,PLAINTEXT), (host2,9093,SSL)	Topics: 2	Partitions: 12	Leaders: 4	InSync: 8	Trailing: 0	Topic Details: topic2 with 6 partitions, topic1 with 6 partitions	Partition Details: (topic2: 5,+4,3,2,+1,0), (topic1: 5,+4,3,2,+1,0)	
BrokerId: 3	Hostname: host3	Rack: rack3	Endpoints: (host3,9092,PLAINTEXT), (host3,9093,SSL)	Topics: 2	Partitions: 12	Leaders: 4	InSync: 8	Trailing: 0	Topic Details: topic2 with 6 partitions, topic1 with 6 partitions	Partition Details: (topic2: +5,4,3,+2,1,0), (topic1: +5,4,3,+2,1,0)

 

Compatibility, Deprecation, and Migration Plan

None yet.

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.

  • No labels