You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 14 Next »

Here is some information on actually running Kafka as a production system. This is meant as a page for people to record their operational and monitoring practices to help people gather knowledge about successfully running Kafka is production. Feel free to add a section for your configuration if you have anything you want to share. There is nothing magically about most of these configurations, you may be able to improve on them, but they may serve as a helpful starting place.

LinkedIn

Hardware

We are using dual quad-core Intel Xeon machines with 24GB of memory. In general this should not matter too much, we only see pretty low CPU usage at peak even with GZIP compression enabled and a number of clients that don't batch requests. The memory is probably more than is needed for caching the active segments of the log.

The disk throughput is important. We have 8x7200 rpm SATA drives in a RAID 10 array. In general this is the performance bottleneck, and more disks is more better. Depending on how you configure flush behavior you may or may not benefit from more expensive disks (if you flush often then higher RPM SAS drives may be better).

OS Settings

We use linux. Ext4 is the filesystem and we run using software RAID 10. We haven't benchmarked filesystems so other filesystems may be superior.

We have added two tuning changes: (1) we upped the number of file descriptors since we have lots of topics and lots of connections, and (2) we upped the max socket buffer size to enable high-performance data transfer between data centers (described here).

Java

$ java -version
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)

Here are our command line options:

java -server -Xms3072m -Xmx3072m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
     -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -Xloggc:logs/gc.log -Djava.awt.headless=true
     -Dcom.sun.management.jmxremote -classpath <long list of jars>

Kafka

We are running Kafka 0.7 right now but may move to trunk as we fix bugs.

The most important server configurations for performance are those that control the disk flush rate. The more often data is flushed to disk, the more "seek-bound" Kafka will be and the lower the throughput. However we don't hand out data until is is sync'd to disk, so delaying flush also adds some latency. The flush behavior can be set by either giving a timeout (flush at most every 30 seconds, say) or a number of messages (flush every 1000 messages). This can be overridden at the topic level, if desired. The setting always applies to each partition.

The most important client configurations for performance are (1) compression, (2) sync vs async production, (3) batch size for async production, (4) fetch size.

Here is our server configuration:

kafka.log.default.flush.interval.ms=10000
kafka.log.file.size=1073741824
kafka.log.default.flush.scheduler.interval.ms=2000
kafka.log.flush.interval=3000kafka.socket.send.buffer=2097152
kafka.socket.receive.buffer=2097152
kafka.monitoring.period.secs=30
kafka.num.threads=8
kafka.log.cleanup.interval.mins=30
kafka.log.retention.hours=168
kafka.zookeeper.sessiontimeoutms=6000
kafka.zookeeper.connection.timeout=2000
kafka.num.partitions=1

Client configuration varies a fair amount.

Monitoring

Our monitoring is done though a centralized monitoring system custom to LinkedIn, but it keys off the JMX stats exposed from kafka. To see what is available the easiest thing is just to start a kafka broker and/or client and fire up JConsole and take a look.

Server Stats

  • bean name: kafka:type=kafka.SocketServerStats 
      def getProduceRequestsPerSecond: Double
  def getFetchRequestsPerSecond: Double
  def getAvgProduceRequestMs: Double
  def getMaxProduceRequestMs: Double
  def getAvgFetchRequestMs: Double
  def getMaxFetchRequestMs: Double
  def getBytesReadPerSecond: Double
  def getBytesWrittenPerSecond: Double
  def getNumFetchRequests: Long
  def getNumProduceRequests: Long
  def getTotalBytesRead: Long
  def getTotalBytesWritten: Long
  def getTotalFetchRequestMs: Long
  def getTotalProduceRequestMs: Long

  • bean name: kafka:type=kafka.BrokerAllTopicStat kafka:type=kafka.BrokerAllTopicStat.[topic]

      def getMessagesIn: Long
  def getBytesIn: Long
  def getBytesOut: Long
  def getFailedProduceRequest: Long
  def getFailedFetchRequest: Long
  • bean name: kafka:type=kafka.LogFlushStats 
      def getFlushesPerSecond: Double
  def getAvgFlushMs: Double
  def getTotalFlushMs: Long
  def getMaxFlushMs: Double
  def getNumFlushes: Long

Producer stats

  • bean name: kafka:type=kafka.KafkaProducerStats 
      def getProduceRequestsPerSecond: Double
  def getAvgProduceRequestMs: Double
  def getMaxProduceRequestMs: Double
  def getNumProduceRequests: Long
  • bean name: kafka.producer.Producer:type=AsyncProducerStats 
      def getAsyncProducerEvents: Int
  def getAsyncProducerDroppedEvents: Int

Consumer stats

  • bean name: kafka:type=kafka.ConsumerStats 
      def getPartOwnerStats: String
  def getConsumerGroup: String
  def getOffsetLag(topic: String, brokerId: Int, partitionId: Int): Long
  def getConsumedOffset(topic: String, brokerId: Int, partitionId: Int): Long
  def getLatestOffset(topic: String, brokerId: Int, partitionId: Int): Long

  • bean name: kafka:type=kafka.ConsumerAllTopicStat kafka:type=kafka.ConsumerTopicStat.[topic]

      def getMessagesPerTopic: Long
  def getBytesPerTopic: Long
  • bean name: kafka:type=kafka.SimpleConsumerStats 
      def getFetchRequestsPerSecond: Double
  def getAvgFetchRequestMs: Double
  def getMaxFetchRequestMs: Double
  def getNumFetchRequests: Long
  def getConsumerThroughput: Double

Audit

The final alerting we do is on the correctness of the data delivery. We audit that every message that is sent is consumed by all consumers and measure the lag for this to occur. For important topics we alert if a certain completeness is not achieved in a certain time period. The details of this are discussed in KAFKA-260.

Zookeeper

Zookeeper is essential for the correct operation of Kafka. There are a number of things that must be done to keep zookeeper running happily as we have learned the hard way, hopefully Dave and Neha will add this since I don't know what we did.

  • No labels