...

Here is some information on actually running Kafka as a production system. This is meant as a page for people to record their operational and monitoring practices to help people gather knowledge about successfully running Kafka is production. Feel free to add a section for your company if you have anything to share.

LinkedIn

This is taken from production clusters at LinkedIn. There is nothing magically about most of these configurations, you may be able to improve on them, but they may serve as a helpful starting place. If do have operational or tuning improvements to share, please send them to kafka-users@incubator.apache.org.

Hardware

We are using dual quad-core Intel Xeon machines with 24GB of memory. In general this should not matter too much, we only see pretty low CPU usage at peak even with GZIP compression enabled and a number of clients that don't batch requests. The memory is probably more than is needed for caching the active segments of the log.

The disk throughput is important. We have 8x7200 rpm SATA drives in a RAID 10 array. In general this is the performance bottleneck, and more disks is more better. Depending on how you configure flush behavior you may or may not benefit from more expensive disks (if you flush often then higher RPM SAS drives may be better).

OS Settings

We use linux. Ext4 is the filesystem and we run using software RAID 10. We haven't benchmarked filesystems so other filesystems may be superior.

We have added two tuning changes: (1) we upped the number of file descriptors since we have lots of topics and lots of connections, and (2) we upped the max socket buffer size to enable high-performance data transfer between data centers (described here).

Java

We are running Sun Java:

$ java -version
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)

...

java -server -Xms3072m -Xmx3072m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
     -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -Xloggc:logs/gc.log -Djava.awt.headless=true
     -Dcom.sun.management.jmxremote -classpath <long list of jars>

Kafka

We are running Kafka 0.7 right now but may move to trunk as we fix bugs.

...

Client configuration varies a fair amount.

Monitoring

Our monitoring is done though a centralized monitoring system custom to LinkedIn, but it keys off the JMX stats exposed from kafka. To see what is available the easiest thing is just to start a kafka broker and/or client and fire up JConsole and take a look.

Server Stats

• JMX Stats
  • Number of fetch and produce requests per second
  • MB produced and fetched per second
  • Avg and max fetch and produce request time
  • Number of log flushes on the server
  • Average and max log flush time
• System Stats
  • CPU
  • Network throughput
  • Disk throughput (MB/sec, reads and writes)
  • Disk space remaining
  • IO service time
  • IO Operations/sec
  • # files descriptors used/available

Application stats

These are monitored per-application

• Number of produce and fetch requests sent
• Request latency
• Queued messages not yet sent (we use the async producer)

Audit

The final audit we do is on the correctness of the data delivery. We audit that every message that is sent is consumed by all consumers and measure the lag for this to occur. For important topics we alert if a certain completeness is not achieved in a certain time period. The details of this are discussed in KAFKA-260.

...