...

Here is some information on actually running Kafka as a production system. This is meant as a page for people to record their operational and monitoring practices to help people gather knowledge about successfully running Kafka is production. Feel free to add a section for your company if you have anything to share.

LinkedIn

This is taken from production clusters at LinkedInto the recommendations if you have any suggestions not covered here. There is nothing magically about most of these configurations, you may be able to improve on them, but they may serve as a helpful starting place. If do have operational or tuning improvements to share, please send them to kafka-users@incubator.apache.org.

Hardware

We are using dual quad-core Intel Xeon machines with 24GB of memory. In general this should not matter too much, we only see pretty low CPU usage at peak even with GZIP compression enabled and a number of clients that don't batch requests. The memory is probably more than is needed for caching the active segments of the log.

...