Page History

Here is some information on actually running Kafka as a production system. This is taken from production clusters at LinkedIn. There is nothing magically about most of these configurations, you may be able to improve on them, but they may serve as a helpful starting place. If do have operational or tuning improvements to share, please send them to kafka-users@incubator.apache.org.

...

We have added two tuning changes: (1) we upped the number of file descriptors since we have lots of topics and lots of connections, and (2) we upped the max socket buffer size to enable high-performance data transfer between data centers (described here).

Java

We are running Sun Java:

...

kafka.log.default.flush.interval.ms=10000
kafka.log.file.size=1073741824
kafka.log.default.flush.scheduler.interval.ms=2000
kafka.log.flush.interval=3000kafka.socket.send.buffer=10485762097152
kafka.socket.receive.buffer=1048576kafka2097152
kafka.monitoring.period.secs=30
kafka.num.threads=8
kafka.log.cleanup.interval.mins=30
kafka.log.retention.hours=168
kafka.zookeeper.sessiontimeoutms=6000
kafka.zookeeper.connection.timeout=2000
kafka.num.partitions=1

...