Page History

...

fetch.wait.max.ms controls how long a fetch request will wait on the broker in the normal case. The issue is that if there is a hard crash on the broker (host is down), the client may not realize this immediately since TCP will try very hard to maintain the socket connection. By setting socket.timeout.ms, we allow the client to break out sooner in this case. Typically, socket.timeout.ms should be set to be at least fetch.wait.max.ms or a bit larger. It's possible to specify an indefinite long poll by setting fetch.wait.max.ms to a very large value. It's not recommended right now due to https://issues.apache.org/jira/browse/KAFKA-1016. The consumer-config documentation states that "The actual timeout set
will be max.fetch.wait + socket.timeout.ms." - however, that change seems to have been lost in the code a while ago. https://issues.apache.org/jira/browse/KAFKA-1147 is filed to fix it.

I am not setting the partition key and have more than one partition but only one consumer is getting the messages from my producer?

When many producers where used in 0.8 during the beta period an issue was found https://issues.apache.org/jira/browse/KAFKA-1017 which changed how partitions got message when the partition key was null. If you are development/testing and and want to mimic in your development/testing multiple partitions/consumers getting your messages then what you could do is for the partition key have it be UUID.randomUUID().toString. If you are in production then with a sufficient amount of producers the net net will have all consumers getting messages... if not then you may have to make the partition key help you do this then too and not rely on a null partition key. For more detail see this thread http://mail-archives.apache.org/mod_mbox/kafka-dev/201310.mbox/%3CCAFbh0Q0aVh%2Bvqxfy7H-%2BMnRFBt6BnyoZk1LWBoMspwSmTqUKMg%40mail.gmail.com%3E

Brokers

How many topics can I have?

...