Status

Current state: Under Discussion

...

PULL REQUEST: https://github.com/apache/kafka/pull/4818

Motivation

ConsumerPerformance fails to consume all messages on topics with large number of partitions due to a relatively short default polling loop timeout (1000 ms) that is not reachable and modifiable by the end user. 

...

• Increasing polling loop timeout is probably a good thing to handle this case
• Increasing it too much by default (say, 5 minutes) may lead to a slower feedback from the tool in some cases 
• Making polling loop timeout optionally changeable is a good thing since it would give us more control for the corner cases
• Printing a warning message in case the timeout is exceeded should also be a good thing to know what's going on  

Public Interfaces

Providing the command line tool kafka-consumer-perf-test.sh with an optional --polling-loop-timeout parameter: 

Proposed Changes

• Increasing polling loop timeout in ConsumerPerformance implementation up to 60 000 ms 
• Making polling loop timeout reachable and modifiable by the end user from the command line kafka-consumer-perf-test tool
• If the polling loop timeout is exceeded, print a warning message

Proposed changes in code available in this PR: https://github.com/apache/kafka/pull/4818

Compatibility, Deprecation, and Migration Plan

• Positive impact on existing users since the utility generates more accurate results 
• The proposed parameter is optional, will not break anything

Rejected Alternatives

...