Page History

...

• Latency histogram for producer
• MB/sec and messages/sec produced
• MB/sec and messages/sec consumed
• Flush time
• Errors (should not be any)
• Consumer cache hit ratio (both the bytes and count, specifically 1 - #physical_reads / #requests and 1 - physical_bytes_read / bytes_read)
• Write merge ratio (num_hits_physical_writes/num_produce_requests and avg_request_size/avg_physical_write_size)
• CPU, network, io, etc

Phase II: Automation

...

• Producer perf csvs
• Consumer perf csvs
• dstat csvs
• jmx csvs
• env file

Scenarios to test:

• Producer throughput with no consumers. We should cover the following cases:
  • Vary the number of topics
  • Vary the async batch size
  • Vary the flush size
  • Vary the message size
• Consumer throughput with no producer
  • Vary the message size
  • Vary the number of topics
• Single producer/consumer pair
  • Cold consumption (i.e. not in cache)
  • Active consumption (i.e. consumer caught up to producer)
  • Vary the number of topics
• Multiple consumers for one topic

We should add a script to take two scenarios and produce a summary/diff of them, i.e. what go worse and what got better. We can use this to track things over time. We can also rsync these up to a public location as a service to open source developers.

Phase III: Correctness

The correctness testing can be very straight-forward, all we want to validate is that every message produced gets consumed. This could be as simple as logging out a simple message id in the consumer and comparing it to the produced value.

Simplest idea is just to have each producer produce a set of known messages (say sequential integers in some unique range). Then have consumers validate that all integers were consumed (no gaps) and record the number of duplicates (if any).

Ideally we would repeat this scenario and script in broker failures (kills), server pauses (simulated), etc.