You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

It would be worthwhile to automate our performance testing to act as a generic integration test suite.

The goal of this would be to do basic performance analysis and correctness testing in a distributed environment.

Phase I: Perf Tools

The goal of this phase is just to create tools to help run perf tests. We already have some of these so this will primarily just be about expanding and augmenting these.

  • kafka-producer-perf-test.sh - We will add a csv option to this to dump incremental statistics in csv format for consumption by automated tools.
  • kafka-consumer-perf-test.sh - Likewise we will add a csv option here.
  • jmx-dump.sh - This will just poll the kafka and zookeeper jmx stats every 30 seconds or so and output them as csv.
  • dstat - This is the existing perf tool to get read/write/fs stats
  • draw-performance-graphs.r - This will be an R script to draw relevant graphs given a bunch of csv files from the above tools. The output of this script will be a bunch of png files that can be combined with some html to act as a perf report.

Here are the graphs that would be good to see:

  • Latency histogram for producer
  • MB/sec and messages/sec produced
  • MB/sec and messages/sec consumed
  • Flush time
  • Consumer cache hit ratio (both the bytes and count, specifically 1 - #physical_reads / #requests and 1 - physical_bytes_read / bytes_read)
  • Write merge ratio (num_hits
  • CPU, network, io, etc

Phase II: Automation

This phase is about automating the deployment and running of the performance tests. At the end of this phase we want to have a script that pulls from svn every night, runs a set of performance scenarios, and produces reporting on these.

We need the following helper scripts for this:

  • kafka-deploy-kafka.sh - This script will take a set of hosts and deploy kafka to each of them.
  • kafka-start-cluster.sh - This will start the kafka broker on the list of hosts
  • kafka-stop-cluster - Stops cluster

The tests will be divided up into scenarios. Each scenario is a directory which contains the following:

  • broker config
  • producer config
  • consumer config
  • producer perf test command
  • consumer perf test command
  • env file that contain # brokers, # producers, and # consumers

The output of the scenario will be a directory which contains the following:

  • Producer perf csvs
  • Consumer perf csvs
  • dstat csvs
  • jmx csvs
  • env file

Scenarios to test:

Phase III: Correctness

The correctness testing can be very straight-forward, all we want to validate is that every message produced gets consumed. This could be as simple as logging out a simple message id in the consumer and comparing it to the produced value.

Simplest idea is just to have each producer produce a set of known messages (say sequential integers in some unique range). Then have consumers validate that all integers were consumed (no gaps) and record the number of duplicates (if any).

  • No labels