Page History

This tutorial will show you how to set up a test cluster and run Kafka system tests on your local machine. I'll assume here that you have a clone of Kafka trunk.

This is broken down into a series of short-ish videos. A summary of the commands run is below each video clip, so you can skip any video clip if you want to.

...

Ensure Vagrant and Virtualbox are installed

We'll need both Vagrant and Virtualbox to create a test cluster

...

To install Virtualbox, go here

Install necessary Vagrant plugins

...

Bootstrap Vagrant for system testing 

$$ # vagrantThis plugindoes install vagrant-cachier
$
$ # Note: if you already have vagrant-hostmanager v1.6.0 or higher, you'll first need to uninstall it
$ # by running vagrant plugin uninstall vagrant-hostmanager
$ vagrant plugin install vagrant-hostmanager --plugin-version 1.5.0

(Optional) Create a base box for the test vms

This step isn't strictly necessary, but if you do this once on your local machine, you'll significantly reduce the amount of time it takes to create each new virtual machine.

$ cd kafka
$ cp vagrant/system-test-Vagrantfile.local Vagrantfile.local
$ 
$ # Now openthings like installing necessary Vagrant plugins, creating a Vagrantfile.local, and set num_workers = 1
$ 
$ vagrant up   # This will build one vm from "scratch"
$ vagrant halt # Shut down the newly created vm
$ vagrant package worker1 # This creates creating a reusable "packaged" version of the new vm, stored in package.Vagrant base box
$ vagrant box add kafkatest-worker package.box  # This will take a little while...
$
$ # Now you have a reusable box! 
$
$ # Now open Vagrantfile.local, and set base_box = "kafkatest-worker"
$
$ # We're all done, so we can clean up after ourselves:
$ rm -f package.box
$ vagrant destroy -ftests/bootstrap-test-env.sh

...

Bring up the test cluster

...

$ # If you skipped the previous step, make sure you have a Vagrantfile.local in your Kafka directory
$ cp vagrant/system-test-Vagrantfile.local Vagrantfile.local
$ 
$ # Update Vagrantfile.local, and set num_workers = 10
$ # If you packaged a box,        set base_box = "kafkatest-worker"
$
$ vagrant up # This will bring up your cluster
$
$ # Other useful vagrant commands
$ vagrant status 		# See how many vms you have, whether they're running, etc.	
$ vagrant ssh <vm_name> # ssh into a vm
$ vagrant hostmanager   # update /etc/hosts on each vm
$ vagrant halt 			# shut down without destroying
$ vagrant destroy -f 	# destroy the vms
 

...

Run the tests

Here I'll install ducktape, introduce a few useful ducktape commands, show how to actually run the tests, and give an overview of test output.

...