This tutorial will show you how to set up a test cluster and run Kafka system tests on your local machine. I'll assume here that you have a clone of Kafka trunk.
This is broken down into a series of short-ish videos. A summary of the commands run is below each video clip, so you can skip any video clip if you want to.
We'll need both Vagrant and Virtualbox to create a test cluster
$ # Easy to check with these commands: $ vagrant --version # We want vagrant >= 1.6.4 $ vboxmanage --version |
To install Vagrant, go here
To install Virtualbox, go here
$ # This does things like installing necessary Vagrant plugins, creating a Vagrantfile.local, and creating a reusable Vagrant base box $ tests/bootstrap-test-env.sh |
Bring up the test cluster
This part is fairly simple, but can take a while, so bring a good book.
$ # If you skipped the previous step, make sure you have a Vagrantfile.local in your Kafka directory $ cp vagrant/system-test-Vagrantfile.local Vagrantfile.local $ $ # Update Vagrantfile.local, and set num_workers = 10 $ # If you packaged a box, set base_box = "kafkatest-worker" $ $ vagrant up # This will bring up your cluster $ $ # Other useful vagrant commands $ vagrant status # See how many vms you have, whether they're running, etc. $ vagrant ssh <vm_name> # ssh into a vm $ vagrant hostmanager # update /etc/hosts on each vm $ vagrant halt # shut down without destroying $ vagrant destroy -f # destroy the vms |
Here I'll install ducktape, introduce a few useful ducktape commands, show how to actually run the tests, and give an overview of test output.
$ # Install ducktape: $ pip install ducktape $ $ cd tests $ $ ducktape --version # Good way to check if ducktape is properly installed $ ducktape <path> --collect-only # See which tests ducktape discovers underneath <path> $ $ # Run "test_console_consumer.py" with the debug flag turned on $ ducktape kafkatest/sanity_checks/test_console_consumer.py --debug $ $ # Run all tests $ ducktape kafkatest/tests |