THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
The rough plan to move forward is as follows:
Pick a new framework, or developone (see furtheroneWe chose to use ducktape (https://github.com/confluentinc/ducktape - see notes below)
Identify “MVP” tests - i.e. a set of high priority tests to fill in major gaps in our existing system test coverage (see further notes below)
Identify “long tail” of less urgent tests.
Stabilize and prune existing system tests so they can be run regularly until they are ported(?)
Framework Choice
We are currently evaluating evaluated Zopkio (https://github.com/linkedin/Zopkio) which is used internally at LinkedIn for Samza and other projects, and Confluent’s ducttape ducktape (https://github.com/confluentinc/ducktape) which is used in system tests for the Confluent Platform (https://github.com/confluentinc/ducttape)muckrake)
and concluded that ducktape better matched our use case (see comparison table below). Continued development on ducktape is needed, but it is far enough along to be a useful tool for a wide variety of system testsBoth options require further development effort but represent improvement over the existing framework.
ducttape | zopkio | |
runnable on cloud or locally (provisioning capability) | yes | no |
cluster management | yes | no |
run tests in parallelno | not implemented, but written with this use case in mind | use of globals makes this harder |
easy to read/understand | somewhat | somewhat |
good reporting | noyes | yes |
regressionno | not yet | yes (at least, naarad claims to support this) |
automatablecrude test driver available | yes - has flexible test discovery mechanism | test suites run in a self-contained way, but a script to drive multiple test suites would still be needed |
Test Coverage
Among other things, we want to include those standalone programs into our system test so that we have reasonable coverage on each important feature.
...