THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Code Block |
---|
1. Check out kafka codebase: a. ~ $ git clone https://git-wip-us.apache.org/repos/asf/kafka.git b. ~ $ cd <kafka> 2. Under <kafka>, build kafka a. <kafka> $ ./gradlew jar 3. Set JAVA_HOME environment variable (optional, but recommended): a. export JAVA_HOME=/usr/java/jdk1.7.0_67 4. Under <kafka>/system_test, execute the following command to start System Test : $ python -u -B system_test_runner.py 2>&1 | tee system_test_output.log |
...
Code Block |
---|
<kafka> |- /bin |- /config |- /contrib |- /core |- /lib |. |. |. |- /system_test | |- system_test_runner.py | | # system_test_runner.py is the main script to start system test as follows : | # | # 1. for each test suite directory (XXXX_testsuite) under systemTestEnv.SYSTEM_TEST_BASE_DIR (<kafka>/system_test) : | # 2. get a list of module scripts (*.py) in test suite directory | # 3. for each file in the list of module scripts : | # 4. get class name from the Python module script | # 5. retrieve corresponding suite & module names from class name | # 6. dynamically load the module and start the test class | # 7. save each test case result in systemTestEnv.systemTestResultsList | # 8. for each result in systemTestEnv.systemTestResultsList : | # 9. print result | |- /utils # This is a directory that contains all helper classes / util functions for system test | | |- kafka_system_test_utils.py # utilities specific to Kafka system testing (e.g. Kafka test cases data loss validation) | |- replication_utils.py # utilities specific to replication testing (e.g. Leader election log message pattern) | |- setup_utils.py # generic helper for system test setup (e.g. System Test environment setup) | |- system_test_utils.py # utilities for generic testing purposes (e.g. reading JSON data file) | |- testcase_env.py # testcase environment setup (e.g. data structure initialization such as brokers-pid mapping) | |- metrics.py # utilities for metrics collection | |- pyh.py # from http://code.google.com/p/pyh (open source) | |- cluster_config.json # this file contains the following properties: | # 1. what entities (Producer, Consumer) should be running | # 2. which cluster (source or target) | # 3. where they should be running (physical nodes) | | # cluster_config.json is used to specify the logical machines configuration : | # | # entity_id : In each testcase, there may be zookeeper(s), broker(s), producer(s) and | # consumer(s) involved. "entity_id" is used to uniquely identify each | # component inside the system test. | # hostname : It is used to specify the name of the machine in the distributed environment. | # "localhost" is used by default. | # role : The supported values are "zookeeper", "broker", "mirror_maker", "migration_tool", | # "producer", "consumer". | # cluster_name : The supported values are "source", "target" | # kafka_home : Specify the Kafka installation directory of each machine in a distributed environment. | # "default" is used by default and ../ is assumed to be "kafka_home". | # java_home : Specify the JAVA_HOME of each machine in a distributed environment. | # 1. "default" is used by default. If JAVA_HOME is specified in the environment, | # this value will be used. Otherwise, System Test executes a shell command "which java" | # to find java bin dir and set JAVA_HOME accordingly. If no java binary can be found, | # it throws Exception and exit. | # 2. If a path is specified other than "default", System Test will verify java binary. | # Otherwise, it throws Exception and exit. | # jmx_port : Specify a JMX_PORT for each component. It must be unique inside the cluster_config.json. | # | # { | # "cluster_config": [ | # { | # "entity_id" : "0", <----------------------------------------------------------------------------------| | # "hostname" : "localhost", "entity_id" in cluster_config must match the corresponding "entity id" in | | # "role" : "zookeeper", testcase_NNNN.properties as shown below | | # "cluster_name" : "source", | | # "kafka_home" : "default", | | # "java_home" : "default", | | # "jmx_port" : "9990" | | # }, | | # . | | # . . | # . . | # } . | |- /XXXX_testsuite | | # XXXX_testsuite is a directory which contains Python scripts and test case directories for a specific group of functional testings. | # For example, mirror_maker_testsuite contains mirror_maker_test.py in which it will start mirror maker instances which is not | # required for the test cases in migration_tool_testsuite. | # | # The following are the existing test suites: | # migration_tool_testsuite | # mirror_maker_testsuite | # replication_testsuite | |- <testsuite>.py | | # <testsuite>.py : This Python script may be implemented with the test logic for a group of test scenarios. | # For example, in the "mirror_maker_testsuite", mirror_maker_test.py is the testsuite script | # which is implemented with the following : | # 1. start zookeeper(s) in source cluster | # 2. start broker(s) in source cluster | # 3. start zookeeper(s) in target cluster | # 4. start broker(s) in target cluster | # 5. start mirror maker(s) | # 6. start producer | # 7. start consumer | # 8. stop all entities | # 9. validate no data loss | # | # The above test logic is implemented in a generalized fashion and read from | # testcase_NNNN_properties.json for the following test case arguments : | # 1. replication factor (broker) | # 2. no. of partitions (broker) | # 3. log segment bytes (broker) | # 4. topics (producer / consumer) | # . | # . | # . | # | # Therefore, each testsuite can be thought of a functional / feature test group | # by varying a combination of settings for easier maintenance. | |- /config | | # config : This config directory contains the TEMPLATE properties files for this testsuite. | # For "replication_testsuite", only server.properties and zookeeper.properties are | # required. However, "mirror_maker_testsuite" has mirror_consumer.properties and | # mirror_producer.properties as well. | # | # System Test reads the properties from each files in this directory as a TEMPLATE | # and override the values with those from testcase_NNNN_properties.json accordingly. | |- server.properties |- zookeeper.properties |- (migration_consumer.properties) # only in migration_tool_testsuite |- (migration_producer.properties) # only in migration_tool_testsuite |- (mirror_maker_consumer.properties) # only in mirror_maker_testsuite |- (mirror_maker_producer.properties) # only in mirror_maker_testsuite | |- /testcase_NNNN | | # testcase_NNNN is a directory which contains a json file to specify the arguments | # required for the <testsuite>.py script to execute. | # | # The main arguments are : | # 1. testcase_args : These are the arguments specific for that test case such as | # "replica_factor", "num_partition", "bounce_broker", ... etc. | # 2. entities : These are the arguments required to start running a certain | # entity such as a broker : port, replication factor, log file | # directory, ... etc. | |- (config) # generated when this testcase is executed in system test runtime | |- (logs) # generated when this testcase is executed in system test runtime | |- testcase_NNNN_properties.json # this file contains new values to override the default settings of | # various entities (e.g. ZK, Broker, Producer, Mirror Maker, ...) | # such as "log.segment.bytes", "num.partitions", "broker.id", ... | # { | # "description": {"01":"Replication Basic : Base Test", | # "02":"Produce and consume messages to a single topic - single partition.", | # "03":"This test sends messages to 3 replicas", | # . | # . | # . | # }, | # "testcase_args": { | # "broker_type" : "leader", | # "bounce_broker" : "false", | # "replica_factor" : "3", | # "num_partition" : "1", | # "num_iteration" : "1", | # . . | # . . | # . . | # }, | | # "entities": [ | | # { | | # "entity_id" : "0", <---------------------------------------------------------------| | # "clientPort" : "2188", 1. All attributes defined in this entity must be the | # "dataDir" : "/tmp/zookeeper_0", corresponding attributes for the "role" specified | # "log_filename" : "zookeeper_2188.log", by "entity_id" defined above in cluster_config | # "config_filename" : "zookeeper_2188.properties" 2. In this case, the attributes are all zookeeper | # }, related as the cluster_config has already defined | # . entity_id 0 as role "zookeeper" above | # . | # . | # } |
...