...
Draft. | Rupert Smith. | 22nd Feb 2007 | Document started. |
---|---|---|---|
Working Copy. | Rupert Smith. | 6th Mar 2007 | Document updated from feedback to draft on qpid-dev list. Last requirement # used: 47 |
Introduction:
The requirements in this specification use a common format, an example of which is given below:
...
The requirements are numbered from 1.
Purpose:
- Test sending from and receiving by each of the clients in Qpid over both of the broker implementations.
...
- Run just a few tests to begin with; more can be added later. Establish a working test framework as quickly as possible.
Constraints:
IOP-1. | Operating System. | The test client scripts must run on Unix and Windows. If a test client implementation is only available on one of these platforms it only needs to run on its supported platform. |
---|---|---|
IOP-2. | Scripting Language. | Each test client must be startable from a Unix shell script. Tests run on Windows will use Cygwin to run these scripts. There is no need to support Windows .bat scripts. |
Functional Requirements:
Introduction.
These requirements describe the behaviour of test clients for interop testing between the different client implementations in Qpid. Each client is expected to be a single program that is capable of sending test messages to other clients and receiving and responding to test messages received from other test clients. The clients are not to be run as seperate programs for the sending and receiving parts for the sake of convenience in being able to run the clients as part of an automated build. The clients will listen for control messages broadcast by a master coordinator client, to enlist them in tests, tell them which test to run, when to begin their tests, and when to shutdown.
A centralized approach has been chosen, using a single coordinator client, as test framework code which would otherwise have to be duplicated amongst all the clients will generally be put in the coordinator. This means that code will only have to be written and maintained in one place. This code will include code for enlisting clients for tests, deciding which test case to run, and formatting and logging out the results. The alternative would be to have a de-centralized approach, where each client broadcasts the test enlist messages, finds out what other clients are available to talk to, choses which tests to run and outputs the test results. One advantage of the centralized approach, is that the coordinator should know which clients are available, and therefore which clients cannot run particular tests, or fail completely to run particular tests, and should therefore be able to log out failures for clients that do not implement some tests in a more reliable way.
Common Requirements.
IOP-3. | Directory Structure. | All scripts to start and stop brokers and run test clients will be placed in a directory structure underneath a top-level directory called 'interop' that sits at the top level of the Qpid project. |
---|---|---|
IOP-4. | Test Output Format. | Output in junit xml format (because a lot of automated build software understands this format). There doesn't seem to be a schema or DTD for this format but it is simple enough. See Appendix B for an example. |
IOP-5. | Terminate On Timeout. | Each client will keep a timeout count. Every time it gets a message it will reset this count. If it does not hear from the broker at all for 60 seconds then it will assume that the broker has died or that the other test clients are failing to communicate with it, and will terminate. Test clients will only wait on this timeout when they are actually expecting messages, for example after enlisting to a test and expecting a role assignment message, or during a test when they are expecting to be sent a test message. If neccessary, this timeout can be extended to a longer time period than 60 secods, its purpose is to ensure eventual termination of all clients during a fully automated build. |
IOP-6. | Default Virtual Host. | All test clients will use the default virtual host (no name) for all tests, unless overriden by test parameters for a particular test case, or by command line options when starting the client. |
IOP-7. | Broadcast Control Topic. | All test clients will listen to control messages broadcast on the routing key 'iop.control' on the default virtual host on the default topic exchange. This control topic is used for communicating with the test coordinator client. |
IOP-8. | No Environment for Scripts. | In general, start up scripts should be intelligent enough to configure the environment variables that they need in order to run. It should be sufficient to have a path configured for the neccessary run time tools (such as Java) when calling scripts. Environment variables, such as QPID_HOME, should be set by startup scripts themselves, figured out from their installation locations. |
IOP-9. | Wait Until Background Process Started. | Scripts that start processes running in the background should not terminate until the process they are starting has succesfully started. This is neccessary for reliable testing, to ensure that subsequent scripts can be run, knowing that previous scripts have completed, with dependant proccesses in a known state. For example, it is important to start all test clients prior to starting the coordinator. |
Use Case 1. Starting a Broker.
Run the broker start script.
The script starts a broker running and tries to connect to it (or otherwise ping it) until it is verified to be running.
Once the broker is verified to be running the script terminates with no error code.
...
IOP-10. | Broker Start Script. | The Java and C++ brokers will define scripts that can start the broker running on the local machine, and these scripts will be located at interop/java/broker/start and interop/cpp/broker/start. The Java and C++ build processes will generate these scripts (or copy pre-defined ones to the output location) as part of their build processes. |
---|---|---|
IOP-11. | Broker Start Failure. | If a broker fails to start within 60 seconds its start script will timeout. Script will terminate with error code 1. |
IOP-12. | Broker Start Succesfull. | When the broker starts succesfully the script will terminate with error code 0. |
Use Case 2. Stopping a Broker.
Run the broker stop script.
The script terminates the broker that was started with the start script if it is still running.
...
IOP-13. | Broker Stop Script. | The Java and C++ brokers will define scripts that can stop the broker running on the local machine, and these scripts will be located at interop/java/broker/stop and interop/cpp/broker/stop. The Java and C++ build processes will generate these scripts (or copy pre-defined ones to the output location) as part of their build processes. |
---|---|---|
IOP-14. | Broker Stop Timeout. | If a broker fails to terminate within 60 seconds its stop script will timeout. Script will terminate with error code 1. |
IOP-15. | Broker Stop Succesfull. | When the broker stops succesfully the script will terminate with error code 0. |
Use Case 3. Starting a Test Client.
Run the client start script. The caller will pass in the address of the broker to connect to.
The script starts a client running.
The client starts running but waits for further instruction before running its tests.
The start script will terminate but leave the client running as a forked process.
...
IOP-16. | Client Start Scripts. | For each client implementation, <client>, there will be a start script located at interop/<client>/client/start. The build processes for each client will generate these scripts and output them to this location as part of their build process. |
---|---|---|
IOP-17. | Client Start Timeout. | If the client fails to start and connect to the specified broker within 60 seconds the script will terminate with error code 1. |
IOP-18. | Client Start Succesfull. | When the client starts successfully its script will terminate with error code 0. |
IOP-19. | Client Start Broker Url. | The -b <broker_url> option will be used to call the start script to connect to the broker specified in the url. |
IOP-20. | Client Virtual Host. | The default virtual host to connect to, may be overridden with the -h <virtual_host> command line option, which will be accepted by all test clients. |
IOP-21. | Client Start General Parameters. | General parameters may be passed to the client start scripts using the synax name=value. These name/value pairs may be used by specific test cases to override default test parameters. |
Use Case 4. Starting the Coordinator.
- The requirements defined for Use Case 3, also apply to this use case.
...
IOP-22. | Coordinator Test Script. | There will be a coordinator test script that kicks off the testing process once all clients have been started. It is to be located at interop/testall. It will start a coordinator test client that issues test invites, assigns roles, collects results and terminates test clients when all tests have been run. |
---|
Use Case 5. Overall Test Procedure.
Start a broker running using its start script as described by Use Case 1.
Call the start all clients script on each of the machines where there are clients that are to be tested. The caller will pass in address of the broker to connect to and the number of messages to send per test.
The start all script will scan for all start scripts located under interop/<client>/client/start and call each of them forwarding its command line arguments on the call. This performs Use Case 3 for each client.
Call the coordinator test client script. This is described as Use Case 4.
The coordinator test script will broadcast an invite message, with no test name on the control topic. The lack of a test name indicates that this is a compulsory invite, to which all clients must enlist.
Each of the clients will receive the invite message, with no test name. Each client will repond with an enlist message. This message will contain the routing key on the default direct exchange to which the client has bound its private control queue.
The coordinator retain the list of available clients.
...
IOP-23. | Start All Script. | There will be a start all clients script, located at interop/startall. The startall script finds all client starts scripts under interop/<client>/client/start and calls them. | ||||
---|---|---|---|---|---|---|
IOP-24. | Start All Script Options Forwarding. | The start all script will take the same command line options as the client start scripts and will pass these command line options on to them. | ||||
IOP-25. | Invite Message. |
| ||||
IOP-26. | Initial Invite. | At the start of the test procedure the coordinator will broadcast a compulsory invite, to which all available clients must enlist, in order to declare their availability and to enable the coordinator to detect when there are clients that did not participate in some tests. The compulsory invite will be differentiated form an ordinatory invite because it will have no "TEST_NAME" header field. | ||||
IOP-27. | Enlist Message. |
| ||||
IOP-28. | Assign Role Message. |
| ||||
IOP-29. | Accept Role Message. |
| ||||
IOP-30. | Start Message. |
| ||||
IOP-31. | Report Message. |
| ||||
IOP-32. | End Role Message. |
| ||||
IOP-33. | Test Done Message. |
| ||||
IOP-34. | Terminate Message. |
| ||||
IOP-35. | Client Name. | Each test client will provide a unique name for itself that reflects its implementation language and distinguishes it from the other clients. Clients should append a timestamp or UUID onto this name to cater for the case where the same client is used multiple times in an interop test. For example, the same client might be run on two different operating systems, in order to check that it works correctly on both. | ||||
IOP-36. | Private Client Control Topic. | Each test client will listen for test control messages directed specifically to it on the default topic exchange. The routing key for these messages will consist of "iop.control." followed by the client name (see IOP-35). A topic exchange is used, rather than a direct exchange, to cater for the situation where multiple instances of a client are run in parallel and tests are to be scaled accross many clients. It also allows a listener to be attached to the default topic exchange to listen to all control messages using a wildcard selector. | ||||
IOP-37. | Seperate Connection for Control Topic. | Test clients should create open a seperate connection to communicate with the control topics on the default topic exchange, to that which they use to perform tests. This is so that a channel level error that results in the closing of a connection during a test, may still allow a client to succesfully send a failure report to the coordinator. |
Common Requirements for Test Cases.
Test cases that use these requirements mention them in the description of the test case.
IOP-38. | Client Message Receive. | Whenever a test client recieves a message from another test client it will increment the total count of messages received from that client since the last reset. The message will contain the name of the sending client in the header field "CLIENT_NAME", and the count will be held against that name. | ||||
---|---|---|---|---|---|---|
IOP-39. | Client Reset Message. |
| ||||
IOP-40. | Client Status Request Message. |
| ||||
IOP-41. | Client Status Message. |
| ||||
IOP-42. | Correlation Id. | When sending test messages, clients will identify all messages using a unique correlation id for the test case instance. This will differentiate test messages in a situation where the same client is scaled up to run a test case many times in parallel. | ||||
IOP-43. | Test Connections. | Test clients will create connections to send test messages on when they are assigned roles. In many cases this will consist of creating a single connection, and a producer or consumer for the test routing key or queue. In some tests, which simulate the activity of many message receivers, multiple connections may be opened. |
Test Case 1. Dummy Run.
The sending client will not send any test messages at all. It will send a report message on the control topic, declaring that the test has passed.
The purpose of this test case is to check that clients can interoperate succesfully with the test coordinator and participate in the sequencing of the tests.
Test Case 2. Basic P2P Test.
- This test case uses requirements IOP-38 to 43 inclusive.
...
IOP-44. | Basic P2P Setup. | Prior to assigning roles, the coordinator will bind a queue to the default direct exchange with a routing key, the same as the queue name. It will create a fresh queue for every test case instance. | ||||
---|---|---|---|---|---|---|
IOP-45. | Basic P2P Assign Role Parameters. |
|
Test Case 3. Basic Pub/Sub Test.
- This test case uses requirements IOP-38 to 43 inclusive.
...
IOP-46. | Basic Pub/Sub Setup. | Prior to assigning roles, the coordinator will choose a routing key for the test. If will create a fresh key for every test case instance. | ||||
---|---|---|---|---|---|---|
IOP-47. | Basic Pub/Sub Invite Parameters. |
|
Waiting Room:
Contains ideas for possible future directions relating to this spec.
...
Allow scaling of test clients. Each test client should only be run once (in each environment) and they create unique names for themselves. Tests are only run between pairs of single clients, with a single sender and number of receivers defined by the test case (often 1). Clients listen for control messages on topics, and use correlation id's in all tests messages to differentiate themselves were multiple senders to be active. This has been done deliberately to allow for future expansion of the test framework to allow scaling up of the tests by starting more clients.
Appendix A, General Notes:
Brokers that need to be interop tested: C++ and Java
Clients that need to be interop tested: C++ , Java, Java 1.4 retrotranslation, C++, .Net 2.0, .Net 1.1, (Mono?), Python, Ruby.
Appendix B, Example of XML Format for Test Ouput:
I don't think there is a DTD or schema for this but the XML output from JUnit looks like the example below. This is a convenient choice for the output format from these test results even if the code does not actually use JUnit (or cppunit or nunit) iternally, because automated build servers generally understand and are able to produce test reports from it.
...
No Format |
---|
<?xml version="1.0" encoding="UTF-8" ?> <testsuite errors="0" skipped="0" tests="18" time="0.02" failures="0" name="org.apache.qpid.framing.BasicContentHeaderPropertiesTest"> <properties> <property value="Java(TM) 2 Runtime Environment, Standard Edition" name="java.runtime.name"/> ... (there were lots of properties). </properties> <testcase time="0.02" name="testRejectedExecution"/> ... (there were lots of test cases). </testsuite> |
Appendix C, Command line options.
IOP-21 states that general parameters can be passed on the command line using name=value syntax. The coordinator understands the following parameters, and will use them to override the default values for the tests. Individual test cases refer to the command line parameter that they take their test parameters from.
Parameter | Default |
---|---|
P2P_NUM_MESSAGES | 50 |
PUBSUB_NUM_RECEIVERS | 5 |
PUBSUB_NUM_MESSAGES | 10 |
Appendix D, Deleted Requirements:
Put deleted requirements here, in case they can be re-used.
...