Interop Testing Spec.

1. Qpid Interop Testing Spec. Working Copy.

Introduction:

The requirements in this specification use a common format, an example of which is given below:

The requirements are numbered from 1.

Purpose:

• Test sending from and receiving by each of the clients in Qpid over both of the broker implementations.

• Make tests robust enough to run as part of an automated build. The scripts should pass or fail, not hang, wait forever, run out of memory or otherwise cause an automated build process to fail to complete.

• Be capable of running the full test suite on several machines in a hands free way. In particular C++ tests need to run on unix and .Net on windows, necessitating a multi-box solution for full interop testing.

• Run just a few tests to begin with; more can be added later. Establish a working test framework as quickly as possible.

Constraints:

Functional Requirements:

Introduction.

These requirements describe the behaviour of test clients for interop testing between the different client implementations in Qpid. Each client is expected to be a single program that is capable of sending test messages to other clients and receiving and responding to test messages received from other test clients. The clients are not to be run as seperate programs for the sending and receiving parts for the sake of convenience in being able to run the clients as part of an automated build. The clients will listen for control messages broadcast by a master coordinator client, to enlist them in tests, tell them which test to run, when to begin their tests, and when to shutdown.

A centralized approach has been chosen, using a single coordinator client, as test framework code which would otherwise have to be duplicated amongst all the clients will generally be put in the coordinator. This means that code will only have to be written and maintained in one place. This code will include code for enlisting clients for tests, deciding which test case to run, and formatting and logging out the results. The alternative would be to have a de-centralized approach, where each client broadcasts the test enlist messages, finds out what other clients are available to talk to, choses which tests to run and outputs the test results. One advantage of the centralized approach, is that the coordinator should know which clients are available, and therefore which clients cannot run particular tests, or fail completely to run particular tests, and should therefore be able to log out failures for clients that do not implement some tests in a more reliable way.

Common Requirements.

Use Case 1. Starting a Broker.

Run the broker start script.
The script starts a broker running and tries to connect to it (or otherwise ping it) until it is verified to be running.
Once the broker is verified to be running the script terminates with no error code.

Failure path: The broker fails to start or does not appear to be running after a timeout has passed. The script fails with an error code.

Use Case 2. Stopping a Broker.

Run the broker stop script.
The script terminates the broker that was started with the start script if it is still running.

Failure path: The broker won't terminate. The script fails with an error code.

Use Case 3. Starting a Test Client.

Run the client start script. The caller will pass in the address of the broker to connect to.
The script starts a client running.
The client starts running but waits for further instruction before running its tests.
The start script will terminate but leave the client running as a forked process.

Failure path: The client will not start, or fails to connect to the specified broker. The script will terminate with error code 1.

Use Case 4. Starting the Coordinator.

• The requirements defined for Use Case 3, also apply to this use case.

Run the testall start script. The caller will pass in the address of the broker to connect to.
The script starts the coordinator client running.
The coordinator will manage the test procedure.
The scipt will terminate when the coordinator has completed.

Failure path: The coordinator will not start, or fails to connect to the broker. The script will terminate with error code 1.

Use Case 5. Overall Test Procedure.

Start a broker running using its start script as described by Use Case 1.
Call the start all clients script on each of the machines where there are clients that are to be tested. The caller will pass in address of the broker to connect to and the number of messages to send per test.
The start all script will scan for all start scripts located under interop/<client>/client/start and call each of them forwarding its command line arguments on the call. This performs Use Case 3 for each client.
Call the coordinator test client script. This is described as Use Case 4.
The coordinator test script will broadcast an invite message, with no test name on the control topic. The lack of a test name indicates that this is a compulsory invite, to which all clients must enlist.
Each of the clients will receive the invite message, with no test name. Each client will repond with an enlist message. This message will contain the routing key on the default direct exchange to which the client has bound its private control queue.
The coordinator retain the list of available clients.

The coordinator will broadcast an invite to a named test. This invite may also contain any parameters needed to configure the test, that are relevant to a clients choice to accept the invite or not.
All clients that are able to participate in this test will reply to the invite with enlist messages. The enlist messages will contain the routing key to which the client has bound its private control topic on the default topic exchange. Clients may opt to participate in the test depending on the test parameters, if desired.
The coordinator will send messages to assign roles to the sender and receivers private control topics. These messages will contain the test parameters and roles. The test parameters may also include additional parameters not in the original invite, for test parameters that are to be set on a per test instance basis. In addition, the sender will be sent the routing key for the recivers private control topic on the default topic exchange and likewise the reciver will be sent the senders private control topic key.
The clients will respond with accept role messages, giving their client id, on the control topic.
The coordinator will wait until it has received acceptances from both roles.
The coordinator will issue a start message to the client with the sender role.
The sender client will perform the test case and decide if it has passed or failed. Once the test has completed the sender will send a report message to the coordinator, including the client id.
The coordinator will wait until it receives a report message from the sender, or until its timeout expires.
The coordinator will issue role completed messages to the sender and reciever, telling them to end their assigned roles.
The clients will perform any necessary clean up in order to prepare for a new role and reply to the coordinator with a test done message.
The coordinator will wait until it receives test done messages from all clients, before moving on to the next test case.
The coordinator will check its list of available clients and log out failures for any combinations of clients that were not tested because they did not enlist for the test.

Once all test cases are complete, the coordinator will broadcast a shutdown message.
All clients will terminate on receipt of the shutdown message.
The coordinator will terminate.
Terminate the broker using its stop script.

          "CONTROL_TYPE",               "INVITE"
          "TEST_NAME",                  "<test_case"
          ... optional test parameters.

          "CONTROL_TYPE",               "ENLIST"
          "CLIENT_NAME",                "<client_name>"             (see IOP-35 for rules about the client name).
          "CLIENT_PRIVATE_CONTROL_KEY", "iop.control.<client_name>" (see IOP-36)

          "CONTROL_TYPE",               "ASSIGN_ROLE"
          "OTHER_END_CONTROL_KEY",      "iop.control.<client_name>" (see IOP-36)
          ... full test parameters.

          "CONTROL_TYPE",               "ACCEPT_ROLE"
          "CLIENT_NAME",                "<client_name>"             (see IOP-35 for rules about the client name).

          "CONTROL_TYPE",               "START"

          "CONTROL_TYPE",               "REPORT"
          "RESULT",                     "PASS|FAIL"
          "CLIENT_NAME",                "<client_name>"             (see IOP-35 for rules about the client name).
          Message body,                 Reason for failures.

          "CONTROL_TYPE",               "END_ROLE"

          "CONTROL_TYPE",               "TEST_DONE"

          "CONTROL_TYPE",               "TERMINATE"

Common Requirements for Test Cases.

Test cases that use these requirements mention them in the description of the test case.

          "CONTROL_TYPE",               "RESET"
          "CLIENT_NAME",                "<client_name>"

          "CONTROL_TYPE",               "STATUS_REQUEST"
          "CLIENT_NAME",                "<client_name>"

          "CONTROL_TYPE",               "STATUS"
          "CLIENT_NAME",                "<client_name>"
          "MESSAGE_COUNT",              <count>         (signed 32 bit integer)

Test Case 1. Dummy Run.

The sending client will not send any test messages at all. It will send a report message on the control topic, declaring that the test has passed.

The purpose of this test case is to check that clients can interoperate succesfully with the test coordinator and participate in the sequencing of the tests.

Test Case 2. Basic P2P Test.

• This test case uses requirements IOP-38 to 43 inclusive.

The sending client creates a fresh correlation id, and the entire test case conversation uses this id.
The sending client sends a reset message to the receiving clients private control topic.
The receiving client will reset its message count to zero, (for that client and id).
The sending client will send the required number of test messages to the test routing key.
The sending client will send a status request message to the receiving clients control topic.
The receiving client will send a status message to the control topic of the sending client. This status message will contain the number of messages received since the last reset.
The sending client will compare the messages received to the messages sent and pass or fail the test accordingly.

          "P2P_QUEUE_AND_KEY_NAME",     "<name>"
          "P2P_NUM_MESSAGES",           <count>  (signed 32 bit int), P2P_NUM_MESSAGES property.

Test Case 3. Basic Pub/Sub Test.

• This test case uses requirements IOP-38 to 43 inclusive.

The sending client creates a fresh correlation id, and the entire test case conversation uses this id.
The sending client sends a reset message to the receiving clients control topic.
The receiving client will reset its message count to zero.
The sending client will send the required number of test messages to the receiving clients test topic routing key.
The sending client will send a status request message to the receiving clients control topic.
The receiving client will send a status message to the control topic of the sending client. This status message will contain the number of messages received since the last reset. This number will be the number of messages sent multiplied by the number of receivers being simulated by the receiving client.
The sending client will compare the messages received to the messages sent and pass or fail the test accordingly.

          "PUBSUB_KEY",                  "<key>"
          "PUBSUB_NUM_RECEIVERS",        <count> (signed 32 bit int), PUBSUB_NUM_RECEIVERS property.
          "PUBSUB_NUM_MESSAGES",         <count> (signed 32 bit int), PUBSUB_NUM_MESSAGES property.

Waiting Room:

Contains ideas for possible future directions relating to this spec.

Command processor. Test cases to be written using a command language (perhaps in XML) on top of a common client API. Interpreter for this to be implemented using each client library. Test cases need only be written once and can be run by the interpreters. Command language rich enough to exercise the whole AMQP protocol. May not handle client specific edge cases. Good for ensuring test consistency, but may take a fair amount of time to do.

How I anticipate this being run as part of a fully automated build. Will try to get a free licence for Anthill Pro 3 as they offer free licences for open source projects. Viewtier Parabuild is another possibility. Anthill Pro runs a central build server that does all its work through build agents that can run on many boxes. It also lets you define build workflows. I imagine running a Unix agent to build the c++, java and python stuff, and a Windows agent for the .net stuff. Will define a workflow that starts a broker on the unix box, then starts all clients built on the unix and windows boxes in parallel, then runs the entire test procedure across all clients, then terminates the broker on the unix box. The agents send back the test results to the central server.

Full testing of field tables. Make sure that every possible data type is tested and confirmed to encode and decode correctly between all client implementations.

Testing more of the protocol. Add tests to more fully exercise the complete AMQP protocol.

Allow scaling of test clients. Each test client should only be run once (in each environment) and they create unique names for themselves. Tests are only run between pairs of single clients, with a single sender and number of receivers defined by the test case (often 1). Clients listen for control messages on topics, and use correlation id's in all tests messages to differentiate themselves were multiple senders to be active. This has been done deliberately to allow for future expansion of the test framework to allow scaling up of the tests by starting more clients.

Appendix A, General Notes:

Brokers that need to be interop tested: C++ and Java

Clients that need to be interop tested: C++ , Java, Java 1.4 retrotranslation, C++, .Net 2.0, .Net 1.1, (Mono?), Python, Ruby.

Appendix B, Example of XML Format for Test Ouput:

I don't think there is a DTD or schema for this but the XML output from JUnit looks like the example below. This is a convenient choice for the output format from these test results even if the code does not actually use JUnit (or cppunit or nunit) iternally, because automated build servers generally understand and are able to produce test reports from it.

Example:

<?xml version="1.0" encoding="UTF-8" ?>
<testsuite errors="0" skipped="0" tests="18" time="0.02" failures="0" name="org.apache.qpid.framing.BasicContentHeaderPropertiesTest">
  <properties>
    <property value="Java(TM) 2 Runtime Environment, Standard Edition" name="java.runtime.name"/>
    ... (there were lots of properties).
  </properties>
  <testcase time="0.02" name="testRejectedExecution"/>
  ... (there were lots of test cases).
</testsuite>

Appendix C, Command line options.

IOP-21 states that general parameters can be passed on the command line using name=value syntax. The coordinator understands the following parameters, and will use them to override the default values for the tests. Individual test cases refer to the command line parameter that they take their test parameters from.

Appendix D, Deleted Requirements:

Put deleted requirements here, in case they can be re-used.