Project Winston - Consolidated Test Data (WIP)

Purpose of Project Winston is to create a centralised test system that will allow the Apache CloudStack community to contribute to testing.  This system can be split into separate functions:

1. Tester submissions of environments types which they have available for testing on
2. Automated submission of compatible tests+test scenarios to testers
3. Running of tests
4. Submission of test results to a central location

This document covers item 4 - submission of tests

Consolidated Test Data Overview

Elasticsearch has been chosen as the storage repository for the combined data, due to the fact that it is designed to handle text documents and it's general popularity which means a large number of people will be able to work with it.

Elasticsearch stores data as JSON documents, the data ingestion is therefore designed to be natively in JSON to avoid the need for additional processing/complexity - ie no need for logstash.

Along with the individual test results, various pieces of metadata are required to form a full picture of the test results.

These are collected in the following files from a Marvin test run:

• test_data.py -> test_data.py_contents
• marvin_cfg file -> env_data
• nose test output (in JSON) -> results
• additional test run metadata -> test_metadata

JSON document structure

{
  "testrun": {
    "testrun_uuid": "$TESTRUN_UUID",
    "results": [{
      "classname": "tests.smoke.test_global_settings:TestUpdateConfigWithScope",
      "type": "success",
      "name": "test_UpdateConfigParamWithScope",
      "time": 0.18374300003051758
      }],
    "test_data.py_contents": "%23+Licensed+to+the+Apache+Software+Foundation+%28ASF%29+under+on......",
    "test_metadata": {
      "base_cloudstack_version": "4.10",
      "pr_id": "null",
      "rc_num": "null",
      "marvin_tags": "advanced",
      "marvin_hypervisor": "kvm",
      "hypervisor_version": "CentOS 6",
      "hypervisor_minor_version": "",
      "mgmt_server_os": "CentOS 7",
      "mgmt_server_os_ver": "7.3",
      "testrun_date": ""
      },
    "env_data": ""

  }
}

Elasticsearch configuration

PUT /test_results

{
  "mappings": {
    "testruns": {
      "properties": {
        "testrun_uuid": {
          "type": "text"
        },
        "results": {
          "type": "nested"
        },
        "env_data": {
          "type": "nested"
        },
        "test_metadata": {
          "type": "nested"
        }
      }

    }
  }
}