Validating check-ins for your local changes, using Simulator

We segregated current Integration test cases in to two categories and added attributed appropriately, as below:

1. "selfservice"  : tests which can run on simulator and thus basically tests CS business logic. These tests does not require real hardware. Tests are added with attribute by  name "required_hardware", with value "false"
2. "provisioning": tests which requires real hardware. Tests are added with attribute by name "required_hardware", with value "true"
   
   

• From developers' point of view, whenever they do a change on their local environment, they can now easily validate the changes by running "selfservice" tests before check-in the change.
• Using "simulator", they can run these tests locally on their development environment, without requiring any hardware .  This will help them to make sure that CS code is not broken with their changes, by running basic integration tests. 

Below are Steps for Running Simulator and Tests against it:

Step1: Get the simulator up and running on your local environment using below mvn commands. 

    mvn -Pdeveloper -Dsimulator clean install -Dmaven.test.skip=true
    mvn -Pdeveloper -pl developer -Ddeploydb
    mvn -Pdeveloper -pl developer -Ddeploydb-simulator
    mvn -pl client jetty:run -Dsimulator

Step2: Create the Data Center using marvin by running the command below:

    Setup Marvin (if not done already): python tools/marvin/setup.py install

    (Requires python-dev or python-devel)

    python tools/marvin/marvin/deployDataCenter.py -i setup/dev/advanced.cfg

Step3:  Now run "selfservice" tests on the created Data Center, using nose command below. Before running tests, mention a new  log folder path for "LogFolderPath", under config file ( setup/dev/advanced.cfg ).

• For running all test suites under a given directory:
  
  nosetests-2.7 --with-marvin --marvin-config=<path_to_marvin_config>   --with-xunit  --xunit-file=<path_to_out_xml>  -w <path_to_test_cases_dir> -a tags=advanced,required_hardware=false  --zone=<zone_name>  --hypervisor=simulator

• For running a single test suite:
  

  nosetests-2.7 --with-marvin --marvin-config=<path_to_marvin_config>   --with-xunit  --xunit-file=<path_to_out_xml>  <path_to_test_suite> -a tags=advanced,required_hardware=false  --zone=<zone_name>  --hypervisor=simulator

  
  
• For running a single test class under a given test suite:
  nosetests-2.7 --with-marvin --marvin-config=<path_to_marvin_config>   --with-xunit  --xunit-file=<path_to_out_xml>  <path_to_test_suite>:<TestClass> -a tags=advanced,required_hardware=false  --zone=<zone_name>  --hypervisor=simulator
  
  

• All test logs post run are available under the mentioned LogFolderPath for your analysis.
• All nose run logs with results are dumped to xunit xml file for your analysis. 
  
  

 Note: 

1. You should have marvin installed on your local environment( This is one time setup ). This is required for deploying Data Center and running tests. To install marvin and use, follow the link. .
   Marvin - Testing with Python
2. The segregation of test cases is currently available in master and 4.4 branch.
3. Any issues you see while running simulator,  you can log a bug for that branch in ASF jira, selecting component "simulator" against CS.
4. Sometimes if your environment has some old marvin or nose installed, pip install --upgrade may not completely remove the old installation. In that case, you need to manually remove them, otherwise, it may fail.
   
   
   

Capabilities of Simulator:

Using Simulator, we can mock many Cloudstack features. With out real hardware, we can use it to create complete data center having zones, pods, clusters, networks etc.

We can create multiple network types like shared, isolated etc, multiple zone types like advanced vs basic etc. Deploy and Stop Instances, Register and Download templates etc.