Phase1: Marvin Changes

1. Fixed few issues reported.

2. Separating TestClient from deployDataCenter: Earlier, testclient creation is also part of deployDatacenter. It is not part of this module idea to create test client in general. Now with this separation, testclient creation is now moved to cloudstackTestClient module and is responsible for creating and providing the test client for regression run. Added changes to both deployDataCenter and cloudstackTestClient. Fixed few misc issues and did some clean up here. Now deployDC will just take the required configuration and parameters and create the DataCenter we needed.

3. Deploy data center earlier continues even if there is a problem with any api call. Now, if there are any failures, it will check for return value, exit with dumping to log. There is no way to delete a data center created earlier. Added a new DeleteDataCenter Interface to delete the earlier created datacenter. If user wants to delete a datacenter created and recreate, deleting has to be manual earlier. With this change, he can delete a data center created earlier. It will be useful at many places for testing to destroy and recreate with new settings etc.
when deploy data center runs, it does not adequately verifies whether a given entry is created or not, it just continues. Added checks to see if an entry is not successful, it reports, dumps to log and exit gracefully.
DataCenter creation as Transaction : At any given point of time during deploy data center, if an issue happens half way through, earlier created entries were still available in CS, it wont clean up i.e., data center is half created, Again, if tried to recreate , the earlier existing entries wont allow to recreate. Now, data center works as transaction, either all entries are created as part of data center or none, added a clean up to delete the half created data center.
Export the created Data Center so that user can use it to further delete the data center with these exported entries etc.

4. Logging enhancements: During the regression run,earlier we had various and a little confusing log options. Now, marvin run logs will be streamlined at one place. Here, for the entire run, we will see different logs viz., run\results\exceptions+passfail log all at one place. We have added MarvinLog to explicitly provide logging facility to Marvin. Before running test suites, we just need to configure the path for logs. MarvinInit along with MarvinLog will now create all logs required for the run under this path. Before every run, it will clean up the previously created MarvinLogs folder before proceeding with run. MarvinLogs folder will be created at the mentioned and configured path and all test suites will dump different logs here. Each test run will now have a test module name for log for easy identification. Logs for Run info, exception information ,results information etc are all provided here. So, all logs separated by different category help resolve issues quickly. The log folder path once configured, will be used to provide logs separately for each test module. All Test script Exceptions will now be caught at one place under exception\failed log. It will help identify script issues vs product issues. Added few more log levels here.

https://issues.apache.org/jira/browse/CLOUDSTACK-5498

5. Earlier, there were few hard coding for running test modules against a CS setup. Under Test Suites, we had some hard coding values used to pickup first zone, first pod, first host,first template etc. So, even though we had multiple zones, hosts, it always used to pickup first zone,pod etc. With new changes added, we can now run automation for multiple hosts across zones on one management server driven by configuration. Earlier, If we are to run BVT\Regression per host, we need to clean up and start CS for next run basically create a new deployDC for next host run. Now, with changes, objective is to run multiple regressions on single CS across multiple zones. Each test suite is now parameterized to take the zone parameter and can run bvt\regression on a given zone. We don't need to start\stop the management server to run regression for xen, kvm etc. All can now run across multiple zones and test modules will now directed to run against which zone etc through a config etc. This will help test Management Server with multiple deployments and parallel regression runs for multiple hosts, each one in each zone. This will give good enhancements to CS testing and multiple zones,hosts under zone can be used for testing parallelly.
https://issues.apache.org/jira/browse/CLOUDSTACK-5674

6. Marvin Init Enhancements: Earlier, we didn't had Marvin initialization, the object of initialization is to create test client, parse configuration using config library, create logging structures, deploy DC, parse test data required. All these are done at initialization state and are made available to test suites through test client. Earlier test and data are little tied up through service class. Need to remove all data required from tests to be data driven. Based upon the config provided, tests will run now.

7. We added MarvinInit to initialize marvin earlier. Now, added few more changes to it. So, the init will exit based upon testclient creation, logging creation, config etc and exit.

8. cloudstackConnection.py: Did few changes to this module. Fixed a bug and there was some cluttered code earlier. Did some clean up here. marvinRequest used by cloudstackClientAPI for sending requests to CS will now return FAILED in case of any issue like exception, invalid arguments, etc. Test Modules can now verify whether the command is FAILED or not before proceeding further. Currently, at many places we just proceed with out verifying the output from marvinRequest.

9. ConfigManager: Changed few naming conventions, cleanup and config parameters.

10. Added few log levels to runlogger, so tests can dump the log type they wanted. Tests should use exception\error for dumping that level information compared to debug level.

11. Earlier, if test modules has issues with either invalid import, indentation etc were prepared to run by noseplugin. Added explicit check to import and see its importability and if not exit with log rather than proceeding further. Avoids unwanted exceptions from noseplugin for bad modules.

Note1: Now, related to note5, we need few changes to tests. I have logged and assigned bug to track this. Once this change is done, we will proceed to do more changes to run regression on new infrastructure.

Note2: Need to Test the changes, will soon move all these changes to topic branch created for marvin refactoring. Once "tested" and agreed , merge the changes with main branches of CS.

Space shortcuts

Child pages