EC2 Query API Test Suite Automation

These tests are run using jclouds.

All tests are written keeping regions in mind. Null value  have been passed for region. To get the endpoint it was making call to describeRegions, which is not supported by Cloudstack so code changes have been made for that.

A  new project cloudsatck-query-ec2  have been created in jclouds. These changes are on repo in cloudstack-ec2-queryapi branch and  in cloudstack-query-ec2 project.

In summary 13  API calls in tests scenarios are passing. Some failing/skipped ones are dependent on failed tests which are expected to pass once the failed test is ok. The APIs which are passing are as follows

1. describeImages
2. describeInstances
3. describeAddresses
4. createVolume
5. describeVolumes
6. describeSnapshots
7. describeAvailabilityZones
8. createKeyPair
9. describeKeyPairs
10. deleteKeyPair
11. createSecurityGroup
12. deleteSecurityGroup
13. describeSecurityGroups

These tests are failing or skipped(dependent on failed tests)

1. authorizeSecurityGroupIngress
2. revokeSecurityGroupIngress
3. createSnapshot
4. createVolumeFromSnapshot
5. deleteVolume
6. attachVolume
7. detachVolume
8. runInstances
9. terminateInstances
10. deleteSnapshot

These tests are not implemented

1. StopInstances
2. StartInstances
3. RebootInstances
4. DescribeInstanceAttribute
5. CreateImage
6. DeregisterImage
7. DescribeImageAttribute
8. ModifyImageAttribute
9. ResetImageAttribute
10. RegisterImage
11. ImportKeyPair
12. AllocateAddress 
13. AssociateAddress
14. DisassociateAddress
15. ReleaseAddress
16. GetPassword
17. createTags
18. describeTags
19. deleteTags

Below things are in following format

Class containing below scenarios

1. test scenario
   1. EC2 API Calls made in this scenario (parameters passed)

Below information describe the API calls covered with what parameters. For Example

For CloudStackQueryEC2AMIClientLiveTest (these changes are done to pass this test)

1. This test scenario(testDescribeImages) tests following API call
   1. This API call is made (describeImagesInRegion) and returns the correct response i.e. parsed successfully by jclouds
   2. This API call is made ( describeImagesInRegion with parameter imageid ) and returns the correct response i.e. parsed successfully by jclouds.

Tests which are passing

CloudStackQueryEC2AMIClientLiveTest

1. testDescribeImages    It expects two images(normal one and ebs one) so its expectation has been changed to one image.
   1. describeImagesInRegion
   2. describeImagesInRegion (imageid)
      

CloudStackQueryEC2InstanceClientLiveTest

1. testDescribeInstancesNo change
   1. describeInstancesInRegion

CloudStackQueryEC2ElasticIPAddressClientLiveTest

1. testDescribeAddresses  No Change
   1. describeAddressesInRegion
   2. describeAddressesInRegion(publicIp)

CloudStackQueryEC2ElasticBlockStoreClientLiveTest 

1. *testCreateVolumeInAvailabilityZone *have to change result size to 2( as some test not cleaned up the created volume,  work in progress)
   1. createVolumeInAvailabilityZone(zone)
   2. describeVolumesInRegion(volumeid)
2. *testDescribeSnapshots *no change
   1. describeSnapshotsInRegion
   2. describeSnapshotsInRegion(snapshotid)
3. testDescribeVolumes    command is working fine 
   1. describeVolumesInRegion

CloudStackQueryEC2AvailabilityZoneAndRegionClientLiveTest

1. testDescribeAvailabilityZones  no change
   1. describeAvailabilityZonesInRegion
   2. describeAvailabilityZonesInRegion(zoneid)

CloudStackQueryEC2KeyPairClientLiveTest

1. *testCreateKeyPair    *have to remove one delete keypair call as this was deleting non-existent keypair.
   1. createKeyPairInRegion(keyName)
   2. describeKeyPairsInRegion(keyName)
   3. deleteKeyPairInRegion(keyName)
2. testDescribeKeyPairschanges related to region
   1. describeKeyPairsInRegion
   2. describeKeyPairsInRegion(keyName)

CloudStackQueryEC2SecurityGroupClientLiveTest

1. *testCreateSecurityGroup *have to remove one delete security group call as this was deleting non-existent security group.
   1. createSecurityGroupInRegion(groupName, groupDescription)
   2. deleteSecurityGroupInRegion(groupName)
2. *testDescribe   *changes related to region
   1. describeSecurityGroupsInRegion
   2. describeSecurityGroupsInRegion(groupName)
      

Tests which are failing

CloudStackQueryEC2AMIClientLiveTest

1. testDescribeImageBadId   This is negative test, work in progress .
2. testDescribeImageNotExists This is negative test, work in progress.

CloudStackQueryEC2SecurityGroupClientLiveTest

1. testAuthorizeSecurityGroupIngressCidr     For Ingress tests generated API call is not in correct format i.e. parameters missing Ippermissions.n prefix.
   1. createSecurityGroupInRegion(groupName)
   2. authorizeSecurityGroupIngressInRegion(groupName, IpProtocol, Ippermissions)
   3. revokeSecurityGroupIngressInRegion(groupName, IpProtocol, Ippermissions)
   4. deleteSecurityGroupInRegion(groupName)
2. testAuthorizeSecurityGroupIngressSourceGroup
   1. createSecurityGroupInRegion(groupName)
   2. authorizeSecurityGroupIngressInRegion(groupName, IpProtocol, Ippermissions)
   3. revokeSecurityGroupIngressInRegion(groupName, IpProtocol, Ippermissions)
   4. deleteSecurityGroupInRegion(groupName)
3. testAuthorizeSecurityGroupIngressSourcePort
   1. createSecurityGroupInRegion(groupName)
   2. authorizeSecurityGroupIngressInRegion(groupName, IpProtocol, Ippermissions)
   3. revokeSecurityGroupIngressInRegion(groupName, IpProtocol, Ippermissions)
   4. deleteSecurityGroupInRegion(groupName)

CloudStackQueryEC2ElasticBlockStoreClientLiveTest

1. testCreateSnapshotInRegionSpecified VolumeId is not in Ready state, but  in state Allocated. Cannot take snapshot. It requires created instance but did not contain any instance creation call. It depends on attach test which is not written. For cloudstack it needs instance to attach volume.
   1. createSnapshotInRegion(volumeId)
   2. describeSnapshotsInRegion(snapshotId)
      

Skipped test which are dependent on failed tests

CloudStackQueryEC2ElasticBlockStoreClientLiveTest

1. testCreateVolumeFromSnapshotInAvailabilityZone
   1. createVolumeFromSnapshotInAvailabilityZone(snapshotId)
   2. deleteVolumeInRegion(volumeId)
   3. describeVolumesInRegion(volumeId)
2. testCreateVolumeFromSnapshotInAvailabilityZoneWithSize
   1. createVolumeFromSnapshotInAvailabilityZone(snapshotId)
   2. deleteVolumeInRegion(volumeId)
   3. describeVolumesInRegion(volumeId)
3. testDeleteVolumeInRegion
   1. deleteVolumeInRegion(volumeId)
   2. describeVolumesInRegion(volumeId)
4. testGetCreateVolumePermissionForSnapshot
   1. getCreateVolumePermissionForSnapshotInRegion(snapshotId)
5. testDeleteSnapshotInRegion
   1. deleteSnapshotInRegion(snapshotId)
   2. describeSnapshotsInRegion(snapshotId)
      

Tests which are not implemented in jclouds but test available as TODO.

1. testAttachVolumeInRegion requires instance creation
2. testDetachVolumeInRegion

API Calls which are not called in any tests

1. StopInstances
2. StartInstances
3. RebootInstances
4. DescribeInstanceAttribute
5. CreateImage
6. DeregisterImage
7. DescribeImageAttribute
8. ModifyImageAttribute
9. ResetImageAttribute
10. RegisterImage
11. ImportKeyPair
12. AllocateAddress 
13. AssociateAddress
14. DisassociateAddress
15. ReleaseAddress
16. GetPassword
    

Things which are not in jclouds or not working

1. Tags functionality is not present in jclouds in ec2 project
2. Ingress related calls are generated without Ippermissions.n prefix in parameters
3. No importKeyPair API Call