EC2 Query API Test Suite Automation

These tests are run using jclouds.

All tests are written keeping regions in mind. Null value  have been passed for region. To get the endpoint it was making call to describeRegions, which is not supported by Cloudstack so code changes have been made for that.

The changes are in ec2 project. These  changes are on repo in cloudstack-ec2-queryapi branch .

In summary 34  API calls in tests scenarios are passing. Some failing/skipped ones are dependent on failed tests which are expected to pass once the failed test is ok. The APIs which are passing are as follows

1. describeImages
2. describeInstances
3. describeAddresses
4. createVolume
5. describeVolumes
6. describeSnapshots
7. describeAvailabilityZones
8. createKeyPair
9. describeKeyPairs
10. deleteKeyPair
11. createSecurityGroup
12. deleteSecurityGroup
13. describeSecurityGroups
14. createSnapshot
15. createVolumeFromSnapshot
16. deleteVolume
17. runInstances
18. terminateInstances
19. deleteSnapshot

The API calls which are working fine with jclouds parser after my changes

1. attachVolume
2. detachVolume
3. StopInstances
4. StartInstances
5. RebootInstances
6. DescribeInstanceAttribute
7. CreateImage
8. DeregisterImage
9. DescribeImageAttribute
10. ResetImageAttribute
11. RegisterImage
12. AllocateAddress 
13. AssociateAddress
14. DisassociateAddress
15. ReleaseAddress

These tests are failing or skipped(dependent on failed tests) on jclouds as some API call is broken in jclouds

1. authorizeSecurityGroupIngress
2. revokeSecurityGroupIngress

These tests are not implemented in jclouds

1. ModifyImageAttribute
2. GetPassword
3. createTags
4. describeTags
5. deleteTags
6. ImportKeyPair

Below things are in following format

Class containing below scenarios

1. test scenario
   1. EC2 API Calls made in this scenario (parameters passed)

Tests which are passing

CloudStackQueryEC2AMIClientLiveTest

1. testDescribeImages    It expects two images(normal one and ebs one) so its expectation has been changed to one image.
   1. describeImagesInRegion Here describeImages API call is made without any parameters. This test checks whether images in response are not null and correct in number.
   2.  describeImagesInRegion (imageid) This call is made with parameter imageid. Test checks whether response contains only one image with the image id as imageid.
      

CloudStackQueryEC2InstanceClientLiveTest

1. testDescribeInstancesNo change
   1. describeInstancesInRegion Here describeInstances API call is made without any parameters. This test checks whether instnaces in response are not null and are greater than or equal to 0.

CloudStackQueryEC2ElasticIPAddressClientLiveTest

1. testDescribeAddresses  No Change
   1. describeAddressesInRegion Here describeAddresses API call is made without any parameters. This test checks whether public IP pairs  in response are not null .
   2. describeAddressesInRegion(publicIp) If above call return result greater than 0 then make describeAddresses API call  with  parameter publicIp. It checks whether response contains the single pair with with public IP piblicIp.

CloudStackQueryEC2ElasticBlockStoreClientLiveTest 

1. *testCreateVolumeInAvailabilityZone *have to change result size to 2( as some test not cleaned up the created volume,  work in progress)
   1. createVolumeInAvailabilityZone(zone, size) Here createVolume API call is made with parameters zone and size. It checks whether response is not null and zone of volume is zone
   2. describeVolumesInRegion(volumeid) Here describeVolumes API call is made with parameters volumeid. It checks whether result in response is not null and volume id is volumeid
2. *testDescribeSnapshots *no change
   1. describeSnapshotsInRegion Here describeSnapshots API call is made without any parameters. This test checks whether snapshots in response are not null.
   2. describeSnapshotsInRegion(snapshotid) This call is made with parameter snapshotid. Test checks whether response contains only one snapshot with the snapshot id as snapshotid.
3. testDescribeVolumes    command is working fine 
   1. describeVolumesInRegion Here describeImages API call is made without any parameters. This test checks whether  response is parsed correctly.

CloudStackQueryEC2AvailabilityZoneAndRegionClientLiveTest

1. testDescribeAvailabilityZones  no change
   1. describeAvailabilityZonesInRegion Here describeAvailabilityZones API call is made without any parameters. This test checks whether Availability Zones in response are not null and correct in number.
   2. describeAvailabilityZonesInRegion(zoneid) This call is made with parameter zoneid. Test checks whether response contains only one zone with the zone id as zoneid.

CloudStackQueryEC2KeyPairClientLiveTest

1. testCreateKeyPair    *have to remove one delete keypair call as this was deleting non-existent keypair.
   1. createKeyPairInRegion(keyName) Here createKeyPair API call is made with parameter keyName. This test checks whether result in response is not null. It also checks whether various parameters of key Pair are not null. Parameters checked are keyMaterial, SHA1 of private Key and keyName.
   2. describeKeyPairsInRegion(keyName) Here describeKeyPairs API call is called with  parameter keyName. It checks whether response contains the single pair with with key pair name as keyname.
   3. deleteKeyPairInRegion(keyName) Here deleteKeyPair API call is made with parameter keyName. It checks whether response is parsed correctly by jclouds
2. testDescribeKeyPairschanges related to region
   1. describeKeyPairsInRegion Here describeKeyPairs API call is made without any parameters. This test checks whether Key Paiirs in response are not null .
   2. describeKeyPairsInRegion(keyName) If above call return result greater than 0 then make describeKeyPairs API call  with  parameter keyName. It checks whether response contains the single pair with with key pair name as keyname.

CloudStackQueryEC2SecurityGroupClientLiveTest

1. testCreateSecurityGroup *have to remove one delete security group call as this was deleting non-existent security group.
   1. createSecurityGroupInRegion(groupName, groupDescription) Here createSecurityGroup API call is made with parameters groupName and groupDescription. It checks whether response is not null and and group is created with parameters passed correctly.
   2. deleteSecurityGroupInRegion(groupName) Here deleteSecurityGroup API call is made with parameter groupName. It checks whether response is parsed correctly by jclouds
2. testDescribe   *changes related to region
   1. describeSecurityGroupsInRegion Here describeSecurityGroups API call is made without any parameters. This test checks whether security groups  in response are not null.
   2. describeSecurityGroupsInRegion(groupName) If above call return result greater than 0 then make describeSecurityGroups API call  with  parameter groupName. It checks whether response contains the single security group with with name groupName.

Tests which are failing

CloudStackQueryEC2AMIClientLiveTest

1. testDescribeImageBadId   This is negative test, work in progress .
2. testDescribeImageNotExists This is negative test, work in progress.

CloudStackQueryEC2SecurityGroupClientLiveTest

1. testAuthorizeSecurityGroupIngressCidr     For Ingress tests generated API call is not in correct format i.e. parameters missing Ippermissions.n prefix.
   1. createSecurityGroupInRegion(groupName)
   2. authorizeSecurityGroupIngressInRegion(groupName, IpProtocol, Ippermissions)
   3. revokeSecurityGroupIngressInRegion(groupName, IpProtocol, Ippermissions)
   4. deleteSecurityGroupInRegion(groupName)
2. testAuthorizeSecurityGroupIngressSourceGroup
   1. createSecurityGroupInRegion(groupName)
   2. authorizeSecurityGroupIngressInRegion(groupName, IpProtocol, Ippermissions)
   3. revokeSecurityGroupIngressInRegion(groupName, IpProtocol, Ippermissions)
   4. deleteSecurityGroupInRegion(groupName)
3. testAuthorizeSecurityGroupIngressSourcePort
   1. createSecurityGroupInRegion(groupName)
   2. authorizeSecurityGroupIngressInRegion(groupName, IpProtocol, Ippermissions)
   3. revokeSecurityGroupIngressInRegion(groupName, IpProtocol, Ippermissions)
   4. deleteSecurityGroupInRegion(groupName)

CloudStackQueryEC2ElasticBlockStoreClientLiveTest

1. testCreateSnapshotInRegionSpecified VolumeId is not in Ready state, but  in state Allocated. Cannot take snapshot. It requires created instance but did not contain any instance creation call. It depends on attach test which is not written. For cloudstack it needs instance to attach volume.
   1. createSnapshotInRegion(volumeId)
   2. describeSnapshotsInRegion(snapshotId)
      

Skipped test which are dependent on failed tests

CloudStackQueryEC2ElasticBlockStoreClientLiveTest

1. testCreateVolumeFromSnapshotInAvailabilityZone
   1. createVolumeFromSnapshotInAvailabilityZone(snapshotId)
   2. deleteVolumeInRegion(volumeId)
   3. describeVolumesInRegion(volumeId)
2. testCreateVolumeFromSnapshotInAvailabilityZoneWithSize
   1. createVolumeFromSnapshotInAvailabilityZone(snapshotId)
   2. deleteVolumeInRegion(volumeId)
   3. describeVolumesInRegion(volumeId)
3. testDeleteVolumeInRegion
   1. deleteVolumeInRegion(volumeId)
   2. describeVolumesInRegion(volumeId)
4. testGetCreateVolumePermissionForSnapshot
   1. getCreateVolumePermissionForSnapshotInRegion(snapshotId)
5. testDeleteSnapshotInRegion
   1. deleteSnapshotInRegion(snapshotId)
   2. describeSnapshotsInRegion(snapshotId)
      

Tests which are not implemented in jclouds but test available as TODO.

1. testAttachVolumeInRegion requires instance creation
2. testDetachVolumeInRegion

API Calls which are not called in any tests

1. ModifyImageAttribute
2. ImportKeyPair
3. AllocateAddress 
4. AssociateAddress
5. DisassociateAddress
6. ReleaseAddress
7. GetPassword
   

Things which are not in jclouds or not working

1. Tags functionality is not present in jclouds in ec2 project
2. Ingress related calls are generated without Ippermissions.n prefix in parameters
3. No importKeyPair API Call