Virtual Router: Smoke and Integration tests gap analysis

This is summary of gap analysis done on Marvin tests that touch the virtual router functionality.

Networking in CloudStack is feature rich and flexible. Networks can be composed with flexible set of services. Network can be deployed in many different models. Here is the listing of some of the dimensions in which networks and functionality can be deployed/used.

Network (only VR) supported services

• DHCP
• DNS
• Gateway
• Firewall
• LB
• VPN
• SourceNAT
• StaticNAT
• PortForwarding
• Network ACL
• UserData
• MetaData
• Password

VR service capabilities

• LB
  • LB algorithms
    • round robin
    • least connection
    • source
  • LB isolation
    • dedicated
  • HA proxy stickiness
  • protocols
• Firewall
  • traffic direction (ingress, egress)
  • supported ingress/egress protocols
  • multiple IP's
• VPN
  • supported protocols (l2tp, pptp, ipsec)
  • vpn type: remote access vpn
• DNS
  • suffix modification
• DHCP
  • multiple subnets
• Source NAT
  • per account
  • per network

Network offering

• pick and choose set of services and capabilities

Networking isolated Type:

• isolated
• shared network with just (DNS, DHCP, user data)
• shared network with full services
• shared network with security groups

Zone type:

• Advanced Zone
• Basic Zone

Network model

• VPC
• non VPC

Network life cycle

• create
• delete
• restart
• update
• restart
• update

Router life cycle

• start router
• stop router
• reboot router
• destroy router
• change service offering for router

Network Garbage collection

• network is shutdown in GC, and start again on VM deployment

Redundant routers

• master/slave fail overs

VR split network service other network elements

Upgrade

Component tests

test_acl_isolatednetwork.py: test access control on network

• test_01_createNetwork_admin
• test_02_createNetwork_admin_foruserinsamedomain
• test_03_createNetwork_admin_foruserinotherdomain
• test_04_createNetwork_domaindmin
• test_05_createNetwork_domaindmin_foruserinsamedomain
• test_06_createNetwork_domaindmin_foruserinsubdomain
• test_07_createNetwork_domaindmin_forcrossdomainuser
• test_08_createNetwork_user
• test_09_createNetwork_user_foruserinsamedomain
• test_10_createNetwork_user_foruserinotherdomain
• test_11_deployvm_admin
• test_12_deployvm_admin_foruserinsamedomain
• test_13_deployvm_admin_foruserinotherdomain
• test_13_1_deployvm_admin_foruserinotherdomain_crossnetwork
• test_14_deployvm_domaindmin
• test_15_deployvm_domaindmin_foruserinsamedomain
• test_16_deployvm_domaindmin_foruserinsubdomain
• test_17_deployvm_domaindmin_forcrossdomainuser
• test_17_1_deployvm_domainadmin_foruserinotherdomain_crossnetwork
• test_18_deployvm_user
• test_19_deployvm_user_foruserinsamedomain
• test_20_deployvm_user_foruserincrossdomain
• test_20_1_deployvm_user_incrossnetwork
• test_21_restartNetwork_admin
• test_22_restartNetwork_admin_foruserinsamedomain
• test_23_restartNetwork_admin_foruserinotherdomain
• test_24_restartNetwork_domaindmin
• test_25_restartNetwork_domaindmin_foruserinsamedomain
• test_26_restartNetwork_domaindmin_foruserinsubdomain
• test_27_restartNetwork_domaindmin_forcrossdomainuser
• test_28_restartNetwork_user
• test_29_restartNetwork_user_foruserinsamedomain
• test_30_restartNetwork_user_foruserinotherdomain

test_acl_isolatednetwork_delete.py: test access control on delete network

• test_deleteNetwork_admin
• test_deleteNetwork_admin_foruserinsamedomain
• test_deleteNetwork_admin_foruserinotherdomain
• test_deleteNetwork_domaindmin
• test_deleteNetwork_domaindmin_foruserinsamedomain
• test_deleteNetwork_domaindmin_foruserinsubdomain
• test_deleteNetwork_domaindmin_forcrossdomainuser
• test_deleteNetwork_user
• test_deleteNetwork_user_foruserinsamedomain
• test_deleteNetwork_user_foruserinotherdomain

test_acl_sharednetwork.py: Test cases for access control for Vm deployment in Shared Network with scope all,account and Domain

• test_deployVM_in_sharedNetwork_scope_all_domainuser
• test_deployVM_in_sharedNetwork_scope_all_domainadminuser
• test_deployVM_in_sharedNetwork_scope_all_subdomainuser
• test_deployVM_in_sharedNetwork_scope_all_subdomainadminuser
• test_deployVM_in_sharedNetwork_scope_all_ROOTuser
• test_deployVM_in_sharedNetwork_scope_all_ROOTadmin
• test_deployVM_in_sharedNetwork_scope_domain_nosubdomainaccess_domainuser
• test_deployVM_in_sharedNetwork_scope_domain_nosubdomainaccess_domainadminuser
• test_deployVM_in_sharedNetwork_scope_domain_nosubdomainaccess_subdomainuser
• test_deployVM_in_sharedNetwork_scope_domain_nosubdomainaccess_subdomainadminuser
• test_deployVM_in_sharedNetwork_scope_domain_nosubdomainaccess_parentdomainuser
• test_deployVM_in_sharedNetwork_scope_domain_nosubdomainaccess_parentdomainadminuser
• test_deployVM_in_sharedNetwork_scope_domain_nosubdomainaccess_ROOTuser
• test_deployVM_in_sharedNetwork_scope_domain_nosubdomainaccess_ROOTadmin
• test_deployVM_in_sharedNetwork_scope_domain_withsubdomainaccess_domainuser
• test_deployVM_in_sharedNetwork_scope_domain_withsubdomainaccess_domainadminuser
• test_deployVM_in_sharedNetwork_scope_domain_withsubdomainaccess_subdomainuser
• test_deployVM_in_sharedNetwork_scope_domain_withsubdomainaccess_subdomainadminuser
• test_deployVM_in_sharedNetwork_scope_domain_withsubdomainaccess_parentdomainuser
• test_deployVM_in_sharedNetwork_scope_domain_withsubdomainaccess_parentdomainadminuser
• test_deployVM_in_sharedNetwork_scope_domain_withsubdomainaccess_ROOTuser
• test_deployVM_in_sharedNetwork_scope_domain_withsubdomainaccess_ROOTadmin
• test_deployVM_in_sharedNetwork_scope_account_domainuser
• test_deployVM_in_sharedNetwork_scope_account_domainadminuser
• test_deployVM_in_sharedNetwork_scope_account_user
• test_deployVM_in_sharedNetwork_scope_account_differentdomain
• test_deployVM_in_sharedNetwork_scope_account_ROOTuser
• test_deployVM_in_sharedNetwork_scope_account_ROOTadmin

test_acl_sharednetwork_deployVM-impersonation.py: Test cases for deploying Virtual Machine using impersonation (passing account and domainId parameters) for shared network

• test_deployVM_in_sharedNetwork_as_admin_scope_all_domainuser
• test_deployVM_in_sharedNetwork_as_admin_scope_all_domainadminuser
• test_deployVM_in_sharedNetwork_as_admin_scope_all_subdomainuser
• test_deployVM_in_sharedNetwork_as_admin_scope_all_subdomainadminuser
• test_deployVM_in_sharedNetwork_as_admin_scope_all_ROOTuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_nosubdomainaccess_domainuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_nosubdomainaccess_domainadminuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_nosubdomainaccess_subdomainuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_nosubdomainaccess_subdomainadminuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_nosubdomainaccess_parentdomainuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_nosubdomainaccess_parentdomainadminuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_nosubdomainaccess_ROOTuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_withsubdomainaccess_domainuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_withsubdomainaccess_domainadminuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_withsubdomainaccess_subdomainuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_withsubdomainaccess_subdomainadminuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_withsubdomainaccess_parentdomainuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_withsubdomainaccess_parentdomainadminuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_withsubdomainaccess_ROOTuser
• test_deployVM_in_sharedNetwork_as_admin_scope_account_domainuser
• test_deployVM_in_sharedNetwork_as_admin_scope_account_domainadminuser
• test_deployVM_in_sharedNetwork_as_admin_scope_account_user
• test_deployVM_in_sharedNetwork_as_admin_scope_account_differentdomain
• test_deployVM_in_sharedNetwork_as_admin_scope_account_ROOTuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_all_domainuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_all_domainadminuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_all_subdomainuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_all_subdomainadminuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_all_ROOTuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_all_crossdomainuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_nosubdomainaccess_domainuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_nosubdomainaccess_domainadminuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_nosubdomainaccess_subdomainuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_nosubdomainaccess_subdomainadminuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_nosubdomainaccess_parentdomainuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_nosubdomainaccess_parentdomainadminuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_nosubdomainaccess_ROOTuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_withsubdomainaccess_domainuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_withsubdomainaccess_domainadminuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_withsubdomainaccess_subdomainuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_withsubdomainaccess_subdomainadminuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_withsubdomainaccess_parentdomainuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_withsubdomainaccess_parentdomainadminuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_withsubdomainaccess_ROOTuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_account_domainuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_account_domainadminuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_account_user
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_account_differentdomain
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_account_ROOTuser
• test_deployVM_in_sharedNetwork_as_regularuser_scope_all_anotherusersamedomain
• test_deployVM_in_sharedNetwork_as_regularuser_scope_all_crossdomain

test_add_remove_network.py: tests for Add Remove Network to VM

• test_01_add_nw_running_vm
• test_02_add_nw_stopped_vm
• test_03_add_nw_multiple_times
• test_04_vpc_nw_running_vm
• test_05_add_vpc_nw_stopped_vm
• test_06_add_nw_ipaddress_running_vm
• test_10_add_nw_invalid_ipaddress_running_vm
• test_14_add_nw_different_account
• test_24_add_nw_different_domain
• test_25_add_nw_above_account_limit
• test_07_remove_nic_running_vm
• test_08_remove_default_nic
• test_09_remove_foreign_nic
• test_29_remove_nic_CS22503
• test_30_remove_nic_reattach
• test_vm = VirtualMachine.create
• test_vm.stop
• test_vm.remove_nic
• test_vm.add_nic
• test_vm.start
• test_11_update_nic_running_vm
• test_12_make_default_nic_as_default
• test_13_set_foreign_nic_as_default
• test_15_add_nic_wrong_vm_id
• test_16_add_nic_wrong_network_id
• test_17_add_nic_different_zone
• test_18_add_nic_basic_zone
• test_26_add_nic_insufficient_permission
• test_19_remove_nic_wrong_vm_id
• test_20_remove_nic_wrong_nic_id
• test_27_remove_nic_insufficient_permission
• test_21_update_nic_wrong_vm_id
• test_22_update_nic_wrong_nic_id
• test_23_update_nic_incorrect_vm_state
• test_28_update_nic_insufficient_permission

test_advancedsg_networks.py: tests for networks in advanced zone with security groups

• test_01_createZone_secGrp_enabled
• test_02_createZone_secGrp_disabled
• test_03_createIsolatedNetwork
• test_04_createSharedNetwork_withoutSG
• test_05_deployVM_SharedwithSG
• test_06_SharedNwSGAccountSpecific
• test_07_SharedNwSG_DomainWide_SubdomainAcccess
• test_08_SharedNwSGAccountSpecific_samevlan_samesubnet
• test_09_SharedNwDomainWide_samevlan_samesubnet
• test_10_deleteSharedNwSGAccountSpecific_InUse
• test_11_deleteSharedNwSG_DomainWide_InUse
• test_29_deleteSharedNwSG_ZoneWide_InUse
• test_12_deleteSharedNwSGAccountSpecific_NotInUse
• test_13_deleteSharedNwSG_DomainWide_NotInUse
• test_30_deleteSharedNwSG_ZoneWide_NotInUse
• test__14_createSharedNwWithSG_withoutParams
• test__15_createVPC
• test__16_AccountSpecificNwAccess
• test__17_DomainSpecificNwAccess
• test_18_DeployVM_NoFreeIPs
• test_19_DeployVM_DefaultSG
• test_20_DeployVM_SecGrp_sharedNetwork
• test_24_DeployVM_Multiple_Shared_Networks
• test_25_Deploy_Multiple_VM_Different_Shared_Networks_Same_SG
• test_26_Destroy_Deploy_VM_NoFreeIPs
• test_27_start_stop_vm
• test_28_destroy_recover_expunge_vm
• test_31_Deploy_VM_multiple_shared_networks_sg
• test_33_VM_Migrate_SharedNwSG
• test_34_DeployVM_in_SecondSGNetwork
• test_21_DeployVM_WithoutSG
• test_22_DeployVM_With_Custom_SG
• test_23_DeployVM_MultipleSG
• test_32_delete_default_security_group
• test_34_restart_shared_network_sg
• test_35_Enable_Host_Maintenance
• test_36_ssh_vm_other_sg
• test_37_ping_vm_other_sg
• test_38_ssh_vm_other_sg_new_vm
• test_39_ssh_vm_other_sg_from_multiple_sg_vm
• test_40_ssh_vm_other_sg_reboot

test_egress_fw_rules.py: Egress Firewall component tests

• test_01_egress_fr1: Test By-default the communication from guest n/w to public n/w is allowed.
• test_01_1_egress_fr1: Test By-default the communication from guest n/w to public n/w is NOT allowed
• test_02_egress_fr2: Test Allow Communication using Egress rule with CIDR + Port Range + Protocol
• test_02_1_egress_fr2: Test Allow Communication using Egress rule with CIDR + Port Range + Protocol.
• test_03_egress_fr3: Test communication allowed on network  with network offering with egress policy is allow
• test_03_1_egress_fr3: Test communication blocked on network with network offering with egress policy is deny
• test_04_egress_fr4: Test Create Egress rule and check the Firewall_Rules DB table
• test_04_1_egress_fr4: Test Create Egress rule and check the Firewall_Rules DB table
• test_05_egress_fr5: Test Create Egress rule and check the IP tables on network with network offering egress policy is allow
• test_05_1_egress_fr5: Test Create Egress rule and check the IP tables on network with network offering egress policy is deny
• test_06_egress_fr6: Test Create Egress rule without CIDR on network with network offering egress policy is allow
• test_06_1_egress_fr6: Test Create Egress rule without CIDR on network with network offering egress policy is deny
• test_07_egress_fr7: Test Create Egress rule without End Port
• test_07_1_egress_fr7: Test Create Egress rule without End Port
• test_08_egress_fr8: Test Port Forwarding and Egress Conflict
• test_08_1_egress_fr8: Test Port Forwarding and Egress Conflict
• test_09_egress_fr9: Test Delete Egress rule
• test_09_1_egress_fr9: Test Delete Egress rule
• test_10_egress_fr10: Test Invalid CIDR and Invalid Port ranges
• test_10_1_egress_fr10: Test Invalid CIDR and Invalid Port ranges
• test_11_egress_fr11: Test Regression on Firewall + PF + LB + SNAT
• test_11_1_egress_fr11:Test Regression on Firewall + PF + LB + SNAT
• test_12_egress_fr12: Test Reboot Router
• test_12_1_egress_fr12: Test Reboot Router
• test_13_egress_fr13: Test Redundant Router : Master failover
• test_13_1_egress_fr13: Test Redundant Router : Master failover

 test_haproxy.py: HAProxy tests

• test_01_create_sticky_policy_default_values
• test_02_create_sticky_policy_custom_values
• test_03_supported_policies_by_network
• test_04_delete_lb_rule
• test_05_error_alerts_after_create
• test_06_release_ip
• test_07_delete_account
• test_08_create_policy_router_stopped
• test_09_create_policy_router_destroy
• test_10_create_policy_enable_disable_vpn
• test_11_invalid_params

test_ip_reservation.py: test for guest IP reservations

• test_vm_create_after_reservation
• test_vm_create_outside_cidr_after_reservation
• test_update_cidr_multiple_vms_not_all_inclusive
• test_update_cidr_single_vm_not_inclusive
• test_nat_rules
• test_RVR_network
• test_ip_reservation_in_multiple_networks_same_account
• test_restart_network_with_cleanup
• test_update_network_guestvmcidr
• test_reservation_after_router_restart
• test_destroy_recreate_router
• test_network_not_implemented
• test_vm_create_after_reservation
• test_reservation_after_router_restart
• test_vm_create_outside_cidr_after_reservation

 test_lb_secondary_ip.py:  tests for Load balancing rules for secondary IP

• test_01_lb_rule_for_primary_ip
• test_02_lb_rule_for_primary_ip
• test_03_lb_rule_for_primary_and_secondary_ip
• test_04_lb_rule_primary_secondary_multiple_vms
• test_05_lb_rule_wrong_vm_id
• test_06_lb_rule_wrong_vm_ip
• test_07_lb_rule_used_ip
• test_08_lb_rule_remove_used_ip
• test_09_lbvmips_true
• test_10_lbvmips_false
• test_11_ssh_to_secondary_ip
• test_12_ssh_to_primary_secondary_ip
• test_13_delete_lb_rule
• test_14_remove_lb_rule_secondary_ip
• test_15_remove_lb_rule_primary_ip
• test_16_delete_vm_from_lb_rule
• test_20_destroy_recover_vm
• test_21_modify_lb_rule_algorithm
• test_17_restart_router
• test_18_restart_network_cleanup_true
• test_19_restart_network_cleanup_false
• test_22_network_gc
• test_23_lb_rule_functioning_with_netscaler

test_multiple_ip_ranges.py: test related to multiple public IP ranges

• test_01_add_ip_same_cidr
• test_02_add_ip_diff_cidr
• test_03_del_ip_range
• test_04_add_noncontiguous_ip_range
• test_05_add_overlapped_ip_range
• test_06_add_ip_range_overlapped_with_two_ranges
• test_07_add_iprange_superset
• test_08_add_iprange_subset

test_multiple_ips_per_nic.py: Multiple IPs per NIC feature

• test_add_ip_to_nic
• test_remove_ip_from_nic
• test_remove_invalid_ip
• test_list_nics
• test_operations_non_root_admin_api_client
• test_add_PF_rule
• test_delete_PF_nat_rule
• test_disassociate_ip_mapped_to_secondary_ip_through_PF_rule
• test_add_static_nat_rule
• test_disable_static_nat
• test_delete_vm
• test_recover_vm
• test_network_restart_cleanup_true
• test_network_restart_cleanup_false
• test_reboot_router_VM

test_network_offering.py

• test_01_network_off_without_conserve_mode
• test_02_network_off_with_conserve_mode
• test_03_network_off_CS5332
• test_04_network_without_domain_CS19303
• test_01_nwupgrade_netscaler_conserve_on
• test_02_nwupgrade_netscaler_conserve_off
• test_create_network_with_snat

test_redundent_router (test_redundant_router_cleanups.py, test_redundant_router_services.py, test_redundant_router_upgrades.py)

• test_restart_ntwk_no_cleanup
• test_restart_ntwk_with_cleanup
• test_network_gc
• test_restart_network_with_destroyed_masterVR
• test_enableVPNOverRvR
• test_upgradeVR_to_redundantVR
• test_downgradeRvR_to_VR

test_routers.py: test VR general functionality

• test_01_AdvancedZoneRouterServices: general tests: create a network, check the services against the services in network offering. if VR is running, ensure network is implemented and has at least one VM running,
• test_02_NetworkGarbageCollection: test to ensure network is garbage collected when all instanes are stopped.
• test_03_RouterStartOnVmDeploy: Test router start on first VM deploy
• test_01_RouterStopCreatePF: stop VR, create PF rule, start VR, rule should become active
• test_01_RouterStopCreateLB: stop VR, create PF rule, start VR, rule should become active
• test_01_RouterStopCreateFW:stop VR, create PF rule, start VR, rule should become active

test_security_groups.py: secuirty group tests

• test_01_deployVM_InDefaultSecurityGroup
• test_02_listSecurityGroups
• test_03_accessInDefaultSecurityGroup
• test_01_authorizeIngressRule
• test_01_revokeIngressRule
• test_01_dhcpOnlyRouter
• test_01_deployVMWithUserData
• test_01_delete_security_grp_running_vm
• test_02_delete_security_grp_withoout_running_vm
• test_01_authorizeIngressRule_AfterDeployVM
• test_02_revokeIngressRule_AfterDeployVM
• test_03_stopStartVM_verifyIngressAccess
• test_ingress_rules_specific_IP_set
• test_ingress_rules_specific_IP_set_non_def_sec_group

test_shared_networks.py: shared network tests

• test_sharedNetworkOffering_01
• test_sharedNetworkOffering_02
• test_sharedNetworkOffering_03
• test_createSharedNetwork_All
• test_createSharedNetwork_accountSpecific
• test_createSharedNetwork_domainSpecific
• test_createSharedNetwork_projectSpecific
• test_createSharedNetwork_usedVlan
• test_createSharedNetwork_usedVlan2
• test_deployVM_multipleSharedNetwork
• test_deployVM_isolatedAndShared
• test_networkWithsubdomainaccessTrue
• test_networkWithsubdomainaccessFalse
• test_escalation_ES1621
• test_restart_network
• test_reboot_router
• test_stop_start_router
• test_acquire_ip
• test_guest_traffic_port_groups_shared_network

test_vpc.py: tests for VPC functionality

• test_01_list_vpc_apis
• test_02_restart_vpc_no_networks
• test_03_restart_vpc_with_networks
• test_04_delete_vpc_no_networks
• test_05_delete_vpc_with_networks
• test_06_list_vpc_apis_admin
• test_07_restart_network_vm_running
• test_08_delete_vpc
• test_09_vpc_create
• test_10_nonoverlaping_cidrs
• test_11_deploy_vm_wo_network_netdomain
• test_12_deploy_vm_with_netdomain
• test_13_deploy_vm_with_vpc_netdomain
• test_14_deploy_vm_1
• test_15_deploy_vm_2
• test_16_deploy_vm_for_user_by_admin
• test_17_deploy_vm_for_user_by_domain_admin
• test_18_create_net_for_user_diff_domain_by_doadmin
• test_19_create_vpc_wo_params
• test_20_update_vpc_name_display_text
• test_21_deploy_vm_with_gateway_ip
• test_22_vpn_customer_gw_with_hostname

test_vpc_network.py: tests for VPC network functionality

• test_01_create_network
• test_02_create_network_fail
• test_04_create_multiple_networks_with_lb
• test_05_create_network_ext_LB
• test_06_create_network_with_rvr
• test_07_create_network_unsupported_services
• test_08_create_network_without_sourceNAT
• test_09_create_network_shared_nwoff
• test_10_create_network_with_conserve_mode
• test_01_create_network_outside_range
• test_02_create_network_outside_range
• test_03_create_network_inside_range
• test_04_create_network_overlapping_range
• test_05_create_network_diff_account
• test_01_network_services_upgrade
• test_02_network_vpcvr2vr_upgrade
• test_01_wait_network_gc
• test_02_start_vm_network_gc
• test_03_restart_vpcvr
• test_stop_start_vpc_router

 test_vpc_network_lbrules.py: tests for VPC network functionality - Load Balancing Rules

• test_01_VPC_LBRulesListing
• test_02_VPC_CreateLBRuleInMultipleNetworks
• test_03_VPC_CreateLBRuleInMultipleNetworksVRStoppedState
• test_04_VPC_CreateLBRuleInMultipleNetworksVRStoppedState
• test_05_VPC_CreateAndDeleteLBRule
• test_06_VPC_CreateAndDeleteLBRuleVRStopppedState
• test_07_VPC_CreateAndDeleteAllLBRule
• test_08_VPC_CreateAndDeleteAllLBRuleVRStoppedState
• test_09_VPC_LBRuleCreateFailMultipleVPC
• test_10_VPC_FailedToCreateLBRuleNonVPCNetwork
• test_11_VPC_LBRuleCreateNotAllowed
• test_12_VPC_LBRuleCreateFailForRouterIP
• test_13_VPC_LBRuleCreateFailForPFSourceNATIP
• test_14_VPC_LBRuleCreateFailForStaticNatRule
• test_15_VPC_ReleaseIPForLBRuleCreated

test_vpc_network_pfrules.py tests for VPC network functionality - Port Forwarding Rules.

• test_01_network_services_VPC_StopCreatePF
• test_02_network_services_VPC_CreatePF
• test_03_network_services_VPC_StopCreateMultiplePF
• test_04_network_services_VPC_CreateMultiplePF
• test_05_network_services_VPC_StopDeletePF
• test_06_network_services_VPC_DeletePF
• test_07_network_services_VPC_StopDeleteAllPF
• test_08_network_services_VPC_DeleteAllPF
• test_09_network_services_VPC_StopDeleteAllMultiplePF
• test_10_network_services_VPC_DeleteAllMultiplePF

test_vpc_network_staticnatrule.py: tests for VPC network functionality - static nat Rules.

• test_01_VPC_StaticNatRuleCreateStoppedState
• test_02_VPC_CreateStaticNatRule
• test_03_VPC_StopCreateMultipleStaticNatRuleStopppedState
• test_04_VPC_CreateMultipleStaticNatRule
• test_05_network_services_VPC_DeleteAllPF
• test_06_network_services_VPC_DeleteAllMultiplePF

test_vpc_offerings.py: test for VPC offerings

• test_01_create_vpc_offering
• test_02_deploy_vms_in_vpc_nw
• test_03_vpc_off_without_lb
• test_04_vpc_off_without_static_nat
• test_05_vpc_off_without_pf
• test_06_vpc_off_invalid_services
• test_07_update_vpc_off
• test_08_list_vpc_off
• test_09_create_redundant_vpc_offering

test_vpc_routers.py: tests for VPC - Router Operations

• test_01_stop_start_router_after_creating_vpc
• test_02_reboot_router_after_creating_vpc
• test_03_migrate_router_after_creating_vpc
• test_04_change_service_offerring_vpc
• test_05_destroy_router_after_creating_vpc
• test_01_start_stop_router_after_addition_of_one_guest_network
• test_02_reboot_router_after_addition_of_one_guest_network
• test_03_migrate_router_after_addition_of_one_guest_network
• test_04_chg_srv_off_router_after_addition_of_one_guest_network
• test_05_destroy_router_after_addition_of_one_guest_network

test_vpc_vm_life_cycle.py: test for operation on VM in VPC network

• test_01_deploy_instance_in_network
• test_02_stop_instance_in_network
• test_03_start_instance_in_network
• test_04_reboot_instance_in_network
• test_05_destroy_instance_in_network
• test_07_migrate_instance_in_network
• test_08_user_data
• test_09_meta_data
• test_10_expunge_instance_in_network
• test_01_deploy_instance_in_network
• test_02_stop_instance_in_network
• test_03_start_instance_in_network
• test_04_reboot_instance_in_network
• test_05_destroy_instance_in_network
• test_06_recover_instance_in_network
• test_07_migrate_instance_in_network
• test_08_user_data
• test_09_meta_data
• test_10_expunge_instance_in_network
• test_01_deploy_vm_two_isolated_nw
• test_02_deploy_vm_vpcvr_stopped
• test_guest_traffic_port_groups_vpc_network
• test_01_deploy_instance_in_network
• test_02_stop_instance_in_network
• test_03_start_instance_in_network
• test_04_reboot_instance_in_network
• test_05_destroy_instance_in_network
• test_01_deploy_instance_in_network
• test_02_stop_instance_in_network
• test_03_start_instance_in_network
• test_04_reboot_instance_in_network
• test_05_destroy_instance_in_network
• test_06_migrate_instance_in_network
• test_07_user_data
• test_08_meta_data
• test_09_expunge_instance_in_network

test_vpc_vms_deployment.py: tests VM deployment in VPC network functionality

• test_01_deploy_vms_in_network
• test_02_deploy_vms_delete_network
• test_03_deploy_vms_delete_add_network
• test_04_deploy_vms_delete_add_network_noLb
• test_05_create_network_max_limit
• test_06_delete_network_vm_running
• test_07_delete_network_with_rules
• test_08_ip_reallocation_CS5986

test_vpn_service.py: tests for VPN service

• test_01_VPN_service

test_vpn_users.py:tests for VPN users

• test_01_VPN_user_limit
• test_02_use_vpn_port
• test_03_enable_vpn_use_port
• test_04_add_new_users
• test_05_add_duplicate_user
• test_06_add_VPN_user_global_admin
• test_07_add_VPN_user_domain_admin

Smoke Tests

Smoke:test_vpc_vpn.py

• test_01_vpc_remote_access_vpn

• test_01_vpc_site2site_vpn

• test_01_redundant_vpc_site2site_vpn

Smoke:test_vpc_vpn.py

• test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL

• test_02_redundant_VPC_default_routes

• test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers

• test_04_rvpc_network_garbage_collector_nics

• Test_05_rvpc_multi_tiers

Smoke:test_vpc_vpn.py

• test_01_VPC_nics_after_destroy

• test_02_VPC_default_routes

Smoke: test_routers_network_ops.py

• test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true

• test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false

• test_03_RVR_Network_check_router_state

• test_01_isolate_network_FW_PF_default_routes_egress_true

• test_02_isolate_network_FW_PF_default_routes_egress_false

Smoke:test_routers_iptables_default_policy

• test_01_single_VPC_iptables_policies

• test_02_routervm_iptables_policies

Smoke:test_routers.py

• test_01_router_internal_basic

• test_02_router_internal_adv

• test_03_restart_network_cleanup

• test_04_restart_network_wo_cleanup

• test_05_router_basic

• test_06_router_advanced

• test_07_stop_router

• test_08_start_router

• test_09_reboot_router

Smoke:test_router_dhcphosts.py

• test_router_dhcphosts

Smoke: test_guest_vlan_range.py

• test_dedicateGuestVlanRange

Smoke: test_internal_lb.py

• test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80

• test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80

• test_03_vpc_internallb_haproxy_stats_on_all_interfaces

• Test_04_rvpc_internallb_haproxy_stats_on_all_interfaces

Smoke: test_loadbalance.py

• test_01_create_lb_rule_src_nat

• test_02_create_lb_rule_non_nat

• test_assign_and_removal_lb

• 
  

Smoke: test_multipleips_per_nic.py

• test_nic_secondaryip_add_remove

Smoke: test_network.py

• test_public_ip_admin_account

• test_public_ip_user_account

• test_01_port_fwd_on_src_nat

• test_02_port_fwd_on_non_src_nat

• test_guest_traffic_port_groups_isolated_network

• test_reboot_router

• test_releaseIP

• test_delete_account

• Test_network_rules_acquired_public_ip

Smoke: test_networkacl.py

• Test_network_acl

Smoke: test_nic.py

• test_01_nic

Smoke: test_non_contiguous_vlans.py

• test_extendPhysicalNetworkVlan

Smoke: test_password_server.py

• Test_isolate_network_password_server

Smoke: test_portable_publicip.py

Smoke: test_privategw_acl.py

Smoke: test_public_ip_range.py

• test_dedicatePublicIpRange

High-level Automation Analysis

General Observations

• Hard to find tests specific to type/scenario/use-case easily. All files in one place. Other than name of the file difficult to find the tests
  • Some more test meta data/tags on the tests?

Gaps

• Did not find tests for DNS. Rest of the VR services has some basic tests (even for meta data, password service etc).
  
• No Service capability specific tests (for eg per account NAT, LB algorithms/stickiness etc)
• Not all protocol/traffic types/all parameters have tests.
  
• No multiple public interface on the VR tests.
• Most tests verify the CloudStack VR as black box. No extensive verification on VR on expected configuration for Iptables, dnsmasq, haproxy etc
• no VR upgrade tests
• not all operation on network rule have tests. For e.g, tests veifying functionality after clean up of a (firewll/pf/static nat) rule

test_shared_networks.py

test_routers.py in components 

test_routers.py in smoke

test_security_groups.py