This is summary of gap analysis done on Marvin tests that touch the virtual router functionality.

Networking in CloudStack is feature rich and flexible. Networks can be composed with flexible set of services. Network can be deployed in many different models. Here is the listing of some of the dimensions in which networks and functionality can be deployed/used.

Network (only VR) supported services

• DHCP
• DNS
• Gateway
• Firewall
• LB
• VPN
• SourceNAT
• StaticNAT
• PortForwarding
• Network ACL
• UserData
• MetaData
• Password

VR service capabilities

• LB
  • LB algorithms
    • round robin
    • least connection
    • source
  • LB isolation
    • dedicated
  • HA proxy stickiness
  • protocols
• Firewall
  • traffic direction (ingress, egress)
  • supported ingress/egress protocols
  • multiple IP's
• VPN
  • supported protocols (l2tp, pptp, ipsec)
  • vpn type: remote access vpn
• DNS
  • suffix modification
• DHCP
  • multiple subnets
• Source NAT
  • per account
  • per network

Network offering

• pick and choose set of services and capabilities

Networking isolated Type:

• isolated
• shared network with just (DNS, DHCP, user data)
• shared network with full services
• shared network with security groups

Zone type:

• Advanced Zone
• Basic Zone

Network model

• VPC
• non VPC

Network life cycle

• create
• delete
• restart
• update
• restart
• update

Router life cycle

• start router
• stop router
• reboot router
• destroy router
• change service offering for router

Network Garbage collection

• network is shutdown in GC, and start again on VM deployment

Redundant routers

• master/slave fail overs

VR split network service other network elements

Upgrade

Component tests

test_acl_isolatednetwork.py: test access control on network

• test_01_createNetwork_admin
• test_02_createNetwork_admin_foruserinsamedomain
• test_03_createNetwork_admin_foruserinotherdomain
• test_04_createNetwork_domaindmin
• test_05_createNetwork_domaindmin_foruserinsamedomain
• test_06_createNetwork_domaindmin_foruserinsubdomain
• test_07_createNetwork_domaindmin_forcrossdomainuser
• test_08_createNetwork_user
• test_09_createNetwork_user_foruserinsamedomain
• test_10_createNetwork_user_foruserinotherdomain
• test_11_deployvm_admin
• test_12_deployvm_admin_foruserinsamedomain
• test_13_deployvm_admin_foruserinotherdomain
• test_13_1_deployvm_admin_foruserinotherdomain_crossnetwork
• test_14_deployvm_domaindmin
• test_15_deployvm_domaindmin_foruserinsamedomain
• test_16_deployvm_domaindmin_foruserinsubdomain
• test_17_deployvm_domaindmin_forcrossdomainuser
• test_17_1_deployvm_domainadmin_foruserinotherdomain_crossnetwork
• test_18_deployvm_user
• test_19_deployvm_user_foruserinsamedomain
• test_20_deployvm_user_foruserincrossdomain
• test_20_1_deployvm_user_incrossnetwork
• test_21_restartNetwork_admin
• test_22_restartNetwork_admin_foruserinsamedomain
• test_23_restartNetwork_admin_foruserinotherdomain
• test_24_restartNetwork_domaindmin
• test_25_restartNetwork_domaindmin_foruserinsamedomain
• test_26_restartNetwork_domaindmin_foruserinsubdomain
• test_27_restartNetwork_domaindmin_forcrossdomainuser
• test_28_restartNetwork_user
• test_29_restartNetwork_user_foruserinsamedomain
• test_30_restartNetwork_user_foruserinotherdomain

test_acl_isolatednetwork_delete.py: test access control on delete network

• test_deleteNetwork_admin
• test_deleteNetwork_admin_foruserinsamedomain
• test_deleteNetwork_admin_foruserinotherdomain
• test_deleteNetwork_domaindmin
• test_deleteNetwork_domaindmin_foruserinsamedomain
• test_deleteNetwork_domaindmin_foruserinsubdomain
• test_deleteNetwork_domaindmin_forcrossdomainuser
• test_deleteNetwork_user
• test_deleteNetwork_user_foruserinsamedomain
• test_deleteNetwork_user_foruserinotherdomain

test_acl_sharednetwork.py: Test cases for access control for Vm deployment in Shared Network with scope all,account and Domain

• test_deployVM_in_sharedNetwork_scope_all_domainuser
• test_deployVM_in_sharedNetwork_scope_all_domainadminuser
• test_deployVM_in_sharedNetwork_scope_all_subdomainuser
• test_deployVM_in_sharedNetwork_scope_all_subdomainadminuser
• test_deployVM_in_sharedNetwork_scope_all_ROOTuser
• test_deployVM_in_sharedNetwork_scope_all_ROOTadmin
• test_deployVM_in_sharedNetwork_scope_domain_nosubdomainaccess_domainuser
• test_deployVM_in_sharedNetwork_scope_domain_nosubdomainaccess_domainadminuser
• test_deployVM_in_sharedNetwork_scope_domain_nosubdomainaccess_subdomainuser
• test_deployVM_in_sharedNetwork_scope_domain_nosubdomainaccess_subdomainadminuser
• test_deployVM_in_sharedNetwork_scope_domain_nosubdomainaccess_parentdomainuser
• test_deployVM_in_sharedNetwork_scope_domain_nosubdomainaccess_parentdomainadminuser
• test_deployVM_in_sharedNetwork_scope_domain_nosubdomainaccess_ROOTuser
• test_deployVM_in_sharedNetwork_scope_domain_nosubdomainaccess_ROOTadmin
• test_deployVM_in_sharedNetwork_scope_domain_withsubdomainaccess_domainuser
• test_deployVM_in_sharedNetwork_scope_domain_withsubdomainaccess_domainadminuser
• test_deployVM_in_sharedNetwork_scope_domain_withsubdomainaccess_subdomainuser
• test_deployVM_in_sharedNetwork_scope_domain_withsubdomainaccess_subdomainadminuser
• test_deployVM_in_sharedNetwork_scope_domain_withsubdomainaccess_parentdomainuser
• test_deployVM_in_sharedNetwork_scope_domain_withsubdomainaccess_parentdomainadminuser
• test_deployVM_in_sharedNetwork_scope_domain_withsubdomainaccess_ROOTuser
• test_deployVM_in_sharedNetwork_scope_domain_withsubdomainaccess_ROOTadmin
• test_deployVM_in_sharedNetwork_scope_account_domainuser
• test_deployVM_in_sharedNetwork_scope_account_domainadminuser
• test_deployVM_in_sharedNetwork_scope_account_user
• test_deployVM_in_sharedNetwork_scope_account_differentdomain
• test_deployVM_in_sharedNetwork_scope_account_ROOTuser
• test_deployVM_in_sharedNetwork_scope_account_ROOTadmin

test_acl_sharednetwork_deployVM-impersonation.py: Test cases for deploying Virtual Machine using impersonation (passing account and domainId parameters) for shared network

• test_deployVM_in_sharedNetwork_as_admin_scope_all_domainuser
• test_deployVM_in_sharedNetwork_as_admin_scope_all_domainadminuser
• test_deployVM_in_sharedNetwork_as_admin_scope_all_subdomainuser
• test_deployVM_in_sharedNetwork_as_admin_scope_all_subdomainadminuser
• test_deployVM_in_sharedNetwork_as_admin_scope_all_ROOTuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_nosubdomainaccess_domainuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_nosubdomainaccess_domainadminuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_nosubdomainaccess_subdomainuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_nosubdomainaccess_subdomainadminuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_nosubdomainaccess_parentdomainuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_nosubdomainaccess_parentdomainadminuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_nosubdomainaccess_ROOTuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_withsubdomainaccess_domainuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_withsubdomainaccess_domainadminuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_withsubdomainaccess_subdomainuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_withsubdomainaccess_subdomainadminuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_withsubdomainaccess_parentdomainuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_withsubdomainaccess_parentdomainadminuser
• test_deployVM_in_sharedNetwork_as_admin_scope_domain_withsubdomainaccess_ROOTuser
• test_deployVM_in_sharedNetwork_as_admin_scope_account_domainuser
• test_deployVM_in_sharedNetwork_as_admin_scope_account_domainadminuser
• test_deployVM_in_sharedNetwork_as_admin_scope_account_user
• test_deployVM_in_sharedNetwork_as_admin_scope_account_differentdomain
• test_deployVM_in_sharedNetwork_as_admin_scope_account_ROOTuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_all_domainuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_all_domainadminuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_all_subdomainuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_all_subdomainadminuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_all_ROOTuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_all_crossdomainuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_nosubdomainaccess_domainuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_nosubdomainaccess_domainadminuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_nosubdomainaccess_subdomainuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_nosubdomainaccess_subdomainadminuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_nosubdomainaccess_parentdomainuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_nosubdomainaccess_parentdomainadminuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_nosubdomainaccess_ROOTuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_withsubdomainaccess_domainuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_withsubdomainaccess_domainadminuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_withsubdomainaccess_subdomainuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_withsubdomainaccess_subdomainadminuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_withsubdomainaccess_parentdomainuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_withsubdomainaccess_parentdomainadminuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_domain_withsubdomainaccess_ROOTuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_account_domainuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_account_domainadminuser
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_account_user
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_account_differentdomain
• test_deployVM_in_sharedNetwork_as_domainadmin_scope_account_ROOTuser
• test_deployVM_in_sharedNetwork_as_regularuser_scope_all_anotherusersamedomain
• test_deployVM_in_sharedNetwork_as_regularuser_scope_all_crossdomain

test_add_remove_network.py: tests for Add Remove Network to VM

• test_01_add_nw_running_vm
• test_02_add_nw_stopped_vm
• test_03_add_nw_multiple_times
• test_04_vpc_nw_running_vm
• test_05_add_vpc_nw_stopped_vm
• test_06_add_nw_ipaddress_running_vm
• test_10_add_nw_invalid_ipaddress_running_vm
• test_14_add_nw_different_account
• test_24_add_nw_different_domain
• test_25_add_nw_above_account_limit
• test_07_remove_nic_running_vm
• test_08_remove_default_nic
• test_09_remove_foreign_nic
• test_29_remove_nic_CS22503
• test_30_remove_nic_reattach
• test_vm = VirtualMachine.create
• test_vm.stop
• test_vm.remove_nic
• test_vm.add_nic
• test_vm.start
• test_11_update_nic_running_vm
• test_12_make_default_nic_as_default
• test_13_set_foreign_nic_as_default
• test_15_add_nic_wrong_vm_id
• test_16_add_nic_wrong_network_id
• test_17_add_nic_different_zone
• test_18_add_nic_basic_zone
• test_26_add_nic_insufficient_permission
• test_19_remove_nic_wrong_vm_id
• test_20_remove_nic_wrong_nic_id
• test_27_remove_nic_insufficient_permission
• test_21_update_nic_wrong_vm_id
• test_22_update_nic_wrong_nic_id
• test_23_update_nic_incorrect_vm_state
• test_28_update_nic_insufficient_permission

test_advancedsg_networks.py: tests for networks in advanced zone with security groups

• test_01_createZone_secGrp_enabled
• test_02_createZone_secGrp_disabled
• test_03_createIsolatedNetwork
• test_04_createSharedNetwork_withoutSG
• test_05_deployVM_SharedwithSG
• test_06_SharedNwSGAccountSpecific
• test_07_SharedNwSG_DomainWide_SubdomainAcccess
• test_08_SharedNwSGAccountSpecific_samevlan_samesubnet
• test_09_SharedNwDomainWide_samevlan_samesubnet
• test_10_deleteSharedNwSGAccountSpecific_InUse
• test_11_deleteSharedNwSG_DomainWide_InUse
• test_29_deleteSharedNwSG_ZoneWide_InUse
• test_12_deleteSharedNwSGAccountSpecific_NotInUse
• test_13_deleteSharedNwSG_DomainWide_NotInUse
• test_30_deleteSharedNwSG_ZoneWide_NotInUse
• test__14_createSharedNwWithSG_withoutParams
• test__15_createVPC
• test__16_AccountSpecificNwAccess
• test__17_DomainSpecificNwAccess
• test_18_DeployVM_NoFreeIPs
• test_19_DeployVM_DefaultSG
• test_20_DeployVM_SecGrp_sharedNetwork
• test_24_DeployVM_Multiple_Shared_Networks
• test_25_Deploy_Multiple_VM_Different_Shared_Networks_Same_SG
• test_26_Destroy_Deploy_VM_NoFreeIPs
• test_27_start_stop_vm
• test_28_destroy_recover_expunge_vm
• test_31_Deploy_VM_multiple_shared_networks_sg
• test_33_VM_Migrate_SharedNwSG
• test_34_DeployVM_in_SecondSGNetwork
• test_21_DeployVM_WithoutSG
• test_22_DeployVM_With_Custom_SG
• test_23_DeployVM_MultipleSG
• test_32_delete_default_security_group
• test_34_restart_shared_network_sg
• test_35_Enable_Host_Maintenance
• test_36_ssh_vm_other_sg
• test_37_ping_vm_other_sg
• test_38_ssh_vm_other_sg_new_vm
• test_39_ssh_vm_other_sg_from_multiple_sg_vm
• test_40_ssh_vm_other_sg_reboot

test_egress_fw_rules.py: Egress Firewall component tests

• test_01_egress_fr1: Test By-default the communication from guest n/w to public n/w is allowed.
• test_01_1_egress_fr1: Test By-default the communication from guest n/w to public n/w is NOT allowed
• test_02_egress_fr2: Test Allow Communication using Egress rule with CIDR + Port Range + Protocol
• test_02_1_egress_fr2: Test Allow Communication using Egress rule with CIDR + Port Range + Protocol.
• test_03_egress_fr3: Test communication allowed on network  with network offering with egress policy is allow
• test_03_1_egress_fr3: Test communication blocked on network with network offering with egress policy is deny
• test_04_egress_fr4: Test Create Egress rule and check the Firewall_Rules DB table
• test_04_1_egress_fr4: Test Create Egress rule and check the Firewall_Rules DB table
• test_05_egress_fr5: Test Create Egress rule and check the IP tables on network with network offering egress policy is allow
• test_05_1_egress_fr5: Test Create Egress rule and check the IP tables on network with network offering egress policy is deny
• test_06_egress_fr6: Test Create Egress rule without CIDR on network with network offering egress policy is allow
• test_06_1_egress_fr6: Test Create Egress rule without CIDR on network with network offering egress policy is deny
• test_07_egress_fr7: Test Create Egress rule without End Port
• test_07_1_egress_fr7: Test Create Egress rule without End Port
• test_08_egress_fr8: Test Port Forwarding and Egress Conflict
• test_08_1_egress_fr8: Test Port Forwarding and Egress Conflict
• test_09_egress_fr9: Test Delete Egress rule
• test_09_1_egress_fr9: Test Delete Egress rule
• test_10_egress_fr10: Test Invalid CIDR and Invalid Port ranges
• test_10_1_egress_fr10: Test Invalid CIDR and Invalid Port ranges
• test_11_egress_fr11: Test Regression on Firewall + PF + LB + SNAT
• test_11_1_egress_fr11:Test Regression on Firewall + PF + LB + SNAT
• test_12_egress_fr12: Test Reboot Router
• test_12_1_egress_fr12: Test Reboot Router
• test_13_egress_fr13: Test Redundant Router : Master failover
• test_13_1_egress_fr13: Test Redundant Router : Master failover

 test_haproxy.py: HAProxy tests

• test_01_create_sticky_policy_default_values
• test_02_create_sticky_policy_custom_values
• test_03_supported_policies_by_network
• test_04_delete_lb_rule
• test_05_error_alerts_after_create
• test_06_release_ip
• test_07_delete_account
• test_08_create_policy_router_stopped
• test_09_create_policy_router_destroy
• test_10_create_policy_enable_disable_vpn
• test_11_invalid_params

test_ip_reservation.py: test for guest IP reservations

• test_vm_create_after_reservation
• test_vm_create_outside_cidr_after_reservation
• test_update_cidr_multiple_vms_not_all_inclusive
• test_update_cidr_single_vm_not_inclusive
• test_nat_rules
• test_RVR_network
• test_ip_reservation_in_multiple_networks_same_account
• test_restart_network_with_cleanup
• test_update_network_guestvmcidr
• test_reservation_after_router_restart
• test_destroy_recreate_router
• test_network_not_implemented
• test_vm_create_after_reservation
• test_reservation_after_router_restart
• test_vm_create_outside_cidr_after_reservation

 test_lb_secondary_ip.py:  tests for Load balancing rules for secondary IP

• test_01_lb_rule_for_primary_ip
• test_02_lb_rule_for_primary_ip
• test_03_lb_rule_for_primary_and_secondary_ip
• test_04_lb_rule_primary_secondary_multiple_vms
• test_05_lb_rule_wrong_vm_id
• test_06_lb_rule_wrong_vm_ip
• test_07_lb_rule_used_ip
• test_08_lb_rule_remove_used_ip
• test_09_lbvmips_true
• test_10_lbvmips_false
• test_11_ssh_to_secondary_ip
• test_12_ssh_to_primary_secondary_ip
• test_13_delete_lb_rule
• test_14_remove_lb_rule_secondary_ip
• test_15_remove_lb_rule_primary_ip
• test_16_delete_vm_from_lb_rule
• test_20_destroy_recover_vm
• test_21_modify_lb_rule_algorithm
• test_17_restart_router
• test_18_restart_network_cleanup_true
• test_19_restart_network_cleanup_false
• test_22_network_gc
• test_23_lb_rule_functioning_with_netscaler

test_multiple_ip_ranges.py: test related to multiple public IP ranges

• test_01_add_ip_same_cidr
• test_02_add_ip_diff_cidr
• test_03_del_ip_range
• test_04_add_noncontiguous_ip_range
• test_05_add_overlapped_ip_range
• test_06_add_ip_range_overlapped_with_two_ranges
• test_07_add_iprange_superset
• test_08_add_iprange_subset

test_multiple_ips_per_nic.py: Multiple IPs per NIC feature

• test_add_ip_to_nic
• test_remove_ip_from_nic
• test_remove_invalid_ip
• test_list_nics
• test_operations_non_root_admin_api_client
• test_add_PF_rule
• test_delete_PF_nat_rule
• test_disassociate_ip_mapped_to_secondary_ip_through_PF_rule
• test_add_static_nat_rule
• test_disable_static_nat
• test_delete_vm
• test_recover_vm
• test_network_restart_cleanup_true
• test_network_restart_cleanup_false
• test_reboot_router_VM

test_network_offering.py

• test_01_network_off_without_conserve_mode
• test_02_network_off_with_conserve_mode
• test_03_network_off_CS5332
• test_04_network_without_domain_CS19303
• test_01_nwupgrade_netscaler_conserve_on
• test_02_nwupgrade_netscaler_conserve_off
• test_create_network_with_snat

test_redundent_router (test_redundant_router_cleanups.py, test_redundant_router_services.py, test_redundant_router_upgrades.py)

• test_restart_ntwk_no_cleanup
• test_restart_ntwk_with_cleanup
• test_network_gc
• test_restart_network_with_destroyed_masterVR
• test_enableVPNOverRvR
• test_upgradeVR_to_redundantVR
• test_downgradeRvR_to_VR

test_routers.py: test VR general functionality

• test_01_AdvancedZoneRouterServices: general tests: create a network, check the services against the services in network offering. if VR is running, ensure network is implemented and has at least one VM running,
• test_02_NetworkGarbageCollection: test to ensure network is garbage collected when all instanes are stopped.
• test_03_RouterStartOnVmDeploy: Test router start on first VM deploy
• test_01_RouterStopCreatePF: stop VR, create PF rule, start VR, rule should become active
• test_01_RouterStopCreateLB: stop VR, create PF rule, start VR, rule should become active
• test_01_RouterStopCreateFW:stop VR, create PF rule, start VR, rule should become active

test_security_groups.py: secuirty group tests

• test_01_deployVM_InDefaultSecurityGroup
• test_02_listSecurityGroups
• test_03_accessInDefaultSecurityGroup
• test_01_authorizeIngressRule
• test_01_revokeIngressRule
• test_01_dhcpOnlyRouter
• test_01_deployVMWithUserData
• test_01_delete_security_grp_running_vm
• test_02_delete_security_grp_withoout_running_vm
• test_01_authorizeIngressRule_AfterDeployVM
• test_02_revokeIngressRule_AfterDeployVM
• test_03_stopStartVM_verifyIngressAccess
• test_ingress_rules_specific_IP_set
• test_ingress_rules_specific_IP_set_non_def_sec_group

test_shared_networks.py: shared network tests

• test_sharedNetworkOffering_01
• test_sharedNetworkOffering_02
• test_sharedNetworkOffering_03
• test_createSharedNetwork_All
• test_createSharedNetwork_accountSpecific
• test_createSharedNetwork_domainSpecific
• test_createSharedNetwork_projectSpecific
• test_createSharedNetwork_usedVlan
• test_createSharedNetwork_usedVlan2
• test_deployVM_multipleSharedNetwork
• test_deployVM_isolatedAndShared
• test_networkWithsubdomainaccessTrue
• test_networkWithsubdomainaccessFalse
• test_escalation_ES1621
• test_restart_network
• test_reboot_router
• test_stop_start_router
• test_acquire_ip
• test_guest_traffic_port_groups_shared_network

test_vpc.py: tests for VPC functionality

• test_01_list_vpc_apis
• test_02_restart_vpc_no_networks
• test_03_restart_vpc_with_networks
• test_04_delete_vpc_no_networks
• test_05_delete_vpc_with_networks
• test_06_list_vpc_apis_admin
• test_07_restart_network_vm_running
• test_08_delete_vpc
• test_09_vpc_create
• test_10_nonoverlaping_cidrs
• test_11_deploy_vm_wo_network_netdomain
• test_12_deploy_vm_with_netdomain
• test_13_deploy_vm_with_vpc_netdomain
• test_14_deploy_vm_1
• test_15_deploy_vm_2
• test_16_deploy_vm_for_user_by_admin
• test_17_deploy_vm_for_user_by_domain_admin
• test_18_create_net_for_user_diff_domain_by_doadmin
• test_19_create_vpc_wo_params
• test_20_update_vpc_name_display_text
• test_21_deploy_vm_with_gateway_ip
• test_22_vpn_customer_gw_with_hostname

test_vpc_network.py: tests for VPC network functionality

• test_01_create_network
• test_02_create_network_fail
• test_04_create_multiple_networks_with_lb
• test_05_create_network_ext_LB
• test_06_create_network_with_rvr
• test_07_create_network_unsupported_services
• test_08_create_network_without_sourceNAT
• test_09_create_network_shared_nwoff
• test_10_create_network_with_conserve_mode
• test_01_create_network_outside_range
• test_02_create_network_outside_range
• test_03_create_network_inside_range
• test_04_create_network_overlapping_range
• test_05_create_network_diff_account
• test_01_network_services_upgrade
• test_02_network_vpcvr2vr_upgrade
• test_01_wait_network_gc
• test_02_start_vm_network_gc
• test_03_restart_vpcvr
• test_stop_start_vpc_router

 test_vpc_network_lbrules.py: tests for VPC network functionality - Load Balancing Rules

• test_01_VPC_LBRulesListing
• test_02_VPC_CreateLBRuleInMultipleNetworks
• test_03_VPC_CreateLBRuleInMultipleNetworksVRStoppedState
• test_04_VPC_CreateLBRuleInMultipleNetworksVRStoppedState
• test_05_VPC_CreateAndDeleteLBRule
• test_06_VPC_CreateAndDeleteLBRuleVRStopppedState
• test_07_VPC_CreateAndDeleteAllLBRule
• test_08_VPC_CreateAndDeleteAllLBRuleVRStoppedState
• test_09_VPC_LBRuleCreateFailMultipleVPC
• test_10_VPC_FailedToCreateLBRuleNonVPCNetwork
• test_11_VPC_LBRuleCreateNotAllowed
• test_12_VPC_LBRuleCreateFailForRouterIP
• test_13_VPC_LBRuleCreateFailForPFSourceNATIP
• test_14_VPC_LBRuleCreateFailForStaticNatRule
• test_15_VPC_ReleaseIPForLBRuleCreated

test_vpc_network_pfrules.py tests for VPC network functionality - Port Forwarding Rules.

• test_01_network_services_VPC_StopCreatePF
• test_02_network_services_VPC_CreatePF
• test_03_network_services_VPC_StopCreateMultiplePF
• test_04_network_services_VPC_CreateMultiplePF
• test_05_network_services_VPC_StopDeletePF
• test_06_network_services_VPC_DeletePF
• test_07_network_services_VPC_StopDeleteAllPF
• test_08_network_services_VPC_DeleteAllPF
• test_09_network_services_VPC_StopDeleteAllMultiplePF
• test_10_network_services_VPC_DeleteAllMultiplePF

test_vpc_network_staticnatrule.py: tests for VPC network functionality - static nat Rules.

• test_01_VPC_StaticNatRuleCreateStoppedState
• test_02_VPC_CreateStaticNatRule
• test_03_VPC_StopCreateMultipleStaticNatRuleStopppedState
• test_04_VPC_CreateMultipleStaticNatRule
• test_05_network_services_VPC_DeleteAllPF
• test_06_network_services_VPC_DeleteAllMultiplePF

test_vpc_offerings.py: test for VPC offerings

• test_01_create_vpc_offering
• test_02_deploy_vms_in_vpc_nw
• test_03_vpc_off_without_lb
• test_04_vpc_off_without_static_nat
• test_05_vpc_off_without_pf
• test_06_vpc_off_invalid_services
• test_07_update_vpc_off
• test_08_list_vpc_off
• test_09_create_redundant_vpc_offering

test_vpc_routers.py: tests for VPC - Router Operations

• test_01_stop_start_router_after_creating_vpc
• test_02_reboot_router_after_creating_vpc
• test_03_migrate_router_after_creating_vpc
• test_04_change_service_offerring_vpc
• test_05_destroy_router_after_creating_vpc
• test_01_start_stop_router_after_addition_of_one_guest_network
• test_02_reboot_router_after_addition_of_one_guest_network
• test_03_migrate_router_after_addition_of_one_guest_network
• test_04_chg_srv_off_router_after_addition_of_one_guest_network
• test_05_destroy_router_after_addition_of_one_guest_network

test_vpc_vm_life_cycle.py: test for operation on VM in VPC network

• test_01_deploy_instance_in_network
• test_02_stop_instance_in_network
• test_03_start_instance_in_network
• test_04_reboot_instance_in_network
• test_05_destroy_instance_in_network
• test_07_migrate_instance_in_network
• test_08_user_data
• test_09_meta_data
• test_10_expunge_instance_in_network
• test_01_deploy_instance_in_network
• test_02_stop_instance_in_network
• test_03_start_instance_in_network
• test_04_reboot_instance_in_network
• test_05_destroy_instance_in_network
• test_06_recover_instance_in_network
• test_07_migrate_instance_in_network
• test_08_user_data
• test_09_meta_data
• test_10_expunge_instance_in_network
• test_01_deploy_vm_two_isolated_nw
• test_02_deploy_vm_vpcvr_stopped
• test_guest_traffic_port_groups_vpc_network
• test_01_deploy_instance_in_network
• test_02_stop_instance_in_network
• test_03_start_instance_in_network
• test_04_reboot_instance_in_network
• test_05_destroy_instance_in_network
• test_01_deploy_instance_in_network
• test_02_stop_instance_in_network
• test_03_start_instance_in_network
• test_04_reboot_instance_in_network
• test_05_destroy_instance_in_network
• test_06_migrate_instance_in_network
• test_07_user_data
• test_08_meta_data
• test_09_expunge_instance_in_network

test_vpc_vms_deployment.py: tests VM deployment in VPC network functionality

• test_01_deploy_vms_in_network
• test_02_deploy_vms_delete_network
• test_03_deploy_vms_delete_add_network
• test_04_deploy_vms_delete_add_network_noLb
• test_05_create_network_max_limit
• test_06_delete_network_vm_running
• test_07_delete_network_with_rules
• test_08_ip_reallocation_CS5986

test_vpn_service.py: tests for VPN service

• test_01_VPN_service

test_vpn_users.py:tests for VPN users

• test_01_VPN_user_limit
• test_02_use_vpn_port
• test_03_enable_vpn_use_port
• test_04_add_new_users
• test_05_add_duplicate_user
• test_06_add_VPN_user_global_admin
• test_07_add_VPN_user_domain_admin

Smoke Tests

Smoke:test_vpc_vpn.py

• test_01_vpc_remote_access_vpn

• test_01_vpc_site2site_vpn

• test_01_redundant_vpc_site2site_vpn

Smoke:test_vpc_vpn.py

• test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL

• test_02_redundant_VPC_default_routes

• test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers

• test_04_rvpc_network_garbage_collector_nics

• Test_05_rvpc_multi_tiers

Smoke:test_vpc_vpn.py

• test_01_VPC_nics_after_destroy

• test_02_VPC_default_routes

Smoke: test_routers_network_ops.py

• test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true

• test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false

• test_03_RVR_Network_check_router_state

• test_01_isolate_network_FW_PF_default_routes_egress_true

• test_02_isolate_network_FW_PF_default_routes_egress_false

Smoke:test_routers_iptables_default_policy

• test_01_single_VPC_iptables_policies

• test_02_routervm_iptables_policies

Smoke:test_routers.py

• test_01_router_internal_basic

• test_02_router_internal_adv

• test_03_restart_network_cleanup

• test_04_restart_network_wo_cleanup

• test_05_router_basic

• test_06_router_advanced

• test_07_stop_router

• test_08_start_router

• test_09_reboot_router

Smoke:test_router_dhcphosts.py

• test_router_dhcphosts

Smoke: test_guest_vlan_range.py

• test_dedicateGuestVlanRange

Smoke: test_internal_lb.py

• test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80

• test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80

• test_03_vpc_internallb_haproxy_stats_on_all_interfaces

• Test_04_rvpc_internallb_haproxy_stats_on_all_interfaces

Smoke: test_loadbalance.py

• test_01_create_lb_rule_src_nat

• test_02_create_lb_rule_non_nat

• test_assign_and_removal_lb

• 
  

Smoke: test_multipleips_per_nic.py

• test_nic_secondaryip_add_remove

Smoke: test_network.py

• test_public_ip_admin_account

• test_public_ip_user_account

• test_01_port_fwd_on_src_nat

• test_02_port_fwd_on_non_src_nat

• test_guest_traffic_port_groups_isolated_network

• test_reboot_router

• test_releaseIP

• test_delete_account

• Test_network_rules_acquired_public_ip

Smoke: test_networkacl.py

• Test_network_acl

Smoke: test_nic.py

• test_01_nic

Smoke: test_non_contiguous_vlans.py

• test_extendPhysicalNetworkVlan

Smoke: test_password_server.py

• Test_isolate_network_password_server

Smoke: test_portable_publicip.py

Smoke: test_privategw_acl.py

Smoke: test_public_ip_range.py

• test_dedicatePublicIpRange

High-level Automation Analysis

General Observations

• Hard to find tests specific to type/scenario/use-case easily. All files in one place. Other than name of the file difficult to find the tests
  • Some more test meta data/tags on the tests?

Gaps

• Did not find tests for DNS. Rest of the VR services has some basic tests (even for meta data, password service etc).
  
• No Service capability specific tests (for eg per account NAT, LB algorithms/stickiness etc)
• Not all protocol/traffic types/all parameters have tests.
  
• No multiple public interface on the VR tests.
• Most tests verify the CloudStack VR as black box. No extensive verification on VR on expected configuration for Iptables, dnsmasq, haproxy etc
• no VR upgrade tests
• not all operation on network rule have tests. For e.g, tests veifying functionality after clean up of a (firewll/pf/static nat) rule

S. No	Module	Existing tests	Comments	Additional Marvin test required
1	Egress Firewall Rules	test_egress_fw_rules.py	Tests sufficiently covers TCP, ICMP protocols for both deny/allow default network offering behaviors.	CLOUDSTACK-9487: Egress rules marvin tests for UDP
2	Network services	test_network.py test_routers.py test_ha_proxy.py test_load_balancer.py test_security_groups.py test_vpn_service.py test_egress_fw_rules.py test_reset_ssh_keypair.py	Most of the services has test cases except for DNS	CLOUDSTACK-9488: marvin tests for DNS
3	Persistent Networks	test_persistent_networks.py	Sufficiently automated
4	Firewall and LoadBalancer		Sufficiently automated
5	Password Reset Service	test_vm_passwdenabled.py	covers password reset of the VM and VM password functionality
6	Reset SSh Key pair	test_reset_ssh_keypair.py	Sufficiently automated
7	Network Update on  RVR - CIDR , Network Offerings change
8
9	Network Update on Isolated VR's - CIDR , NW Offerings change		Sufficiently automated
10	Redundant Virtual Routers	test_redundant_router_cleanups.py test_redundant_router_services.py test_redundant_router_upgrades.py smoke: test_vpc_redundent.py	Test sufficiently covers RVR fail over scenarios.
11	VPC Sanity cases	test_vpc.py test_vpc_network.py test_vpc_network_pfrules.py test_vpc_offerings.py test_vpc_vm_life_cycle.py test_vpc_distributed_routing_offering.py test_vpc_network_lbrules.py test_vpc_network_staticnatrule.py test_vpc_routers.py -- tests private gateway test_vpc_vms_deployment.py	Sufficiently covered buy automation
12	VPC RVR	test_vpc_redundant.py	Create a redundant VPC with two networks with two VMs in each network Create a redundant VPC with 1 Tier, 1 VM, 1 ACL, 1 PF and test Network GC Nics
13	Inter Vlan Routing		Not sufficiently covered
14	Access Lists	test_network.py
15	Static routes / Private Gateway	test_escalations_networks.py	listing pw gateways,  -- No check for traffic flow
16	Remote access VPN on VPC
17	Site - Site VPN
18	Remote Access VPN	test_escalations_vpncustomergateways.py  test_vpn_service.py	# api test, pagination # Validate if IPSEC is running on the public, IP by using ike-scan
19	VPN Users	test_vpn_users.py	VPN remote access user limit tests
20	VR + external devices side by side	test_asa1000v_fw.py test_bigswitch_bcf.py
21	Service Monitoring tool in VR	NA	Not covered
22	Shared Network	test_shared_networks.py Tests: test_acquire_ip test_createSharedNetwork_accountSpecific test_createSharedNetwork_All test_createSharedNetwork_domainSpecific test_createSharedNetwork_projectSpecific test_createSharedNetwork_usedVlan test_createSharedNetwork_usedVlan2 test_deployVM_isolatedAndShared test_deployVM_multipleSharedNetwork test_escalation_ES1621 test_guest_traffic_port_groups_shared_network test_networkWithsubdomainaccessFalse test_networkWithsubdomainaccessTrue test_reboot_router test_restart_network test_sharedNetworkOffering_01 test_sharedNetworkOffering_02 test_sharedNetworkOffering_03 test_stop_start_router	Sufficiently covered buy automation
23	Advance Zone using Pvlan	NA	Not covered
24	Manual VR Operations - Stop , start , Destroy ( DomR Testing )	test_routers.py in components test_routers.py in smoke	Sufficiently covered buy automation
25	Security Groups in Basic Zone	test_security_groups.py test_01_authorizeIngressRule test_01_revokeIngressRule test_ingress_rules_specific_IP_set_non_def_sec_group test_ingress_rules_specific_IP_set test_03_stopStartVM_verifyIngressAccess test_02_revokeIngressRule_AfterDeployVM test_01_authorizeIngressRule_AfterDeployVM test_01_dhcpOnlyRouter test_01_deployVMWithUserData test_02_delete_security_grp_withoout_running_vm test_01_delete_security_grp_running_vm test_03_accessInDefaultSecurityGroup test_02_listSecurityGroups test_01_deployVM_InDefaultSecurityGroup	Sufficiently covered buy automation
26	Virtual Router Upgrades		Not covered
27