Versions Compared

Old Version 2

changes.mady.by.user Murali Reddy

Saved on

compared with

New Version Current

changes.mady.by.user Murali Reddy

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

This is summary of gap analysis done on Marvin tests that touch the virtual router functionality.


Networking in CloudStack is feature rich and flexible. Networks can be composed with flexible set of services. Network can be deployed in many different models. Here is the listing of some of the dimensions in which networks and functionality can be deployed/used.

...

  • test_dedicatePublicIpRange

General Observations

  • Hard to find tests specific to type/scenario/use-case easily. All files in one place. Other than name of the file difficult to find the tests
    • Some more test meta data/tags on the tests?

Gaps

...

High-level Automation Analysis

 

S.no FeaturesAvailability in Regression suite
 ISOLATED VR 
1Egress Firewall RulesSanity cases Automated
2Network Services :  DHCP , DNS , Static Nat , Source Nat , Port ForwardingSanity cases Automated
3Persistent NetworksSanity cases Automated
4Firewall and LoadBalancerSanity cases Automated
5Password Reset ServiceManual Testing Required
6User Data ServicesManual Testing Required
7Reset SSh Key pairManual Testing Required
8Network Update on  RVR - CIDR , Network Offerings changeManual Testing Required
9Network Update on Isolated VR's - CIDR , NW Offerings changeSanity cases Automated
10Redundant Virtual RoutersSanity cases Automated
 VPC / VPN 
11VPCSanity cases Automated
12VPC RVRManual Testing Required
13Inter Vlan RoutingManual Testing Required
14Access ListsSanity cases Automated
15Static routes / Private GatewaySanity cases Automated
16Remote access VPN on VPCManual Testing Required
17Site - Site VPNSanity cases Automated
18Remote Access VPNSanity cases Automated
19VPN UsersManual Testing Required
 Basic / Miscellaneous 
20VR + external devices side by sideSanity cases Automated
21Service Monitoring tool in VRManual Testing Required
22Shared NetworkSanity cases Automated - very basic
23Advance Zone using Pvlan Manual Testing Required
24Manual VR Operations - Stop , start , Destroy ( DomR Testing )Manual Testing Required
25Security Groups in Basic ZoneManual Testing Required
26Virtual Router UpgradesManual Testing Required


General Observations

  • Hard to find tests specific to type/scenario/use-case easily. All files in one place. Other than name of the file difficult to find the tests
    • Some more test meta data/tags on the tests?

Gaps

  • Did not find tests for DNS. Rest of the VR services has some basic tests (even for meta data, password service etc).
  • No Service capability specific tests (for eg per account NAT, LB algorithms/stickiness etc)
  • Not all protocol/traffic types/all parameters have tests.
  • No multiple public interface on the VR tests.
  • Most tests verify the CloudStack VR as black box. No extensive verification on VR on expected configuration for Iptables, dnsmasq, haproxy etc
  • no VR upgrade tests
  • not all operation on network rule have tests. For e.g, tests veifying functionality after clean up of a (firewll/pf/static nat) rule



S. NoModuleExisting testsCommentsAdditional Marvin test required
1Egress Firewall Rulestest_egress_fw_rules.pyTests sufficiently covers TCP, ICMP protocols for both deny/allow default network offering behaviors.CLOUDSTACK-9487: Egress rules marvin tests for UDP
2Network services

test_network.py

test_routers.py

test_ha_proxy.py

test_load_balancer.py

test_security_groups.py

test_vpn_service.py

test_egress_fw_rules.py

test_reset_ssh_keypair.py

 

Most of the services has test cases except for DNSCLOUDSTACK-9488: marvin tests for DNS
3Persistent Networkstest_persistent_networks.pySufficiently automated 
4Firewall and LoadBalancer Sufficiently automated 
5Password Reset Servicetest_vm_passwdenabled.pycovers password reset of the VM and VM password functionality 
6Reset SSh Key pairtest_reset_ssh_keypair.pySufficiently automated 
7Network Update on  RVR - CIDR , Network Offerings change   
8    
9Network Update on Isolated VR's - CIDR , NW Offerings change Sufficiently automated 
10Redundant Virtual Routers

test_redundant_router_cleanups.py

test_redundant_router_services.py

test_redundant_router_upgrades.py

smoke: test_vpc_redundent.py

Test sufficiently covers RVR fail over scenarios. 
11 VPC Sanity cases

test_vpc.py
test_vpc_network.py
test_vpc_network_pfrules.py
test_vpc_offerings.py
test_vpc_vm_life_cycle.py
test_vpc_distributed_routing_offering.py
test_vpc_network_lbrules.py
test_vpc_network_staticnatrule.py
test_vpc_routers.py -- tests private gateway
test_vpc_vms_deployment.py

Sufficiently covered buy automation 
12VPC RVRtest_vpc_redundant.py

Create a redundant VPC with two networks with two VMs in each network

Create a redundant VPC with 1 Tier, 1 VM, 1 ACL, 1 PF and test Network GC Nics

 

 
13 Inter Vlan Routing  Not sufficiently covered
Jira
serverASF JIRA
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyCLOUDSTACK-9476
14 Access Lists test_network.py   
15 Static routes / Private Gateway test_escalations_networks.py  listing pw gateways,  -- No check for traffic flow 
16 Remote access VPN on VPC  
Jira
serverASF JIRA
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyCLOUDSTACK-9477
17 Site - Site VPN   
Jira
serverASF JIRA
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyCLOUDSTACK-9478
18 Remote Access VPN

 test_escalations_vpncustomergateways.py 

test_vpn_service.py 

 # api test, pagination

# Validate if IPSEC is running on the public, IP by using ike-scan

 
19 VPN Users test_vpn_users.py  VPN remote access user limit tests 

20

 VR + external devices side by side

 test_asa1000v_fw.py

test_bigswitch_bcf.py

  
21Service Monitoring tool in VRNA Not covered

Jira
serverASF JIRA
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyCLOUDSTACK-9484

22Shared Network
test_shared_networks.py

Tests:
test_acquire_ip
test_createSharedNetwork_accountSpecific
test_createSharedNetwork_All
test_createSharedNetwork_domainSpecific
test_createSharedNetwork_projectSpecific
test_createSharedNetwork_usedVlan
test_createSharedNetwork_usedVlan2
test_deployVM_isolatedAndShared
test_deployVM_multipleSharedNetwork
test_escalation_ES1621
test_guest_traffic_port_groups_shared_network
test_networkWithsubdomainaccessFalse
test_networkWithsubdomainaccessTrue
test_reboot_router
test_restart_network
test_sharedNetworkOffering_01
test_sharedNetworkOffering_02
test_sharedNetworkOffering_03
test_stop_start_router 

Sufficiently covered buy automation

 
23Advance Zone using Pvlan NA Not covered
Jira
serverASF JIRA
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyCLOUDSTACK-9485
24Manual VR Operations - Stop , start , Destroy ( DomR Testing )
test_routers.py in components 
test_routers.py in smoke
Sufficiently covered buy automation 
25Security Groups in Basic Zone
test_security_groups.py

test_01_authorizeIngressRule
test_01_revokeIngressRule
test_ingress_rules_specific_IP_set_non_def_sec_group
test_ingress_rules_specific_IP_set
test_03_stopStartVM_verifyIngressAccess
test_02_revokeIngressRule_AfterDeployVM
test_01_authorizeIngressRule_AfterDeployVM
test_01_dhcpOnlyRouter
test_01_deployVMWithUserData
test_02_delete_security_grp_withoout_running_vm
test_01_delete_security_grp_running_vm
test_03_accessInDefaultSecurityGroup
test_02_listSecurityGroups
test_01_deployVM_InDefaultSecurityGroup

Sufficiently covered buy automation 
26Virtual Router Upgrades  Not covered
Jira
serverASF JIRA
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyCLOUDSTACK-9486
27    
     
     
     
     
     
     
     

 

...