...

Programmability of virtual switches in hypervisor combined with ability to process and take actions on data path flows with OpenFlow opens up different possibilities where L2-L4 services typically provided by virtual/physical appliances are pushed on to edge switches in the hypervisors. Current VPC network services, network ACL and inter-tier routing provided by CloudStack for east-west traffic (inter-tier traffic) can be orchestrated to be provided by virtual switches in hypervisors. Goal of this proposal to add distributed routing and ACL functionality to native SDN controller that leverages OpenVswitch capabilities to provide inter-tier routing and network ACL's at hypervisor level in distributed fashion. This would enable a scale-out model and VPC VR being choke point is avoided. Also traffic trombone problem is eliminated as traffic gets routed directly to destination hypervisor from source hypervisor.

...

flow rules: openflow rules that are configured on an openvswitch

tier: term tier is used interchangeably to network in vpc  

Conceptual model 

This section will describe conceptually how distributed routing and network ACL's are achieved achievable with an example VPC deployment with three tiers with VM's spanning three hostsuse of openflow rules and an additional bridge doing L3 routing between one or more L2 switches. Further sections builds on the concepts /design principles introduced in this section to elaborate the architecture and design on how CloudStack and OVS plug-in can orchestrate setting up VPC's with distributed routing and network ACL's. 

Here is an example VPC deployment with three tiers, with VM's spanning 3 hypervisor hosts as depicted in below diagram. In this example VPC VR is deployed on host 3. A logical router which is a OVS bridge is provisioned on the rest of the hosts (excluding the host running VPC VR) on which VPC spans. On the host on which VPC VR is running there is no logical router. Irrespective of weather a host has VM's belonging to a tier or not, a bridge is setup on each host for each tier on the all of the hosts on which VPC spans. For e.g. host 1, does not have any tier 2 VM's still a bridge is created and is in full-mesh topology with the bridges created for tier 2 on host 2 and 3. Each of the logical router on the host is connected with patch ports [3] to the bridges corresponding to tiers. This setup of logical router is done to emulate a VPC VR which has nics connected to bridges corresponding to each tier. VPC VR still needed to be deployed for north-south traffic and for other network services, so can not be entirely replaced with logical routers only.

With the understanding of how bridges corresponding to the tiers in VPC are interconnected with a logical router using patch ports lets see how flow rules can be setup to achieve distributed routing and network ACL. There are three different flow configurations on different bridges.

...

Architecture & Design description

This section describes design changes that shall be implemented in CloudStack management server and OVS plug-in to enable distributed routing and network acl functionality.

API & Service layer changes

• introduce new 'Connectivity' service capability 'distributedrouting'. This capability shall indicate 'Connectivity' service providers ability to perform distributed routing.
• createVPCOffering API shall be enhanced to take 'distributedrouting' as capability for 'Connectivity' service. Provider specified for the 'Connectivity' service shall be validated with capabilities declared by the service provider, to ensure provider supports 'distributedrouting' capability.
• listVPCOfferings API shall return VpcOfferingResponse response that shall contain 'Connectivity' service's  'distributedrouting' capability details of the offering if it is configured
• createNetworkOffering API shall throw exception if distributedrouting' capability is specified for 'Connectivity' service. 

OVS plug-in enhancements

• OVS element shall declare 'distributedrouting' as supported capability for 'Connectivity' service.
• OvsElement uses prepare() and release() phases in NIC life cycle to implement tunnels and setup bridges on hypervisors. Following changes shall be needed in nic prepare phase enhancements:
  • current logic of preparing a NIC is described as below, if the VM's is first VM from the network being launched on a host.
    • get the list of hosts on which network spans currently
    • create tunnel from the current host on which VM being launched to all the host on which network spans
    • create tunnel from all the host on which network spans to the current host on which VM being launched
  • a check shall be made if network is part of VPC, if its part of VPC, and VPC offering does not have 'distributedrouting' capability enabled current flow of actions outlined above shall be prepared performed during the nic prepare phase
  • if network is part of VPC, and VPC offering has 'distributedrouting' capability enabled then following actions shall be performed.
    • if there is VPC VR running on the current host on which VM is being launched then proceed with steps outlined above (i.e setting up tunnels just with the bridge corresponding to network).
    • if VPC VR is running on different host than the current host on which VM is being launched, then following actions shall be performed:
      • for each network in the VPC create a bridged networkcreate a bridge that shall act as logical router and connect each bridge created in previous step with patch port to logical router
      • for each of the bridge created for the tier in the VPC, form full mesh of tunnels with the hosts on which network/tier spans
      • create a bridge that shall act as logical router and connect each bridge created in previous step with patch port to logical router
      • set up flow rules on each bridge to;
        • exclude mac learning and flooding on patch port
        • for traffic destined to other tiers send the traffic on the patch port
        • for the rest of the traffic from VIF's connected to VM, tunnel interface and patch port do normal (L2 switching) processing
      • set up flow rules on logical router bridge to:
        • reflect flows corresponding to current ingress ACL's and egress ACL's set on tier
        • set flows to route traffic on appropriate patch port based on the destination ip's subnet
• nic release phase enhancements:
• VM migration handler:NetworkMigrationResponder
• replaceNetworkACLList enhancements

...