Introduction

In the current VPC model in CloudStack, VPC VR provides many L3-L7 services. One of the services provided by VPC VR is to route inter-tier traffic. Entire VPC's inter-tier traffic has to get routed by VPC VR. As the size of VPC increases, VPC VR can easily become choke-point. VPC VR is also a single point-of-failure in current VPC model. There is also traffic trombone [1] problem where routing by VPC VR can become in-efficient if the source and destination VM's are placed far (in different pod/zone for e.g) from the VPC VR. Traffic trombone could become serious problem in case of region-level VPC [2]. Given the trend in recent years where datacenter east-west traffic is growing these problems are relevant.

Programmability of virtual switches in hypervisor combined with ability to process and take actions on data path flows with OpenFlow opens up different possibilities where L2-L4 services typically provided by virtual/physical appliances are pushed on to edge switches on the hypervisors [10][11]. Current VPC network services, network ACL and inter-tier routing provided by CloudStack for east-west traffic (inter-tier traffic in VPC) can be orchestrated to be provided by virtual switches in hypervisors. Goal of this proposal to add distributed routing and ACL functionality to native SDN controller that leverages OpenVswitch capabilities to provide inter-tier routing and network ACL's at hypervisor level in distributed fashion. This would enable a scale-out model and avoids VPC VR being choke point. Also traffic trombone problem is eliminated as traffic gets routed directly to destination hypervisor from source hypervisor.

This enhancement targeted for ACS 4.4 release is tracked under:CLOUDSTACK-6161

References

[1]http://blog.ipspace.net/2011/02/traffic-trombone-what-it-is-and-how-you.html

[2]https://cwiki.apache.org/confluence/display/CLOUDSTACK/Region+level+VPC+and+guest+network+spanning+multiple+zones

[3]http://blog.scottlowe.org/2012/11/27/connecting-ovs-bridges-with-patch-ports/

[4]https://cwiki.apache.org/confluence/display/CLOUDSTACK/OVS+Tunnel+Manager+for+CloudStack

[5]http://openvswitch.org/

[6]http://archive.openflow.org/wp/learnmore/

[7]http://openvswitch.org/cgi-bin/ovsman.cgi?page=vswitchd%2Fovs-vswitchd.8#LIMITS

[8]https://cwiki.apache.org/confluence/display/CLOUDSTACK/Enhancements+to+GRE-based+SDN+overlay

[9]http://archive.openflow.org/documents/openflow-spec-v1.1.0.pdf

[10]http://www.netcraftsmen.net/blogs/entry/practical-sdn-l3-forwarding-in-nsx-dfa-and-aci.html

[11]http://www.netcraftsmen.net/component/easyblog/entry/practical-sdn-l2-forwarding-in-nsx-dfa-and-aci.html

Document History

Version

Author

Date

Changes

0.1

Murali Reddy

25-Feb 2014

 

    

Scope

Glossary & Conventions

OVS:/OpenvSwitch. Open vSwitch[5] is a production quality, multilayer virtual switch designed to enable massive network automation through programmatic extension

Bridge: bridge in this document refers to a OpenVswitch bridge on XenServer/KVM

Host: host in this document shall refer to hypervisor hosts and can be XenServer/KVM

logical router: term 'logical router' shall refer to OVS bridge setup on the hypervisor which is used as a way to interconnect tiers in a VPC

full tunnel mesh: refers to how tunnels are established between the hosts in full mesh topology to create a overlay network. refer to [4] for further details.

flow rules: openflow [6] rules that are configured on an openvswitch

tier: term tier is used interchangeably to a network in the vpc  

Conceptual model 

This section will describe conceptually how distributed routing and network ACL's are achievable with use of openflow rules and with use of an additional bridge doing L3 routing between one or more L2 switches where each L2 switch represents a subnet/tier. Further sections builds on the concepts introduced in this section to elaborate the architecture and design on how OVS plug-in can orchestrate setting up VPC's with distributed routing and network ACL's. 

Here is an example VPC deployment with three tiers, with VM's spanning 3 hypervisor hosts as depicted in below diagram. In this example VPC VR is deployed on host 3. A logical router which is a OVS bridge is provisioned on the rest of the hosts (excluding the host running VPC VR) on which VPC spans. On the host on which VPC VR is running there is no logical router. Irrespective of weather a host has VM's belonging to a tier or not, a bridge is setup on each host for each tier on the all of the hosts on which VPC spans. For e.g. host 1, does not have any tier 2 VM's still a bridge is created and is in full-mesh topology with the bridges created for tier 2 on host 2 and 3. Each of the logical router on the host is connected with patch ports [3] to the bridges on the host corresponding to tiers of the VPC. This setup of logical router is done to emulate a VPC VR (which has nics connected to bridges corresponding to each tier). VPC VR still needed to be deployed for north-south traffic and for other network services, so can not be replaced with logical routers only.

With the understanding of how bridges corresponding to the tiers in VPC are interconnected with a logical router using patch ports lets see how flow rules can be setup to achieve distributed routing and network ACL. There are three different flow configurations on different bridges.

Flows rules for the bridges connected to VPC VR (e.g. bridge for tier 1 network on host 3): no new additional flow rules are added to such bridges apart from what is added by OVS tunnel manager currently.  Bridge will just act as a mac learning L2 switch with rules to handle broadcast/multicast traffic. To recap from [4] below are the flow rules. there is single table 0 for all the rows.

Flows rules for bridge connected to logical router with patch port (e.g. bridge for tier 1 network on host 1): will need additional rules to deal with patch port and ensure:

Below diagram depicts the pipeline processing [9] setup with openflow rules.

 

logical router: Flows rules for bridge acting as logical router:

Flows are setup in pipeline processing model as depicted in below diagram, to emulate packet processing on the VPC VR. A default rule with least priority (0) is set in egress ACL's table to drop all packets. Flow rules are added to egress ACL table with high priority (to overrider default rule) to forward packets to lookup table corresponding to egress network ACL's for the tier. Route look up is done in table 1 which is pre populated to resubmit to next level ingress ACL table depending on the destination subnet.  A default rule with least priority (0) is set in ingress ACL's table to drop all ingress traffic to a port. Flow rules are added to ingress ACL table with high priority (to overrider default rule) to permit packets corresponding to ingress network ACL's for the tier.

Note: When VPC VR route's packet to destination subnet, it may raise ARP request to get the MAC address of destination VM. VPC VR then forward's the packet with an ethernet header where source mac is gateway's mac address and destination mac is that of destination VM. With logical router though ARP can not be performed, end result is achieved with a flow to modify both source and destination mac address for the flow. A flow rule is required for each VM's in the subnet. Downside with this approach is for each VM created a flow rule need to be added on all hosts on which subnet/tier spans. 

Assuming tier1, tier 2 and tier3 has subnets 10.1.1.0/24, 10.1.2.0/24 and 10.1.3.0/24 respectively and corresponding bridges for the tiers are connected to logical routers on openflow ports 1,2,3, flow table would look like below with no ingress and egress rules configured.

table=0,in_port=1 actions=resubmit(,2)

table=0,in_port=2 actions=resubmit(,3)

table=0,in_port=3 actions=resubmit(,4)

table=2, priority=0 actions=drop

table=3, priority=0 actions=drop

table=4, priority=0 actions=drop

table=1,priority=0,nw_dst=10.1.1.0/24 actions=resubmit(,5)

table=1,priority=0,nw_dst=10.1.2.0/24 actions=resubmit(,6)

table=1,priority=0,nw_dst=10.1.3.0/24 actions=resubmit(,7)

table=5, priority=0 actions=drop

table=6, priority=0 actions=drop

table=7, priority=0 actions=drop

Assuming a ingress ACL to permit traffic from tier2, and egress ACL to permit outbound traffic to tier 2 is applied on tier 1 network, below new rules will be added in to flow table of the logical router bridge.

table=2, priority=1000,nw_dst=10.1.2.0/24 actions=resubmit(,1)

table=5, priority=1000,nw_src=10.1.2.0/24 actions=mod_dl_src=mac address for 10.1.2.1, modl_dl_dst=mac address for destination VM,output:1

Packet flows

Lets consider few packet flows to understand how logical router and flow rules achieve distributed routing.

Key concepts

Above example with just three hosts can be extended to VPC that spans large number of hosts. Here are the basic constructs to generalize to any number of hosts:

Fall back mechanism

Given the nature of distributed configuration required to setup bridges and flow rules on multiple hosts, there will be windows of time where the configuration is not up to date or as expected to reflect correct VPC network topology. Following principles shall be used to mitigate the impact:

enable/disable logical router

If the flow rule in the bridge that sends inter tier traffic to go through the patch port to logical router is removed, then traffic will be sent to VPC VR for routing. This fact shall be used to build notion of enable/disable logical router. When a logical router is enabled, flow rule will set on each bridge corresponding to each tier in the VPC to direct inter-tier traffic to logical router. When a logical router is disabled, flow rule set on each bridge corresponding to each tier in the VPC to direct inter-tier traffic to logical router will be removed. 

Failure mode

Architecture & Design description

This section describes design changes that shall be implemented in CloudStack management server and OVS plug-in to enable distributed routing and network acl functionality.

API & Service layer changes

OVS Network Element enhancements

OVS topology guru

Notion of VPC topology guru shall be introduced. Which shall subscribe to VM start/stop/migrate events, network life cycle events, VPC life cycle events, host connect/disconnect events to build below knowledge

OVS tunnel manager enhancements

OvsTunnel manager shall be enhanced with following functionalities;

resource layer commands

Following new resource layer commands shall be introduced.

script enhancements

Ovstunnel script shall be enhanced with following methods

troubleshooting

To aid trouble shoot in case of connectivity or network acl functionality issues when VPC is enabled with distributed router, an admin API shall be introduced in OVS plugin that shall expose below details maintained by OVS topology guru and OVS tunnel manager

UI changes

Convergence time

Following are the events that require state update of configuration and corresponding latency: 

Performance and scaling issues