...

• Consider case where VM1 (assume with IP 10.1.1.20) in tier1 running on host 1, wants to communicate with VM1 (10.1.2.30) in tier 2 running on host 2. sequence of flow would be:
  • 10.1.1.20 sends ARP request for 10.1.1.1 (gateway for tier1)
  • VPC VR sends ARP response with MAC address (say 3c:07:54:4a:07:8f) on which 10.1.1.1 can be reached
  • 10.1.1.20 sends packet to 10.1.2.30 with ethernet destination 3c:07:54:4a:07:8f
  • flow rule on tier 1 bridge on host 1, over rides the default flow (normal l2 switching) and sends the packet on patch port
  • logical router created for VPC on host 1 receives packet on patch port 1. logical router does route look up (flow table 1 action) and does ingress and egress ACL's and modifies source mac address with mac address of 10.1.2.1 and modifies destination mac address with mac address of 10.1.2.30 and sends packet on patch port2.
  • tier 2 bridge on host 1 receives packet on patch port, does a mac lookup.
  • if the destination mac address is found, then sends packet on the port else floods packets on all the ports
  • tier 2 bridge on host 2 receives packet and forward to VM1. 
• Consider case where VM3 (assume IP with 10.1.1.30) in tier 1 running on host 3 wants to communicate with VM1 in tier 2 running on host 2. Sequence of flow would be:
  • 10.1.1.30 sends are request for 10.1.1.1
  • VPC VR sends ARP response with MAC address (say 3c:07:54:4a:07:8f) on which 10.1.1.1 can be reached
  • 10.1.1.30 sends packet to 10.1.2.30 with ethernet destination 3c:07:54:4a:07:8f
  • VPC VR receives packet does a route look up, sends packets out on to tier 2 bridge on host 3, after modifying the packets source and destination mac address with that of 10.1.2.1 and mac address at which 10.1.2.30 is present (possibly after ARP resolution)
  • tier 2 bridge on host 2 receives packet and forward to VM1.  

Architecture & Design description

API & Service layer changes

• introduce new 'Connectivity' service capability 'distributedrouting'
• createVPCOffering API shall be enhanced to take 'distributedrouting' as capability for 'Connectivity' service. Provider specified for the 'Connectivity' service shall be validated with capabilities declared by the service provider, to ensure provider supports 'distributedrouting' capability.
• listVPCOfferings API shall return VpcOfferingResponse response that shall contain 'Connectivity' service's  

...

• 'distributedrouting' capability details of the offering if it is configured

OVS plug-in enhancements

• OVS element shall declare 'distributedrouting' as supported capability for 'Connectivity' service.
• NIC prepare enhancements:
  • current logic of preparing a NIC is described as below, if the VM's is first VM from the network being launched on a host.
    • get the list of hosts on which network spans currently
    • create tunnel from the current host on which VM being launched to all the host on which network spans
    • create tunnel from all the host on which network spans to the current host on which VM being launched
  • a check shall be made if network is part of VPC, if its part of VPC, and VPC offering does not have 'distributedrouting' capability enabled current flow of actions shall be prepared during the nic prepare phase
  • if network is part of VPC, and VPC offering has 'distributedrouting' capability enabled then following actions shall be performed.
    • if there is VPC VR running on the current host on which VM is being launched then proceed with

...

Fall back approach:

Achieving distributed routing and network ACL, would need distributed configuration. Given the scale of changes that would involve its very likely that configuration of switches and flow rules may no