NOTE: This proposal and FS is Obsolete. Please find the re-vamped FS at https://cwiki.apache.org/confluence/display/CLOUDSTACK/portable+public+IP

Introduction

Amazon Wen Services (AWS) provides EIP (Elastic IP) addresses which are static IP's that are associated with account which allows users to dynamically re-map to any instances they own [1][2]. ElP addresses allow to mask instance or availability zone failures by programmatically remapping to any instance account owns. Static allocation and dynamic remapping capabilities signifies the 'elasticity' in EIP and differentiates from the default dynamically allocated public ip for the instance. EIP combined with availability zone are used to build fail-over application architecture in AWS [3].

CloudStack provides users with ability to acquire public IP's (that are static) that are associated with the account. Users also have control to create static nat with any instances they own but with the limitations. An acquired public IP is associated with a guest network, so public IP can only be used with instances in the guest network i.e there is no ability to re-use a acquired public IP, across the instances an account owns with in a zone. With 'regions' and 'object store' vision of CloudStack, users would be able to launch instances across the zones. But there is no ability to transfer static public IP allocated to account across the zones to build a fail over strategy. It is desirable to bring the elastic characteristics of EIP in to CloudStack. 

Goals

This proposal would like to introduce following functionality.

1. EIP service with in 'basic' zone where acquired zone level public IP can be mapped (establish a static NAT) to any instance account owns in that zone. This will allow users to keep the services available against infrastructure failure within a basic zone
2. EIP service with in 'advanced' zone where acquired zone level public IP can be mapped to any instance account owns in that zone. This will allow users to keep the services available against infrastructure failure within a advanced zone
3. to introduce a new category of public IP called 'EIP' that are provisioned by admin at region level and are available for user consumption
4. to provide EIP service across zones (both basic and advanced) where acquired region level public IP can be mapped any instance an account owns in that region. This will allow users to keep the services available against entire zone failure.

Rational for #3, is that transferring a public IP from a zone (data center) to another geographically distinct zone would need additional operational and infrastructural requirements (e.g. ability to route advertise from both data centers, ability to advertise /32 host routes etc) compared to the general public IP pool configured per zone. So these set of IP's that can be transferred across zones be treated separately (similar to AWS which maintains a public IP pool and an 'elastic IP' pool).

Purpose

Purpose of this document is to present the functional requirements and details design to achieve the goals #2, #3, #4 mentioned above. CloudStack already supports for  #1, so this document just describes how it will fit in to region level EIP service.

Document History

References

[1] http://aws.amazon.com/articles/1346

[2] http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html

[3] http://support.rightscale.com/09-Clouds/AWS/02-Amazon_EC2/Designing_Failover_Architectures_on_EC2/00-Best_Practices_for_using_Elastic_IPs_(EIP)_and_Availability_Zones

[4] http://docs.aws.amazon.com/AWSEC2/latest/APIReference/OperationList-query.html

[5] https://cwiki.apache.org/confluence/display/CLOUDSTACK/GSLB+(Global+Server+Load+Balancing)+Functional+specification+and+Design+Document

Scope & Assumptions

• Scope of this proposal is present framework (CloudStack core) level changes (API, Schema, service layer changes, etc) for
  • EIP support in advanced zone (goal 2 )
  • region level EIP address pool management (goal 3)
  • EIP across the zones (goal 4)
• Scope of this proposal is restricted to enhancing NetScaler plug-in to meet goals 2-4.
• it is assumed that EIP service will be provided to isolated/shared guest networks in basic and advanced zone only. EIP service for VPC tier is not in the scope (at least for 4.2 release)
• It is assumed that region level EIP can only be used for establishing static NAT with an instance
• It is assumed that when an EIP is transferred from one zone to another, it is expected that admin performs out-of-band operation that will ensure that traffic for EIP is routed to the new zone.

Functional requirements & non-requirements

• introduce notion of 'EIP' ip pool that will be associated with a region
• Admin be able to add/delete EIP range in to/from the EIP pool
• Users be able to acquire/release EIP
• Users be able to associate/disassociate EIP with an instance
• transfer EIP from one to zone another. i.e) user can re-map an EIP from an instance in one zone to instance in another zone
• Admin and Users be able to list EIP's they own
• Users be able to associate EIP with any VM in either basic or advanced zone
• Admin should have the configuration option to turn of the region level EIP feature off entirely 
• Support for NetScaler as EIP service provider but the design should be generic so any NAT provider can be enhanced to act of EIP service provider
• When EIP is transferred from VM in a zone to VM in another zone, CloudStack will not be responsible for the actual router advertisements
• When EIP is transferred from VM in a zone to VM in another zone, CloudStack shall generate action event on which admin can act up on
• On acquire and release of elastic IP CloudStack shall generate a usage event for resource usage metering
• CloudStack shall not support EIP service for the VPC tiers VM's
• Stop/Destroy/Delete VM: EIP should be disassociated from the VM, but should remain associated with the account until the user explicitly releases it
• Reboot VM: EIP should remain associated with VM if a user reboots VM
• VM Crash: In case of a VM crash, HA VM should get the same IP

EIP semantics

EIP service shall be provided at zone (basic and advanced) level and also at region level with below semantics 

• Basic Zone: Currently EIP service is available in basic zone if the network is created with network offering with EIP service enabled. EIP service in basic zone is provided with below semantics. 
  1. When a user VM is deployed, a public ip from zone public IP pool is automatically acquired. This IP is owned by the vm's account and is marked as 'system' in the db. 
  2. When the user VM starts, static NAT is provisioned on the network EIP service provider
  3. User can also acquire a public ip by calling the associateIp API. When this api call returns, there is a public ip associated with the account, but not mapped to any guest ip.
  4. The IP acquired in (3) can be static NAT to the VM deployed in (2) by calling enableStaticNat API. This causes the IP allocated in (2) to be returned to the zone public ip pool. The network EIP service provider is programmed to remove the original mapping and carry out the new mapping.
  5. The public IP with static NAT enabled in (4) can be removed from static NAT by calling the disableStaticNAT API. This causes the network EIP service provider to remove the mapping. A new public IP is allocated (not necessarily the same as the one allocated in (2) and static NAT'ed with the guest IP of the VM. New mapping in performed on the network EIP service provider.
  6. The public Ip acquired in (2) and (5) cannot be disabled from static NAT using the disableStaticNAT API. Only Ips acquired as a result of associateIp (as in 3) can be removed from static NAT.
  7. When the VM is stopped, if the ip address has isSystem=true(allocated during the vmDeploy):
     1. the public Ip is removed from the static NAT mapping
     2. released into public pool 

• Advanced Zone: EIP service shall be available in advanced zone for the network that are created with network offering with EIP service enabled. EIP service in advaced zone is provided same EIP semantics as that of EIP service in basic zone as described above.

EIP in AWS has the below semantics [2]:

• By default each instance in EC2-Classic gets two IP addresses at launch: a private IP address and a public IP address that is mapped to the private IP address through NAT.
• When you associate an EIP with an instance, its current public IP address is released to the EC2-Classic public IP address pool.
• If you disassociate an EIP from the instance, the instance is automatically assigned a new public IP address.
• On stopping the instance also disassociates the EIP from it.

EIP service at region level combined with advanced/basic zone level EIP service together shall provide AWS EIP like (with semantics differences) semantics.

• region level EIP:
  1. When a user VM is deployed, a public ip from zone (basic/advanced) public IP pool is automatically acquired. This IP is owned by the vm's account and is marked as 'system' in the db.
  2. When the user VM starts, static NAT is provisioned on the network's EIP service provider respectively depending on zone is basic and advanced zone
  3. User can acquire a EIP by calling the allocateElasticIp API. When this api call returns, there is a EIP associated with the account, but not mapped to any guest ip.
  4. The EIP acquired in (3) can be static NAT to the VM deployed in (2) by calling associateElasticIp API. This causes the IP allocated in (2) to be returned to the zone public ip pool. The network EIP service provider is programmed to remove the original mapping. New mapping shall be carried on the 'region EIP service provider' in that zone.
  5. The EIP with static NAT enabled in (4) can be removed from static NAT by calling the disassociateElasticIp API. This causes the region EIP service provider to remove the mapping. A new public IP is allocated (not necessarily the same as the one allocated in (2) and static NAT'ed with the guest IP of the VM. New mapping in performed on the network EIP service provider.
  6. The public Ips acquired in (2) and (5) cannot be disabled from static NAT using the disableStaticNAT API. Only Ips acquired as a result of associateIp (as in 3) can be removed from static NAT.
  7. When the VM is stopped, if the associated ip address is zone public ip that has isSystem=true(allocated during the vmDeploy):
     1. the public Ip is removed from the static NAT mapping
     2. released into public pool 
  8. When the VM is stopped, if the associated ip address is EIP then
     1. the EIP is removed from the static NAT mapping
     2. EIP continue to be associated with account
  9. User can also acquire a public ip from zone public IP pool by calling the associateIp API. When this api call returns, there is a public ip associated with the account, but not mapped to any guest ip. The IP acquired in can be static NAT to the VM deployed in (2) by calling enableStaticNat API. 
  10. enableStaticNat API on the acquired IP in (9) shall only succeed if the VM is currently associated with a system allocated public ip allocated in (1). So if VM is associated with an EIP then calling enableStaticNat API, with acquired zone level public ip shall fail
  11. associateElasticIp on the EIP acquired in (3) shall only succeed if the VM is associated with system allocated public ip (in step 1).

Described region level EIP service differs from AWS EIP in following ways

• in AWS, you can replace the default public IP associated with VM only with EIP, where as in CloudStack user can acquire a zone level public IP to replace default dynamically allocated public IP
• associating EIP with a VM will fail in case of CloudStack if VM is associated with user acquired zone public IP 

Region level EIP Detailed Design

This section presents the detailed design for the region level EIP service functionality. User API are derived from the corresponding Amazon Web Services (AWS) actions ( AllocateAddress, AssociateAddress, DisassociateAddress, ReleaseAddress) [4]. 

User API

Current set of associateIpAddress, disassociateIpAddress API's used to acquire and release a public IP is restricted to zone level public IP. Also, associateIpAddress actually combines acquiring and associating the IP with a network. Once associated with a network, there is only disassociate operation available on public IP, so public IP association can not be changed from one network to another. Due to semantic differences its not best way to extend the existing set of API's dealing with public IP management to be extended to manage EIP pool of IP's. With this premise following set of API's shall be introduced.

allocateElasticIPAddress

acquires an EIP IP from the pool of elastic IP's provisioned at region level

Request parameters:

• account name; account that owns the EIP
• domain id: domain id of the account
• region id: region from which EIP shall be allocated

API Response: ElasticIpAddressReponse

releaseElasticIpAddress

release EIP acquired previously

Request parameters:

• id: elastic IP id

API response: will return 'SuccessResponse' on success. api call shall throw exception if the EIP still associated with a VM

associateElasticIpAddress

Associate acquired elastic IP with VM instance in a zone. 

Request parameters:

• id: elastic IP id
• vm instance id
• nic id
• vm guest Ip: guest IP associated with the specified VM and  Nic Id's
• reassociate: boolean flag to indicate if already assigned to another instance then re-associate the elastic IP with new VM instance. If 'false' and elastic IP is already associated with another vm instance then associateElasticIpAddress api call shall throw exception.

API Response: ElasticIpAddressReponse

disassociateElasticIpAddress

Disassociate elastic IP from a VM instance in a zone with the elastic IP

Request parameters:

• id: elastic IP id
• vm instance id

API response: will return 'SuccessResponse' on success

Admin API

Following administrator API shall be introduced to add/delete/list IP addresses in to/from EIP pool

createElasticIpAddressRange

Request parameters:

• region id
• start ip
• end ip

API Response: list of ElasticIpAddressReponse objects

deleteElasticIpAddressRange

Request parameters:

• region id
• start ip
• end ip

API response: will return 'SuccessResponse' on success

User & Admin API

listElasticIpAddress

Request parameters:

• id: elastic IP id. 
• region id

API Response: list of ElasticIpAddressReponse objects

Api Response

'ElasticIpAddressReponse' shall be introduced as API response object with following details

• UUID: id of the elastic IP
• IP address
• region id
• state: state of the EIP (allocated, free, releasing)
• account name: name of the account to which EIP is associated
• domain id; UUID of the domain to which EIP is associated
• associated: true if EIP is associated with a VM
• zone id: UUID of the zone in which EIP is associated
• Vm id: UUID of the VM to which the EIP is associated
• Nic id: UUID of the Nic of the VM to which EIP is associated
• guest IP: guest Ip to which the EIP is associated

zone id, vm id, nic id, guest IP shall be not null only if 'associated' is true.  account name and domain id shall be not null only if state is allocated or releasing

Service Layer changes

A new service 'ElasticIPService' shall be introduced with following service definitions

• createElasticIpAddressRange
• deleteElasticIpAddressRange
• listElasticIpAddressRange
• allocateElasticIpAddress
• releaseElasticIpAddress
• associateElasticIpAddress
  • shall result in calling applyStaticNatConfig() on the configured 'EipServiceProvider'
• disassociateElasticIpAddress
  • shall result in calling revokeStaticNatConfig() on the configured 'EipServiceProvider'

Region service provider

As part of GSLB feature[5] notion of region level services and providers were introduced. EIP being another region level service, A new region level service 'EIP' shall be added as region level service. New interface 'EipServiceProvider' extending 'RegionLevelServiceProivder' shall be introduced with following interface methods

• applyStaticNatConfig()
• revokeStaticNatConfig()
• isServiceEnabled() 

NetScaler plugin changes

• NetScaler plug-in shall be extended to implement 'EipServiceProvider' providing region level EIP service.
• NetScaler plug-in has the functionality to configure INAT/RNAT rules when acting as EIP service provider in basic zone. This functionality shall be re-used for providing EIP service in advanced zone and region level EIP service as well.

Schema changes

'elastic_ip_address' table shall be added with following columns

• UUID
• account id: account id to which EIP is associated
• domain id: domain id to which EIP is associated
• region id: region id to which EIP belongs
• data_center_id: zone to which EIP is currently associated (set only if EIP is associated with an instance)
• network_id: network to which EIP is associated
• vm_id: vm to which EIP is associated
• state: indicated whether EIP is allocated or free
• allocated: time stamp at which EIP is allocated

Auditing

Action Events

When user transfers from EIP from zone to another zone admin needs to be aware of the transfer so that required out-of-band actions (for e.g enabling EIP to be advertised from the new zone etc.) can be taken. A new 'Action Event' shall be generated on every association and re-association of EIP with a VM in same zone or different zone with details of source and destination zone details.

Usage events

When a EIP is acquired or released an usage event shall be generated.

Upgrade

A basic zone with EIP service will fit in to region level EIP service semantics presented. There is no upgrade specific actions to be performed for basic zones with EIP service.

UI

TBD

Limitation

• EIP can be associated only to instance that are in network (both in advanced and basic zone) that has EIP service enabled
• There can be only be one static NAT rule on an instance
• EIP service will be mutually exclusive with source nat, port forwarding

Open Issues 

none