1. Objectives

In SDN solutions, some services are provided by the SDN, for example

Source NAT
Static NAT
Load Balancer
Port forwarding
Network ACL (VPC)
Firewall rules (Isolated network)

However, there are still some services which are not supported by the SDN provider, for example

Dhcp (supported by some SDN providers)
Dns (supported by some SDN providers)
UserData (it is unsupported by SDN providers)
Vpn

In this case, CloudStack VR is used for provide Dhcp/Dns/Userdata services to the vm instances. It is not in the path of the traffic, it acts as a helper vm

The goal of this feature is to support VPNs in Cloudstack VR

Remote access VPN for SDN networks
Site-to-Site VPN for SDN networks

Please note, this supports VPC only. Isolated networks is not supported

2. High Level Design

As the first step, the CloudStack VR needs a public IP (via static nat).

For Remote access vpn, the diagram is as below

Apache Cloudstack > VPN solutions for SDN networks > image-2024-12-18_15-36-19.png

For Site-to-Site VPN, the diagram is as below

In both cases, as the last step, static routes need to be create by cloudstack and configured in SDN.

3. Implementation

This described some details of the implementation

3.1 database change

a new column is added to the table user_ip_address for public ip address, which indicate if the ip is one-to-one nat to the VPC VR.

`for_router` tinyint(1) DEFAULT '0' COMMENT 'True if the ip address is used by Domain Router to expose services',

3.2 ipsec configuration for Remote access VPN

root@r-26-VM:~# cat /etc/ipsec.d/l2tp.conf 
#ipsec remote access vpn configuration
conn L2TP-PSK
        authby=secret
        rekey=no
        keyingtries=3
        leftfirewall=yes
        type=transport
        left=172.17.1.67  (private IP of first guest NIC)
        leftid=10.0.88.10 (public IP of VPC VR)
        leftprotoport=udp/l2tp
        right=%any
        rightprotoport=udp/%any
        rightsubnetwithin=0.0.0.0/0
        auto=route

root@r-26-VM:~# cat /etc/ipsec.d/ipsec.any.secrets 
: PSK "4QCNMeE9RjDccgYgPzZZmnkc"

for xl2tpd and vpn users

root@r-26-VM:~# cat /etc/xl2tpd/xl2tpd.conf 
[lns default]
ip range = 10.1.2.2-10.1.2.8
local ip = 10.1.2.1
require chap = yes
refuse pap = yes
pppoptfile =    /etc/ppp/options.xl2tpd

root@r-26-VM:~# cat /etc/ppp/chap-secrets 
# Secrets for authentication using CHAP
# client	server	secret			IP addresses
test1 * test1 *
test2 * test2 *
test3 * test3 *

3.3 ipsec configuration for Site-to-Site VPN

root@r-26-VM:~# cat /etc/ipsec.d/ipsec.vpn-10.0.80.31.conf
#conn for vpn-10.0.80.31
conn vpn-10.0.80.31
 left=172.17.1.67    (private IP of first guest NIC)
 leftid=10.0.88.10   (public IP of VPC VR)
# leftsourceip=10.0.88.10
 leftsubnet=172.17.0.0/20
 right=10.0.80.31
 rightsubnet=172.27.0.0/20
 type=tunnel
 authby=secret
 keyexchange=ike
 ike=aes128-sha1-modp1536
 ikelifetime=1440m
 esp=aes128-sha1
 lifetime=60m
 keyingtries=2
 auto=route
 forceencaps=yes
 dpddelay=30
 dpdtimeout=120
 dpdaction=restart

root@r-26-VM:~# cat /etc/ipsec.d/ipsec.vpn-10.0.80.31.secrets 
10.0.88.10 10.0.80.31 : PSK "test2"

3.4 Static Routes for VPNs on gateway

# This is an example
# 172.17.1.67 is the private IP of first guest NIC of VPC VR

root@dummy-gateway-002:~# ip route
default via 10.0.80.1 dev ens3 
10.0.80.0/20 dev ens3 proto kernel scope link src 10.0.88.1 
10.1.2.0/24 via 172.17.1.67 dev ens8           # For Remote Access VPN, the IP range is 10.1.2.1-10.1.2.8
172.17.1.0/24 dev ens8 proto kernel scope link src 172.17.1.1 
172.17.2.0/24 dev ens9 proto kernel scope link src 172.17.2.1 
172.25.0.0/20 via 172.17.1.67 dev ens8         # For Site-to-Site VPN
172.27.0.0/20 via 172.17.1.67 dev ens8         # For Site-to-Site VPN

4. How to set up VPNs

To create a VPC, please refer to https://docs.cloudstack.apache.org/en/4.19.1.0/adminguide/networking_and_traffic.html#configuring-a-virtual-private-cloud

4.1 Remote Access VPN

Remote access VPN is not supported by Source NAT for SDN networks.

(1) acquire a Public IP

Apache Cloudstack > VPN solutions for SDN networks > image-2024-12-18_16-1-3.png

Please use an IP address which is not in the subnet of Source NAT of VPC VR (not Netris)

(2) Enable Remote Access VPN

Apache Cloudstack > VPN solutions for SDN networks > image-2024-12-18_16-3-14.png

the pre-shared key is displayed

Apache Cloudstack > VPN solutions for SDN networks > image-2024-12-18_16-4-4.png

(3) Connect in VPN Client

Please refer to https://docs.cloudstack.apache.org/en/4.19.1.0/adminguide/networking/using_remote_access.html#microsoft-windows-8

Remember , For Windows clients,

Allow these protocols → Enable MSCHAP-v2
Update registry to support L2TP server behind NAT.
- REG ADD HKLM\SYSTEM\CurrentControlSet\Services\PolicyAgent /v AssumeUDPEncapsulationContextOnSendRule /t REG_DWORD /d 0x2 /f
- Restart the server
- Please refer to https://forum.netgate.com/topic/151427/solved-ipsec-doesn-t-work-if-behind-nat/4?_=1734534474834

4.2 Site-to-Site VPN

(1) Create VPN gateway

Apache Cloudstack > VPN solutions for SDN networks > image-2024-12-18_16-10-39.png

If remote access VPN is enabled, it gets the same IP

Otherwise, CloudStack automatically assign a Public IP

(2) Create customer gateway

Apache Cloudstack > VPN solutions for SDN networks > image-2024-12-18_16-12-58.png

(3) Create VPN connection

Apache Cloudstack > VPN solutions for SDN networks > image-2024-12-18_16-13-47.png

5. Test plan

Before testing

Create VPC offering
Create Network offering
Add Vpn Users

5.1 Create VPC, VPC tier and VM

Test cases	Check items and expected results
Create VPC, VPC tier and VM
Create VPC	Should succeed
Create VPC tier-1	Should succeed
Create VM-1 in VPC tier-1	Should succeed VM can get correct VM password can get userdata from VPC VR VPC VR gets placeholder IP (not VM gateway .1) use first guest NIC as default route

5.2 Setup Remote Access VPN

Setup Remote Access VPN
Setup VPN on the Source NAT IP (IP-0)	Should fail
acquire a Public IP (IP-1)	Should succeed
Enable remote access VPN (IP-1)	Should succeed Backend Enable static NAT to first guest NIC/IP of VR Add Static route for guest IP range (next hop: first guest IP of VR)
Test VPN in Windows client (with PSK-1)	Should work from VPN client to VM from VM to VPN client
Disable Static NAT (IP-1)	Should fail
Release IP (IP-1)	Should fail
Disable remote access VPN on (IP-1)	Should succeed Backend Delete Static route for guest IP range (next hop: first guest IP of VR)
Enable remote access VPN (IP-1) again	Should succeed Backend Add Static route for guest IP range (next hop: first guest IP of VR)
Test VPN in Windows client (with PSK-2)	Should work from VPN client to VM from VM to VPN client
Acquire Public IP (IP-2) and Enable VPN	Should fail
Disable remote access VPN on (IP-1)	Should succeed Delete Static route for guest IP range (next hop: first guest IP of VR)
Disable Static NAT (IP-1)	Should succeed Disable static NAT to first guest NIC/IP of VR
Acquire Public IP (IP-3) and Enable VPN	Should succeed Backend Enable static NAT to first guest NIC/IP of VR Add Static route for guest IP range (next hop: first guest IP of VR)
Test VPN in Windows client (with PSK-3)	Should work from VPN client to VM from VM to VPN client
Disable remote access VPN on (IP-3)	Should succeed Delete Static route for guest IP range (next hop: first guest IP of VR)
Disable Static NAT (IP-3)	Should succeed Disable static NAT to first guest NIC/IP of VR

5.3 Setup Site-to-Site VPN

Setup Site-to-Site VPN
Create VPN gateway	Should work Backend Acquire a Public IP (IP-4) automatically (not Source NAT IP) Enable static NAT to first guest NIC/IP of VR
(Optional) If there is no VPN server to test Create another VPC Create VPC tier and VPC VM Create VPN gateway Create VPN customer gateway (in step 1) Create VPN connection	Should work
Create VPN customer gateway (in step 2)	Should work
Create VPN connection	Should work Backend ipsec configs are created in /etc/ipsec.d/ (.conf and .secret) in in VPC VR Add Static routes for multiple CIDRs (next hop: first guest IP of VR)
Check state of VPN connection	Should be Connected or Connecting
Reset VPN connection (in VR of VPC in step 2) ipsec down vpn-xxxxxx ipsec up vpn-xxxxxx	VPN connection is Established ping/ssh work from remote to local tier-1 ping/ssh work from local tier-1 to remote
Reset VPN connection (in VR of VPC in step 1) ipsec down vpn-xxxxxx ipsec up vpn-xxxxxx	VPN connection is Established
Delete VPN connection	Should succeed Backend Delete Static routes for multiple CIDRs (next hop: first guest IP of VR)
Disable Static NAT	Should fail
Release IP	Should fail
Delete VPN gateway	Should work
Disable Static NAT (IP-4)	Should succeed Disable static NAT to first guest NIC/IP of VR

5.4 Setup both Remote Access VPN and S2S VPN

Setup both Remote Access VPN and S2S VPN
Acquire IP and Enable Remote Access VPN	Should work Backend Enable static NAT to first guest NIC/IP of VR Add Static route for guest IP range (next hop: first guest IP of VR)
Create VPN gateway	Should work Same IP as Remote Access VPN
Create VPN connection	Should work Backend ipsec configs are created in /etc/ipsec.d/ (.conf and .secret) in in VPC VR Add Static routes for multiple CIDRs (next hop: first guest IP of VR)
Test VPN in Windows client (with PSK-4)	Should work from VPN client to VM from VM to VPN client
Test Site-to-Site VPN connection	If not Established reset VPN connection in remote server Should work from remote to tier-1 from tier-1 to remote
Remove Site-to-Site VPN connection	Should work Backend Delete Static routes for multiple CIDRs (next hop: first guest IP of VR)
Disable Remote Access VPN	Should work Backend Delete Static route for guest IP range (next hop: first guest IP of VR)
Delete VPN gateway	Should work
Create VPN gateway	Should work
Create VPN connection	Should work Backend ipsec configs are created in /etc/ipsec.d/ (.conf and .secret) in in VPC VR Add Static routes for multiple CIDRs (next hop: first guest IP of VR)
Enable Remote Access VPN	Should work Backend Add Static route for guest IP range (next hop: first guest IP of VR)
Test Site-to-Site VPN connection	If not Established reset VPN connection in remote server Should work from remote to tier-1 from tier-1 to remote
Test VPN in Windows client (with PSK-4)	Should work from VPN client to VM from VM to VPN client

5.5 Test VPC tier creation

Test VPC tier creation
Create VPC tier-2	Should succeed
Create VM-2 in VPC tier-2	Should succeed VM can get correct VM password can get userdata from VPC VR VPC VR gets placeholder IP (not VM gateway .1) use first guest NIC as default route
Test Remote access VPN client to tier-2	Should work from VPN client to tier-2 from tier-2 to VPN client
Test Site-to-Site VPN to tier-2	Should work from remote to tier-2 from tier-2 to remote
Create VPC tier-3	Should succeed
Create VM-3 in VPC tier-3	Should succeed VM can get correct VM password can get userdata from VPC VR VPC VR gets placeholder IP (not VM gateway .1) use first guest NIC as default route
Test Remote access VPN client to tier-3	Should work from VPN client to tier-3 from tier-3 to VPN client
Test Site-to-Site VPN to tier-3	Should work from remote to tier-3 from tier-3 to remote

5.6 Test VPC tier deletion (first tier)

Test VPC tier deletion (first tier)
Expunge VM-1	Should succeed
Remove VPC tier-1	Should succeed Backend Disable static NAT to guest NIC/IP of tier-1 Enable static NAT to guest NIC/IP of tier-2 Delete Static routes for multiple CIDRs (next hop: guest IP of VR of tier-1) Add Static routes for multiple CIDRs (next hop: guest IP of VR of tier-2)
Check VPC VR	VPC VR: use guest NIC of tier-2 as default route l2tp.conf uses the guest IP of tier-2 vpn-xxxxxx uses the guest IP of tier-2
Test Remote access VPN client to tier-2 and tier-3	Should work from VPN client to tier-2 and tier-3 from tier-2 and tier-3 to VPN client
Test Site-to-Site VPN to tier-2 and tier-3	Should work from remote to tier-2 and tier-3 from tier-2 and tier-3 to remote

5.7 Test VPC VR reboot

Test VPC VR reboot
Reboot router	Should work VPC VR gets placeholder IP (not VM gateway .1)
Check VPC VR	VPC VR: use guest NIC of tier-2 as default route l2tp.conf uses the guest IP of tier-2 vpn-xxxxxx uses the guest IP of tier-2
Test Remote access VPN client to tier-2 and tier-3	Should work from VPN client to tier-2 and tier-3 from tier-2 and tier-3 to VPN client
Test Site-to-Site VPN to tier-2 and tier-3	Should work from remote to tier-2 and tier-3 from tier-2 and tier-3 to remote

5.8 Test VPC restart with cleanup

Test VPC restart with cleanup
Restart VPN with cleanup	Should work VPC VR gets placeholder IP (not VM gateway .1)
Check VPC VR	VPC VR: use guest NIC of tier-2 as default route l2tp.conf uses the guest IP of tier-2 vpn-xxxxxx uses the guest IP of tier-2
Test Remote access VPN client to tier-2 and tier-3	Should work from VPN client to tier-2 and tier-3 from tier-2 and tier-3 to VPN client
Test Site-to-Site VPN to tier-2 and tier-3	Should work from remote to tier-2 and tier-3 from tier-2 and tier-3 to remote

5.9 Test VPC tier deletion (from last tier)

Test VPC tier deletion (from last tier)
Expunge VM-3	Should succeed
Remove VPC tier-3	Should succeed Backend No changes.
Expunge VM-2	Should succeed
Remove VPC tier-2	Should succeed Backend Disable static NAT to guest NIC/IP of tier-2 Delete Static routes for multiple CIDRs (next hop: guest IP of VR of tier-2)
Remove VPC	Should fail
Disable Remote access VPN	Should work
Remove VPC	Should fail
Remove VPN connection	Should work
Remove VPC	Should fail
Remove VPC gateway	Should work
Remove VPC	Should work